Probability - Variance

This is the measure of the deviation from the expected value (Probability - Mean)

Sample variance

If we call the average value $\bar{ξ} = (\frac{1}{n}) (ξ_{1} + ξ_{2} + ξ_{3} + . . . ξ_{n})$ , we can define the sample variance $σ^{2}$ as:

σ^{2} = (ξ_{1} - \bar{ξ})^{2} + (ξ_{2} - \bar{ξ})^{2} + . . . (ξ_{n} - \bar{ξ})^{2}

Why is it squared?

We square the difference in this formula so that negative numbers (when $ξ_{n} < \bar{ξ}$ ) also contribute to the variance, which represents the deviation from the mean.

Because of this, the units of the variance are squared from the input units. We express sample variance with $σ^{2}$ because it's most often reported as the more intuitive standard deviation $σ$ , given from the square root of the variance.

True variance

Let's write this considering the true mean (expectation), not just the sample average:

V (ξ) = E ({[ξ - E (ξ)]}^{2})

We can apply this square over the square brackets to get the following:

V (ξ) = E (ξ^{2} - 2 ξ E (ξ) + [E (ξ)]^{2})

As we proved last week and recorded in our Important Formulas, the expectation of a sum is the sum of its expectations. We can apply this here to break these apart:

V (ξ) = E (ξ^{2}) - E (2 ξ E (ξ)) + [E (ξ)]^{2}

Given that $E (E (ξ)) = E (ξ)$ , and also using the expectation with a coefficient rule to move out the two,

V (ξ) = E (ξ^{2}) - 2 {[E (ξ)]}^{2} + [E (ξ)]^{2}

V (ξ) = E (ξ^{2}) - [E (ξ)]^{2}

In terms of the sample variance from earlier, this could be expressed as:

σ^{2} = \bar{ξ^{2}} - {\bar{ξ}}^{2}

(or, drawn a bit bigger to make the bar positions clear:)

Jensen's Inequality

This reminds of Jensen's Inequality, described in part one (it matters what order you take the average in with a function). This was described in the context of Hardy-Weinberg Equilibrium.

Sum of variances

Similar to the expectations, when variances are independent then the "sum of variances is equal to the variance of the sum". This also will go in our Important Formulas:

V (ξ + η) = V (ξ) + V (η)

Since variance has no sign, it's also true that:

V (ξ - η) = V (ξ) + V (η)

If $ξ$ and $η$ are not independent, then the sum of the variance will depend on their relationship:

A positive correlation between them will give $V (ξ + η) > V (ξ) + V (η)$
A negative correlation will give $V (ξ + η) < V (ξ) + V (η)$
No correlation (independence) - they are equal.

Coefficient with variance

$V (a ξ) = a^{2} V (ξ)$
Because:

V (a ξ) = E (a^{2} ξ^{2}) - {[E (a ξ)]}^{2}

From our rule about expectation with a coefficient, we can extract $a^{2}$ :

V (a ξ) = a^{2} (E (ξ^{2}) - {[E (ξ)]}^{2})

Now the part in brackets is just equal to $V (ξ)$ - giving us our initial equation.

Sample Average

To recap, our sample average $\bar{ξ}$ is a "random number" - each time we take a sample it is different from the true expected mean by random amount due to sampling bias. It's defined using the "average" formula:

\bar{ξ} = \frac{1}{n} \sum_{i}^{n} ξ_{i}

Let's find the expectation of the sample average:

E (\bar{ξ}) = E (\frac{1}{n} \sum_{i}^{n} ξ_{i})

Based on our rule for coefficients in the expectation:

E (\bar{ξ}) = \frac{1}{n} E (\sum_{i}^{n} ξ_{i})

Since the sum of expectations = expectation of sums, we can also move the sum sign outside the expectation:

E (\bar{ξ}) = \frac{1}{n} \sum_{i}^{n} E (ξ_{i})

Let's compare with the true expectation formula we derived earlier...

Is this correct?

Honestly looking back at my notes this proof doesn't make much sense, but I will copy my notes here. I think this site explained this proof better.

I wrote: "we are adding up the values from 1 to $n$ , since the order doesn't matter, this is the same as $n \times$ the true mean".

\sum_{i} E (ξ_{i}) = n E (ξ)

Now, we combine this with the earlier formula for $E (\bar{ξ})$ that contained this sum:

E (\bar{ξ}) = \frac{1}{n} (n E (ξ))

E (\bar{ξ}) = E (ξ)

Therefore, the expectation of the sample average (AKA the population mean) is the same as the true mean. This is going in our Probability - Important Formulas - it is very important in statistics.

Variance of the sample average

V (\bar{ξ}) = V (\frac{1}{n} \sum_{i}^{n} ξ_{i})

Based on the variance of coefficient rule:

V (\bar{ξ}) = \frac{1}{n^{2}} V (\sum_{i}^{n} ξ_{i})

If each result is indepedent from the others, we can use the sum of variances rule to move the sum outside. However, this only works if each record in our dataset is independent and is not influenced by previous records.

Independence of records

Examples of how previous records could influence the next one, causing a correlation that makes this proof not true:

Stopping a survey once you receive too many answers
Asking a survey respondent to introduce you to more people to survey

Now, just like with the expectation, we can use this to show how sample variance relates to the true variance:

V (\bar{ξ}) = \frac{1}{n^{2}} \sum_{i}^{n} V (ξ_{i})

V (\bar{ξ}) = \frac{1}{n^{2}} (n V (ξ))

V (\bar{ξ}) = \frac{1}{n} V (ξ)

This is another one of our Probability - Important Formulas. It's important to note that the true variance of a population depends on sample size $n$ when compared to sample variance

Standard Error

We already showed how taking the square root of the variance is important to get the standard deviation (SD, $σ$ ). Similarly, we can take the square root of the variance of sample average to get the "Standard Error", SE. This is usually what is shown on error bars in a plot.

\sqrt{V (\bar{ξ})}

Sample Variance - formula

When you have a limited sample, use this formula. We divide by $n - 1$ instead of $n$ to correct for the fact that there is some bias involved in the sample variance formula shown above.

σ^{2} = \frac{\sum_{i} (ξ_{i} - \bar{ξ})^{2}}{n - 1}

Confusion in proving this

There was a long part proving this that I didn't really understand. Somehow, we get this formula, which comes from the difference between the true variance and the expectation of the sample variance somehow, then rearrange it:

V (ξ) - \frac{1}{n} V (ξ) = E (σ^{2})

\frac{n - 1}{n} V (ξ) = E (σ^{2})

V (ξ) = \frac{n E (σ^{2})}{n - 1}

Then this turns somehow into the sample variance formula above.

I guess that since $E (σ^{2}) = \frac{\sum_{i} (ξ_{i} - \bar{ξ})^{2}}{n}$ (the average of the variance in each value), then $n E (σ^{2}) = \sum_{i} (ξ_{i} - \bar{ξ})^{2}$ , so if we plug that in we get the final equation. I don't exactly understand how it works though, or why the initial formula works.