ANOVA (Maths)

ANOVA (Analysis of Variance)

With several samples, we can calculate if the sample variance corresponds to the Variance of the sample average
This tells us if there are differences between the samples, or if the samples are part of the same distribution

Example:

You are studying the body heights of schoolchildren in different villages. If they are all part of the same population, there will be no difference in sample . If there is some difference in nutrition for example, we might expect that one village will have a lower body height than the others. This is our null hypothesis:

We have $k$ samples of size $n$ each.

$H_{0}$ : No difference between samples (they are all from the same population.)

Our sample averages for each $k$ samples are: ${\bar{ξ}}_{1}, {\bar{ξ}}_{2}, . . . {\bar{ξ}}_{k}$
We also have the total average which we just call $\bar{ξ}$

The best estimate of the variance of the body size based on our samples is given by this formula. If we multiply this by $n$ , their averages should vary such that this is the "full" variance:

n \frac{\sum_{i = 1}^{k} ({\bar{ξ}}_{i} - \bar{ξ})^{2}}{k - 1}

If we divide this by the average of the :

\frac{1}{k} \sum_{i = 1}^{k} \frac{\sum_{j = 1}^{n} (ξ_{i j} - {\bar{ξ}}_{i})^{2}}{n - 1}

If they all behave like one population, the quotient of these two is equal to one. However, due to stochasticity and limited sample size, there is some randomness in both parts, so we will not get exactly one. We call this the famous F-statistic.

Both of these are estimates of the basic variance $V (ξ)$ if the null hypothesis is true. The numerator estimates the variance based on the sample averages $V (\bar{ξ})$ , then compensates by multiplying with $n$ to get $V (ξ)$ based on our formula

Then the denominator uses the sample variance given by our longer formula, then takes the average of all $k$ of them, this is another way of approaching the true variance $V (ξ)$ .

Significance

If the number is bigger than one by a little bit - how to tell if this is just due to stochasticity or if this shows a real differentiation from the true variance?

We need to show that $F$ is significantly bigger than one.

If $ξ$ is normally distributed (which it should be since in our example it's body height), then the distribution of the $F$ -statistic is known. We know how likely it is to get an $F$ -value of 1.5, 1.3, 0.7 etc, if the null hypothesis is true and there is no difference in our samples. We can look up on some tables or calculators online to tell us what the probability of getting a certain $F$ -statistic value.

For example, if we got an $F$ -statistic value of 2.3, let's look up the probability of getting an F-statistic equal to this or higher:
$P (F > 2.3) = 0.038$
So, this is a very unlikely outcome, and it's likely that our null hypothesis is not true.

Degrees of Freedom

We also need to consider degrees of freedom: these are $k - 1$ and $k (n - 1)$ and input these into our calculator/lookup table. This is important because:

$k - 1$ shows us how reliable the numerator is (how many samples we have).
$k (n - 1)$ shows us how reliable our estimate of the sample variance is. If we have small samples, our averages are unreliable, so it might mean that getting a high or low $F$ statistic is actually pretty likely.

Problems with the above example

We are still falling for the the prosecutor's fallacy introduced in Conditional Probability#Bayesian Inference

We have calculated $P (F \geq 2.3 | H_{0})$ , but we really should want know $P (H_{0} | F \geq 2.3)$ - however without probabilities for every alternative hypothesis we cannot calculate this. This problem is everywhere in statistics.