Z-test

This test is not very often used because it relies on knowing the true expectation and variance of your population, but it's used as the basis for most other statistics tests.

We have a population with $E (ξ) = m$ and $V (ξ) = V$ that are known quantities.

In a sample of size $n$ , the sample average $\bar{ξ}$ is subject to sampling error and thus $\bar{ξ} \neq m$ . This difference could be due to sampling error, or it could be due to something happening. This test is meant to discern whether this difference is due to sampling error or due to another reason.

Since the sample average is normally distributed (Central Limit Theorem), if we assume that the difference is due to sampling error and our sample does not differ from the overall population (our null hypothesis $H_{0}$ ), then we can calculate the probability that this conforms with the normal distribution.

The normal distribution in this case is $N (m, \frac{V}{n})$ . If we take the arbitrary 5% cutoff value, we can test if our sample average falls into the upper or lower 2.5% of the normal distrbution and tell if there is a significant difference from what would be expected.

If our value $ξ$ is in either of the red zones above, we would say that the difference is significant at $p = 0.05$ . The 5% cutoff value is arbitrary and this is a valid critique of classical statistics.

Prosecutor's Fallacy

The biggest problem is that this still relies on the prosecutor's fallacy. Using Bayesian statistics, we need to get the posterior probability of the null hypothesis being true based on our data. However, since we can't know the probabilities of every other possibility, we cannot actually calculate this.

In math terms, we determined $P (\bar{ξ} | H_{0}) < 0.05$ . But we need to get $P (H_{0} | \bar{ξ})$

More details:

This is a two-tailed test. In some cases we might want to only look at whether it is bigger, in this case we would just look at if its in the top 5%.

The same test with a higher n

If we do the same test from a larger sample, this means we have a narrower normal distribution (lower s.d., therefore we can estimate the true mean better). As seen below, if we took the same 5% cutoff value, then our results will be significant at a value much less extreme value.

When communicating statistics, it might be better to explain as
"The data represents an outlier under the null hypothesis" - this is what statistics can tell us. It might be misleading to say something is "significant", especially when communicating with the public and people not as well-versed in statistics.

z-score

Why do we call it a z-test?

This is because of an old-fashioned concept called the "z-score" which is not much used anymore.

z = \frac{\bar{ξ} - m}{\sqrt{\frac{V}{n}}} \approx N (0, 1)

If the sample average were close to the true mean, we would end up with zero, and the standard deviation of the $z$ would be one. This means that the $z$ values follow the "standard normal distribution" with $N (0, 1)$ . The reason why this was more popular in the past is that we used to have to look up the normal distribution values in tables in books, and it was easier to look just at the standard normal distribution. Now that we have computers easily available, this isn't used as often since we can just plug in our values into a normal dist calculator online.