Fishes Example (Power Analysis)

Continuing from the example from week 1, let's consider experimental design analysis

We are comparing to populations to determine if they have different trait distributions. To do this, we conduct field sampling where we collect data from $n$ individuals from both populations. But how many do we need to sample to conclude this? How large does $n$ need to be?

Defining the question

Let's formalize our question in statistical terms with the power analysis approach.

Assuming our traits are normally distributed and both mean and variance differ:

Population 1:

Population 2:

The lecture notes have an example of how to set the means and variances ofthese two populations as different values:

A t-test comparing 10 random samples from each shows no significant difference. This is because our sample size was to small compared to our defined variances, so the randomness from our samples doesn't allow us to find a significant difference between them. This wouldn't happen every time of course, we maybe just got "bad luck" with our sample.

Since the population mean estimator, the sample mean, becomes less variable with increasing $n$ (see Important Formulas), we will be less likely to get an "unlucky" p-value with higher $n$.

Factors affecting size of $n$

1. True population mean
2. True population Variance
3. False negative probability we can tolerate
4. Desired significance level

Simulation experiments

Trend

As the difference between population meansincreases, the number of samples required to statistically find the difference decreases.

As variance in the population distribution increases, we need more samples to statistically find the difference.