Covariance

Weight and Height estimates

In this chart of weight and height, there is some correlation. We can tell from this chart that large height values are often associated with higher weights. We can calculate the variance between the two variables with covariance.

The expectation of the product between $ξ - E (ξ)$ and the same formula for $η$ gives us the amount that they co-vary with each other:

C o v (ξ, η) = E [(ξ - E (ξ)) (η - E (η))]

Covariance with independent variables

If the two are independent, then the covariance will be zero. This tracks with what we learned about the product of expectations (Important Formulas) - if they are independent then the expectation of the product is the product of the expectation:

E (ξ - E (ξ)) \times E (η - E (η))

Then, based on the rules for the expectation of a sum, and given that the expectation of a single number is that number: $E (E (ξ)) = E (ξ)$ :

(E (ξ) - E (ξ)) \times (E (η) - E (η))]

$E (ξ) - E (ξ) = 0$ . So, we end up with $0 \times 0$ .

Covariance of a variable with itself

The covariance of a variable with itself (e.g. between height and height) is the same as the variance. In fact, we can think of the covariance as just a special case of the variance.

C o v (ξ, ξ) = E [(ξ - E (ξ))^{2}] = V (ξ)

Sample covariance

\frac{\sum_{i = 1}^{n} (ξ_{i} - \bar{ξ}) (η_{i} - \bar{η})}{n - 1}

This is our unbiased estimate of the covariance -but we can only estimate this, we can never get the "true" value.

Correlation coefficient

What are the units of the covariance?

In our example, we were looking at height ( $c m$ ) and weight ( $k g$ ).
Therefore, the units of our covariance would be $c m \cdot k g$ .
These units are not that useful, so we often look at the "correlation coefficient" $r$ instead:

r = \frac{c o v (ξ, η)}{\sqrt{V (ξ)} \sqrt{V (η)}}

So, the units end up being:

\frac{c m \cdot k g}{\sqrt{c m^{2}} \cdot \sqrt{k g^{2}}}

In other words, it is unitless value between -1 and 1. This is a more useful estimate and can be applied to different units and data sources and be easily understood.

Covariance Matrix

What if we want the covariance between three variables? We can arrange them in a covariance matrix. This is how Principal Component Analysis works - with a matrix of covariances.

Note that $C_{11}$ is the same as $V (1)$ , and $C_{12} = C_{21}$ .

[\begin{matrix} C_{11} & C_{12} & C_{13} \\ C_{21} & C_{22} & C_{23} \\ C_{31} & C_{32} & C_{33} \end{matrix}]