The covariance generalizes the concept of variance to multiple random variables. Instead of measuring the fluctuation of a single random variable, the covariance measures the fluctuation of two variables with each other.
Recall that the variance is the mean squared deviation from the mean for a single random variable : The covariance adopts an analogous functional form.
The covariance of random variables and is defined asNow, instead of measuring the fluctuation of a single variable, the covariance measures how two variables fluctuate together. For the covariance to be large, both and must be large at the same time or, in other words, change together.
It is generally simpler to find the covariance by taking In other words, to compute the covariance, one can equivalently find (in addition to the means of and ).
Similarly, one can find an expression in terms of variances:
A generalized statement of this result is as follows.
Variance of a sum. Given random variables , each with finite variance,
The covariance inherits many of the same properties as the inner product from linear algebra. The proof involves straightforward algebra and is left as an exercise for the reader.
Given a constant and random variables , , and , the following properties hold:
Let and be random variables such that and , where and are constants. Determine .
The inner product properties yield
As a result, the Cauchy-Schwarz inequality holds for covariances.
Cauchy-Schwarz inequality. Given random variables and ,
One of the key properties of the covariance is the fact that independent random variables have zero covariance.
Covariance of independent variables. If and are independent random variables, then
If and are independent, then and therefore . (Recall that is a simple consequence of the fact that .)
Dependent variables with zero covariance. However, the converse is not in general true. As a simple example, suppose that is a standard normal random variable and that . Notice that knowledge of completely determines , in which case and are very clearly dependent. However, by symmetry it holds that
A simple corollary is as follows.
Variance of the sum of independent variables. Given independent random variables , each with finite variance,
Since the are independent, it must be the case that for all , and the result follows directly from the variance of a sum theorem.
When dealing with a large number of random variables , it makes sense to consider a covariance matrix whose th entry is .
Since , the covariance matrix is symmetric.
 DeGroot, Morris H. Probability and Statistics. Second edition. Addison-Wesley, 1985.