For the intuition behind covariance, please refer to this section in our guide on sample covariance. In essence, the covariance captures the linear relationship between two random variables:

Negative covariance	Zero covariance	Positive covariance

Here, note the following:

negative covariance - as $X$ increases, $Y$ tends to decrease linearly.
zero covariance - as $X$ increases, $Y$ tends does not vary linearly.
positive covariance - as $X$ increases, $Y$ tends to increase linearly.

Mathematical properties of covariance

Theorem.

Another formula to compute the covariance

The covariance of random variables $X$ and $Y$ is:

$$\mathrm{cov}(X,Y)=\mathbb{E}\left(XY\right)-\mathbb{E}(X)\cdot\mathbb{E}(Y)$$

This form is also called the computation form of covariance.

Proof. For both discrete and continuous random variables:

$$\begin{align*} \mathrm{cov}(X,Y)&= \mathbb{E}\big[(X-\mu_X)(Y-\mu_Y)\big]\\ &=\mathbb{E}(XY-X\mu_Y-Y\mu_X+\mu_X\mu_Y)\\ &=\mathbb{E}(XY)-\mathbb{E}(\mu_YX)-\mathbb{E}(\mu_XY)+\mu_X\mu_Y\\ &=\mathbb{E}(XY)-\mu_Y\cdot\mathbb{E}(X)-\mu_X\cdot\mathbb{E}(Y)+\mu_X\mu_Y\\ &=\mathbb{E}(XY)-\mu_Y\mu_X-\mu_X\mu_Y+\mu_X\mu_Y\\ &=\mathbb{E}(XY)-\mu_Y\mu_X\\ &=\mathbb{E}(XY)-\mathbb{E}(X)\cdot\mathbb{E}(Y)\\ \end{align*}$$

This completes the proof.

■

Theorem.

Covariance of X and Y is equal to the covariance of Y and X

If $X$ and $Y$ are random variables, then covariance of $X$ and $Y$ is equivalent to the covariance of $Y$ and $X$, that is:

$$\mathrm{cov}(X,Y)= \mathrm{cov}(Y,X)$$

Proof. From the definition of covariance:

$$\begin{align*} \mathrm{cov}(X,Y) &=\mathbb{E}\big[(X-\mu_X)(Y-\mu_Y)\big]\\ &=\mathbb{E}\big[(Y-\mu_Y)(X-\mu_X)\big]\\ &=\mathrm{cov}(Y,X) \end{align*}$$

This completes the proof.

■

Theorem.

Covariance of aX and bY where a and b are constants

The covariance of random variables $aX$ and $bY$ where $a$ and $b$ are constants is:

$$\mathrm{cov}(aX,bY) =\mathrm{cov}(bX,aY)=ab\cdot \mathrm{cov}(X,Y)$$

Proof. Using the definition of covariance:

$$\begin{align*} \mathrm{cov}(X,Y) &=\mathbb{E}\big[(aX-\mu_{aX})(bY-\mu_{aY})\big]\\ &=\mathbb{E}\big[(aX-\mathbb{E}(aX))(bY-\mathbb{E}(aY))\big]\\ &=\mathbb{E}\big[(aX-a\cdot\mathbb{E}(X))(bY-b\cdot\mathbb{E}(Y))\big]\\ &=\mathbb{E}\big[(aX-a\mu_X)(bY-b\mu_Y)\big]\\ &=\mathbb{E}\big[ab(X-\mu_X)(Y-\mu_Y)\big]\\ &=ab\cdot\mathbb{E}\big[(X-\mu_X)(Y-\mu_Y)\big]\\ &=ab\cdot\mathrm{cov}(X,Y) \end{align*}$$

This completes the proof.

■

Theorem.

Covariance of a+X and b+Y where a and b are constants

The covariance of random variables $a+X$ and $b+Y$ where $a$ and $b$ are constants is:

$$\mathrm{cov}(a+X,b+Y)=\mathrm{cov}(X,Y)$$

Proof. Using the TODO definition of covariance:

$$\begin{align*} \mathrm{cov}(a+X,b+Y) &=\mathbb{E}[(a+X-\mu_{a+X})(b+Y-\mu_{b+Y})]\\ &=\mathbb{E}[(a+X-\mathbb{E}(a+X))(b+Y-\mathbb{E}(b+Y))]\\ &=\mathbb{E}[(a+X-(a+\mathbb{E}(X)))(b+Y-(b+\mathbb{E}(Y)))]\\ &=\mathbb{E}[(a+X-(a+\mu_X))(b+Y-(b+\mu_Y))]\\ &=\mathbb{E}[(a+X-a-\mu_X)(b+Y-b-\mu_Y)]\\ &=\mathbb{E}[(X-\mu_X)(Y-\mu_Y)]\\ &=\mathrm{cov}(X,Y)\\ \end{align*}$$

This completes the proof.

■

Theorem.

Covariance of a random variable and a constant

The covariance of a random variable $X$ and a constant $c$ is:

$$\mathrm{cov}(X,c)=0$$

This should make sense intuitively because this means that $c$, which is a constant, doesn't vary at all with $X$.

Proof. From theorem, the covariance can also be computed as:

$$\begin{align*} \mathrm{cov}(X,c) &=\mathbb{E}\left(cX\right)-\mathbb{E}(X)\cdot\mathbb{E}(c)\\ &=c\cdot\mathbb{E}\left(X\right)-c\cdot\mathbb{E}(X)\\ &=0\\ \end{align*}$$

This completes the proof.

■

Theorem.

Covariance of two independent variables

The covariance of two independent variables $X$ and $Y$ is:

$$\mathrm{cov}(X,Y)=0$$

Proof. Recall from theorem that the expected value of independent random variables $X$ and $Y$ is:

$$\begin{equation}\label{eq:Xeo8ybtlEPTFBYChCEp} \mathrm{E}(XY)= \mathbb{E}(X)\cdot\mathbb{E}(Y) \end{equation}$$

Now, theorem tells us that the covariance can be expressed like so:

$$\begin{equation}\label{eq:i72kxQzT471RtXqPJ90} \mathrm{cov}(X,Y)=\mathbb{E}\left(XY\right)-\mathbb{E}(X)\cdot\mathbb{E}(Y) \end{equation}$$

Substituting \eqref{eq:Xeo8ybtlEPTFBYChCEp} into \eqref{eq:i72kxQzT471RtXqPJ90} gives:

$$\begin{align*} \mathrm{cov}(X,Y) &=\mathbb{E}(X)\cdot\mathbb{E}(Y) -\mathbb{E}(X)\cdot\mathbb{E}(Y)\\ &=0 \end{align*}$$

This completes the proof.

■

Theorem.

Covariance of random variable with itself

The covariance of random variable $X$ with itself is:

$$\mathrm{cov}(X,X)=\mathbb{V}(X)$$

Proof. We start with the covariance formula given by theorem:

$$\begin{align*} \mathrm{cov}(X,X) &=\mathbb{E}\big[(X-\mu_X)(X-\mu_X)\big]\\ &=\mathbb{E}\big[(X-\mu_X)^2\big]\\ &=\mathbb{V}(X) \end{align*}$$

Here, the last equality holds because of the definition of variance.

■

Theorem.

Covariance of random variable X+Y and Z

If $X$, $Y$ and $Z$ random random variables, then the covariance of $X+Y$ and $Z$ is:

$$\mathrm{cov}(X+Y,Z)= \mathrm{cov} (X,Z)+ \mathrm{cov}(Y,Z)$$

Proof. By propertylink of covariance, we have that:

$$\begin{align*} \mathrm{cov}(X+Y,Z)&= \mathbb{E}\big[(X+Y)Z\big]-\mathbb{E}(X+Y)\cdot\mathbb{E}(Z)\\ &=\mathbb{E}(XZ+YZ)-\big[\mathbb{E}(X)+\mathbb{E}(Y)\big]\cdot\mathbb{E}(Z)\\ &=\mathbb{E}(XZ)+\mathbb{E}(YZ)-\mathbb{E}(X)\cdot\mathbb{E}(Z)-\mathbb{E}(Y)\cdot\mathbb{E}(Z)\\ &=\big[\mathbb{E}(XZ)-\mathbb{E}(X)\cdot\mathbb{E}(Z)\big]+\big[\mathbb{E}(YZ)-\mathbb{E}(Y)\cdot\mathbb{E}(Z)\big]\\ &=\mathrm{cov}(X,Z)+\mathrm{cov}(Y,Z) \end{align*}$$

This completes the proof.

■

Theorem.

Covariance of series of random variables

If $X_1,X_2,\cdots,X_m$ and $Y_1,Y_2,\cdots,Y_n$ are random variables, while $a_1,a_2,\cdots,a_m$ and $b_1,b_2,\cdots,b_n$ are constants, then:

$$\mathrm{cov}\left(\sum^m_{i=1}a_iX_i,\sum^n_{j=1}b_jY_j\right)=\sum^m_{i=1}\sum^n_{j=1}\big[a_ib_j\cdot \mathrm{cov}(X_i,Y_i)\big]$$

Sketch proof. We will prove this for the case when $m=n=2$, but the proof is easy to generalize. To be able to fit our equations on the page, let's define $z$ as follows:

$$z=\mathrm{cov}(a_1X_1+a_2X_2,b_1Y_1+b_2Y_2)$$

We use the properties of covariance from earlier:

$$\begin{align*} z&=\mathrm{cov}(a_1X_1,b_1Y_1+b_2Y_2)+\mathrm{cov}(a_2X_2,b_1Y_1+b_2Y_2)\\ &=\mathrm{cov}(a_1X_1,b_1Y_1)+\mathrm{cov}(a_1X_1,b_2Y_2)+\mathrm{cov}(a_2X_2,b_1Y_1)+\mathrm{cov}(a_2X_2,b_2Y_2)\\ &=a_1b_1\cdot \mathrm{cov}(X_1,Y_1)+a_1b_2\cdot \mathrm{cov}(a_1X_1,b_2Y_2)+a_2b_1\cdot\mathrm{cov}(a_2X_2,b_1Y_1)+a_2b_2\cdot \mathrm{cov}(X_2,Y_2)\\ &=\sum^2_{i=1}\sum^2_{j=1}a_ib_j\cdot\mathrm{cov}(X_i,Y_j)\\ \end{align*}$$

This completes the proof.

■

Example.

Computing covariance of two random variables

Suppose random variables $X$ and $Y$ have the following joint probability mass function:

$f_{X,Y}(x,y)$		$x$			$f_Y(y)$
		$1$	$2$	$3$
$y$	$1$	$0.2$	$0.1$	$0.1$	$0.4$
	$2$	$0.1$	$0.3$	$0.2$	$0.6$
$f_X(x)$		$0.3$	$0.4$	$0.3$

Compute the covariance of $X$ and $Y$.

Solution. Let's find the covariance of $X$ and $Y$ by using the following property of covariance:

$$\mathrm{cov}(X,Y)=\mathbb{E}\left(XY\right)-\mathbb{E}(X)\cdot\mathbb{E}(Y)$$

Firstly, $\mathbb{E}(XY)$ is:

$$\begin{align*} \mathrm{E}(XY) &=\sum_{x,y}xy\cdot{f_{X,Y}(x,y)}\\ &=(1)(1)(0.2)+ (2)(1)(0.1)+ (3)(1)(0.1)+ (1)(2)(0.1)+ (2)(2)(0.3)+ (3)(2)(0.2)\\ &=3.3 \end{align*}$$

Next, $\mathbb{E}(X)$ and $\mathbb{E}(Y)$ are:

$$\begin{align*} \mathbb{E}(X) &=\sum_xx\cdot{f_X(x)}\\ &=1(0.3)+2(0.4)+3(0.3)\\ &=2\\\\ \mathbb{E}(Y) &=\sum_yy\cdot{f_Y(y)}\\ &=1(0.4)+2(0.6)\\ &=1.6 \end{align*}$$

Therefore, the covariance is:

$$\begin{align*} \mathrm{cov}(X,Y) &=\mathbb{E}\left(XY\right)-\mathbb{E}(X)\cdot\mathbb{E}(Y)\\ &=(3.3)-(2)(1.6)\\ &=0.1 \end{align*}$$

Since the covariance is small, $X$ and $Y$ have a weak positive linear association.

■

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!