search
Search
Map of Data Science
search toc
Thanks for the thanks!
close
account_circle
Profile
exit_to_app
Sign out
search
keyboard_voice
close
Searching Tips
Search for a recipe:
"Creating a table in MySQL"
Search for an API documentation: "@append"
Search for code: "!dataframe"
Apply a tag filter: "#python"
Useful Shortcuts
/ to open search panel
Esc to close search panel
to navigate between search results
d to clear all current filters
Enter to expand content preview
Doc Search
Code Search Beta
SORRY NOTHING FOUND!
mic
Start speaking...
Voice search is only supported in Safari and Chrome.
Shrink
Navigate to
near_me
Linear Algebra
52 guides
keyboard_arrow_down
1. Vectors
2. Matrices
3. Linear equations
4. Matrix determinant
5. Vector space
6. Special matrices
7. Eigenvalues and Eigenvectors
8. Orthogonality
9. Matrix decomposition
check_circle
Mark as learned
thumb_up
0
thumb_down
0
chat_bubble_outline
0
auto_stories new
settings

# Introduction to Eigenvalues and Eigenvectors

schedule Mar 5, 2023
Last updated
local_offer
Linear Algebra
Tags
map
Check out the interactive map of data science
Definition.

# Eigenvalues and eigenvectors

Eigenvectors are special vectors that can be transformed by some square matrix to become a scalar multiple of themselves, that is:

$$\boldsymbol{A}\boldsymbol{x}=\lambda \boldsymbol{x}$$

Where:

• $\boldsymbol{A}$ is the given square matrix.

• $\boldsymbol{x}$ is a non-zero vector referred to as an eigenvector of $\boldsymbol{A}$.

• $\lambda$ is a scalar referred to as an eigenvalue of $\boldsymbol{A}$.

Given a square matrix and a vector, we can easily verify whether or not the vector is an eigenvector of the matrix. Let's now go through some examples.

Example.

## Verifying that a vector is not an eigenvector of a matrix

Suppose we have the following matrix and vector:

$$\boldsymbol{A}= \begin{pmatrix} 1&2\\3&2 \end{pmatrix},\;\;\;\;\;\; \boldsymbol{x}= \begin{pmatrix} 2\\1 \end{pmatrix}$$

Show that $\boldsymbol{x}$ is not an eigenvector of $\boldsymbol{A}$.

Solution. By definition, for $\boldsymbol{x}$ to be an eigenvector of $\boldsymbol{A}$, the following equation must be satisfied:

$$\boldsymbol{Ax}=\lambda\boldsymbol{x}$$

Where $\lambda$ is some scalar. Substituting our matrix and vector gives:

\begin{align*} \begin{pmatrix} 1&2\\3&2 \end{pmatrix} \begin{pmatrix}2\\1\end{pmatrix}&= \lambda \begin{pmatrix}2\\1\end{pmatrix}\\ \begin{pmatrix}4\\8\end{pmatrix}&= \lambda \begin{pmatrix}2\\1\end{pmatrix} \end{align*}

Clearly, the left vector is not a scalar multiple of the right vector, which means that a $\lambda$ satisfying the equation does not exist. Therefore, $\boldsymbol{x}$ is not an eigenvector of $\boldsymbol{A}$.

Example.

## Verifying that a vector is an eigenvector of a matrix

Suppose we have the following matrix and vector:

$$\boldsymbol{A}= \begin{pmatrix} 1&3\\4&2 \end{pmatrix},\;\;\;\;\;\; \boldsymbol{x}= \begin{pmatrix} 3\\4 \end{pmatrix}$$

Show that $\boldsymbol{x}$ is an eigenvector of $\boldsymbol{A}$.

Solution. Again, we check whether $\boldsymbol{Ax}=\lambda\boldsymbol{x}$ holds for some scalar $\lambda$ like so:

\begin{align*} \begin{pmatrix}1&3\\4&2\end{pmatrix} \begin{pmatrix}3\\4\end{pmatrix}&= \lambda \begin{pmatrix}3\\4\end{pmatrix}\\ \begin{pmatrix}15\\20\end{pmatrix}&= \lambda \begin{pmatrix}3\\4\end{pmatrix} \end{align*}

We see that if $\lambda=5$, then the equation holds. Therefore, $\boldsymbol{x}$ is an eigenvector of $\boldsymbol{A}$ and the corresponding eigenvalue is $\lambda=5$.

Interestingly, $\boldsymbol{A}$ may have other eigenvectors and eigenvalues as well. For instance, consider the same matrix $\boldsymbol{A}$ but a different vector:

$$\boldsymbol{A}= \begin{pmatrix} 1&3\\4&2 \end{pmatrix},\;\;\;\;\;\; \boldsymbol{x}'= \begin{pmatrix} -1\\1 \end{pmatrix}$$

Let's check whether $\boldsymbol{x}'$ is an eigenvector of $\boldsymbol{A}$ or not:

\begin{align*} \begin{pmatrix}1&3\\4&2\end{pmatrix} \begin{pmatrix}-1\\1\end{pmatrix}&= \lambda \begin{pmatrix}-1\\1\end{pmatrix}\\ \begin{pmatrix}2\\-2\end{pmatrix}&= \lambda \begin{pmatrix}-1\\1\end{pmatrix} \end{align*}

This equation holds if $\lambda=-2$. Therefore, $\boldsymbol{x}'$ is indeed also an eigenvector of $\boldsymbol{A}$. We will later prove that an $n\times{n}$ matrix can have at most $n$ eigenvalues and corresponding eigenvectors.

As we have just demonstrated, verifying whether or not the given vector is an eigenvector of a matrix is easy. We will later introduce a procedure by which we can derive the eigenvalues and eigenvectors of a given matrix, but let's first go through their geometric intuition.

# Geometric intuition behind eigenvalues and eigenvectors

Recall from this sectionlink that multiplying vectors by a scalar $\lambda$ geometrically means that we either shrink or stretch the vector by a factor of $\lambda$. This is illustrated below:

The matrix-vector product $\boldsymbol{Ax}$ can be thought of as a transformation by which $\boldsymbol{A}$ converts $\boldsymbol{x}$ into some other vector. In most cases, $\boldsymbol{Ax}$ results in a vector that has a different direction to $\boldsymbol{x}$, but eigenvectors are a special type of vectors that do not change direction but instead shrinks or stretches after the transformation.

We will now derive a method to compute the eigenvalues and eigenvectors of given a matrix.

Theorem.

# Characteristic equation and polynomial

The characteristic equation of a square matrix $\boldsymbol{A}$ is defined as:

$$$$\label{eq:CvIvn46Dz2RkVUalY6c} \det(\boldsymbol{A}-\lambda\boldsymbol{I})=0$$$$

Where:

• $\lambda$ is an eigenvalue of $\boldsymbol{A}$.

• $\boldsymbol{I}$ is an identity matrix.

The left-hand side of \eqref{eq:CvIvn46Dz2RkVUalY6c} is called the characteristic polynomial of $\boldsymbol{A}$ - often denoted as $p_\boldsymbol{A}(\lambda)$.

Proof. By definition, for a given square matrix $\boldsymbol{A}$, its eigenvectors $\boldsymbol{x}$ and eigenvalues $\lambda$ satisfy the following equation:

$$$$\label{eq:HhKD2Nu4PiVGZ0yRh9E} \boldsymbol{Ax}=\lambda\boldsymbol{x}$$$$

Remember, eigenvectors are defined to be non-zero, that is, not all elements in the vector are zero.

Let's rearrange \eqref{eq:HhKD2Nu4PiVGZ0yRh9E} to derive a formula that allows us to compute the eigenvalues:

\label{eq:H24LtfkZTQpa7h9Vpr5} \begin{aligned}[b] \boldsymbol{A}\boldsymbol{x}&=\lambda \boldsymbol{x}\\ \boldsymbol{A}\boldsymbol{x}-\lambda \boldsymbol{x} &=\boldsymbol{0}\\ \boldsymbol{A}\boldsymbol{x}-\lambda \boldsymbol{I}\boldsymbol{x}&=\boldsymbol{0}\\ (\boldsymbol{A}-\lambda \boldsymbol{I})\boldsymbol{x}&=\boldsymbol{0}\\ \end{aligned}

We are now going to assume that the matrix $\boldsymbol{A}-\lambda\boldsymbol{I}$ is invertible and thus has an inverse $(\boldsymbol{A}-\lambda\boldsymbol{I})^{-1}$. Multiplying this inverse on both sides gives:

\begin{align*} (\boldsymbol{A}-\lambda \boldsymbol{I})^{-1} (\boldsymbol{A}-\lambda\boldsymbol{I})\boldsymbol{x}&=\boldsymbol{0}\\ \boldsymbol{I}\boldsymbol{x}&=\boldsymbol{0}\\ \boldsymbol{x}&=\boldsymbol{0} \end{align*}

This means that the eigenvector $\boldsymbol{x}$ equals the zero vector. However, we started out by specifying that the eigenvector must be non-zero. Therefore, our assumption that $\boldsymbol{A}-\boldsymbol{I}\lambda$ is invertible is false. Since $\boldsymbol{A}-\boldsymbol{I}\lambda$ is not invertible, we know from theoremlink that the determinant of $\boldsymbol{A}-\lambda\boldsymbol{I}$ is zero:

$$$$\label{eq:tsUuh7krQb7LFI3XLo8} \mathrm{det}(\boldsymbol{A}-\lambda\boldsymbol{I})=0$$$$

This equation is called the characteristic equation of $\boldsymbol{A}$. This completes the proof.

# Computing eigenvalues

Observe how there is only one unknown value in \eqref{eq:tsUuh7krQb7LFI3XLo8} - the eigenvalue $\lambda$. Therefore, this is the equation that allows us to find the eigenvalues of a given matrix! Let's now go through an example.

Example.

## Computing the eigenvalues of a 2x2 matrix

Compute the eigenvalues of the following matrix:

$$\boldsymbol{A}=\begin{pmatrix} 2 & 3\\ 4 & 1\\ \end{pmatrix}$$

Solution. The characteristic equation of $\boldsymbol{A}$ is:

\begin{align*} \det(\boldsymbol{A}-\lambda\boldsymbol{I}) &=0\\ \det\left[ \begin{pmatrix}2&3\\4&1\end{pmatrix} -\lambda\begin{pmatrix}1&0\\0&1\end{pmatrix}\right] &=0\\ \det\left[ \begin{pmatrix} 2-\lambda & 3 \\ 4 & 1-\lambda \\ \end{pmatrix}\right]&=0 \end{align*}

By theoremlink, we can compute the determinant of the $2\times2$ matrix:

\begin{align*} (2-\lambda)(1-\lambda)-(3)(4)&=0\\ 2-2\lambda-\lambda+\lambda^{2}-12&=0\\ \lambda^{2}-3\lambda-10&=0\\ (\lambda-5)(\lambda+2)&=0\\ \lambda&=5,\;-2\\ \end{align*}

This means that the eigenvalues of $\boldsymbol{A}$ are $5$ and $-2$. Typically, we label the eigenvalues using a subscript like so:

\begin{align*} \lambda_1&=5\\ \lambda_2&=-2\\ \end{align*}

# Computing eigenvectors

Now that we have found the eigenvalues of a given matrix $\boldsymbol{A}$, we must find the corresponding eigenvectors. Recall from \eqref{eq:H24LtfkZTQpa7h9Vpr5} that:

$$$$\label{eq:dD8bvE1bboBaLwyStHP} (\boldsymbol{A}-\lambda \boldsymbol{I})\boldsymbol{x}=\boldsymbol{0}$$$$

Here, $\boldsymbol{x}$ is the eigenvectors that we are trying to find. Remember, \eqref{eq:dD8bvE1bboBaLwyStHP} holds for eigenvalues $\lambda_1$ and $\lambda_2$. Let's substitute $\lambda=\lambda_{\color{blue}1}$ to get:

$$$$\label{eq:LilXrzPYgI8IiLQtaEV} (\boldsymbol{A}-\lambda_{\color{blue}1}\boldsymbol{I}) \boldsymbol{x}_{\color{blue}1} =\boldsymbol{0}$$$$

Here, we've added a subscript to $\boldsymbol{x}_{\color{blue}1}$ to indicate that it is an eigenvector corresponding to the eigenvalue $\lambda_{\color{blue}1}$. If $\boldsymbol{A}$ is an $2\times2$ matrix, then the matrix form of \eqref{eq:LilXrzPYgI8IiLQtaEV} is:

\label{eq:Reg09RZlyP1tf0jWFav} \begin{aligned}[b] \Big[\begin{pmatrix}a_{11}&a_{12}\\a_{21}&a_{22}\\\end{pmatrix}-\lambda_{\color{blue}1} \begin{pmatrix} 1&0\\0&1\\ \end{pmatrix}\Big]\begin{pmatrix}x_{{\color{blue}1}1}\\x_{{\color{blue}1}2}\\\end{pmatrix} &= \begin{pmatrix}0\\0\\\end{pmatrix} \\ \begin{pmatrix}a_{11}-\lambda_{\color{blue}1}&a_{12}\\a_{21}&a_{22}-\lambda_{\color{blue}1}\\\end{pmatrix} \begin{pmatrix}x_{{\color{blue}1}1}\\x_{{\color{blue}1}2}\\\end{pmatrix} &= \begin{pmatrix}0\\0\\\end{pmatrix} \end{aligned}

Since $\boldsymbol{A}$ is given, the left matrix in \eqref{eq:Reg09RZlyP1tf0jWFav} is fully defined.

NOTE

Because $\det(\boldsymbol{A}-\lambda\boldsymbol{I})=0$, we have that $\boldsymbol{A}-\lambda\boldsymbol{I}$ is not invertible. By theoremlink, this means that at least one of the rows of $\boldsymbol{A}-\lambda\boldsymbol{I}$ contains a row with all zeros. In other words, one of the rows of $\boldsymbol{A}-\lambda\boldsymbol{I}$ is redundant.

Let's focus on the first row of \eqref{eq:Reg09RZlyP1tf0jWFav} - the corresponding linear equation is:

$$(a_{11}-\lambda_1) x_{{\color{blue}1}1}+a_{12}x_{{\color{blue}1}2}=0$$

The only unknown here is the components of the eigenvectors $x_{{\color{blue}1}1}$ and $x_{{\color{blue}1}2}$. This means that we can express $x_{{\color{blue}1}1}$ in terms of $x_{{\color{blue}1}2}$ (or vice versa) like so:

$$$$\label{eq:vYtNJD1t9mOqz33vzYH} x_{{\color{blue}1}1}=-\frac{a_{12}x_{{\color{blue}1}2}}{a_{11}-\lambda_1}$$$$

For any value $x_{{\color{blue}1}2}$, equation \eqref{eq:vYtNJD1t9mOqz33vzYH} automatically determines the value of $x_{{\color{blue}1}1}$. For instance, if we set $x_{{\color{blue}1}2}=1$, then $x_{{\color{blue}1}1}$ becomes:

$$x_{{\color{blue}1}1} =-\frac{a_{12}}{a_{11}-\lambda_1}$$

The right-hand side is fully defined and is equal to some scalar value, say $c$. Therefore, the eigenvector corresponding to eigenvalue $\lambda_1$ is:

$$\boldsymbol{x}_{{\color{blue}1}}= \begin{pmatrix} x_{{\color{blue}1}1}\\ x_{{\color{blue}1}2}\end{pmatrix} = \begin{pmatrix}c\\1\\\end{pmatrix}$$

Note that we found this particular eigenvector by setting $x_{{\color{blue}1}2}=1$. If we had set a different value for $x_{{\color{blue}1}2}$, then we would end up with a different eigenvector for eigenvalue $\lambda_1$. This means that there exist infinitely many eigenvectors that correspond to a single eigenvalue!

Now that we have found the eigenvectors $\boldsymbol{x}_{{\color{blue}1}}$ corresponding to eigenvalue $\lambda_1$, we must repeat the same process to find eigenvectors $\boldsymbol{x}_{{\color{blue}2}}$ corresponding to eigenvalue $\lambda_2$. This involves:

• substituting $\lambda_2$ into $(\boldsymbol{A}-\lambda \boldsymbol{I})\boldsymbol{x}=\boldsymbol{0}$.

• expressing $x_{{\color{blue}2}1}$ in terms of $x_{{\color{blue}2}2}$ or vice versa.

• setting some value for $x_{{\color{blue}2}2}$ to find $x_{{\color{blue}2}1}$.

Don't worry if you find this step confusing - we will now go through a concrete example of computing the eigenvectors of a $2\times2$ matrix.

Example.

## Computing eigenvectors of a 2x2 matrix

Find the eigenvectors of the following matrix:

$$\boldsymbol{A}=\begin{pmatrix} 2 & 3\\ 4 & 1\\ \end{pmatrix}$$

Note that this is the same matrix as the one in the previous examplelink.

Solution. Recall that the eigenvalues of $\boldsymbol{A}$ were:

\begin{align*} \lambda_1&=5\\ \lambda_2&=-2\\ \end{align*}

Since each eigenvalue has corresponding eigenvectors, we need to compute a set of eigenvectors for $\lambda_1$ and another set for $\lambda_2$ separately.

When ​$\lambda_1=5$, we have that:

\begin{align*} (\boldsymbol{A}-\lambda_{\color{blue}1}\boldsymbol{I})\boldsymbol{x}_{\color{blue}1} &=\boldsymbol{0}\\ \Big[\begin{pmatrix}2&3\\4&1\\\end{pmatrix}-5 \begin{pmatrix} 1&0\\0&1\\ \end{pmatrix}\Big]\begin{pmatrix}x_{{\color{blue}1}1}\\x_{{\color{blue}1}2}\\\end{pmatrix} &=\begin{pmatrix}0\\0\\\end{pmatrix}\\ \begin{pmatrix} 2-5 & 3\\4 & 1-5\\ \end{pmatrix}\begin{pmatrix}x_{{\color{blue}1}1}\\x_{{\color{blue}1}2}\\\end{pmatrix}&= \begin{pmatrix}0\\0\\\end{pmatrix}\\ \begin{pmatrix}-3 & 3\\4 & -4\\\end{pmatrix} \begin{pmatrix}x_{{\color{blue}1}1}\\x_{{\color{blue}1}2}\\\end{pmatrix}&= \begin{pmatrix}0\\0\\\end{pmatrix} \end{align*}

Notice how dividing the top row by $-3$ and dividing the bottom row by $4$ would make the rows identical. This means that one of the rows is redundant. Again, this is guaranteed to happen because $\boldsymbol{A}-\lambda\boldsymbol{I}$ is not invertible and by theoremlink, we have that one of the rows is redundant.

Let's now focus on the top row:

\begin{align*} -3x_{{\color{blue}1}1}+3x_{{\color{blue}1}2}&=0\\ 3x_{{\color{blue}1}1}&=3x_{{\color{blue}1}2}\\ x_{{\color{blue}1}1}&=x_{{\color{blue}1}2} \end{align*}

This tells us that components of eigenvector $\boldsymbol{x}_{{\color{blue}1}}$ must be equal. As a general rule of thumb, we normally pick simple values such as $1$ like so:

$$\boldsymbol{x}_{{\color{blue}1}}= \begin{pmatrix} x_{{\color{blue}1}1}\\x_{{\color{blue}1}2}\\\end{pmatrix}= \begin{pmatrix} 1\\1\\\end{pmatrix}$$

Note that because eigenvectors are defined to be non-zero vectors, we cannot set $x_{{\color{blue}1}1}=0$, which leads to $x_{{\color{blue}1}2}=0$.

Now that we've computed an eigenvector for $\lambda_1$, let's find an eigenvector for $\lambda_2$. We repeat the same process:

\begin{aligned} (\boldsymbol{A}-\lambda_{\color{blue}2}\boldsymbol{I})\boldsymbol{x}_{\color{blue}2} &=\boldsymbol{0}\\ \begin{pmatrix} 2-(-2) & 3\\ 4 & 1-(-2)\\ \end{pmatrix} \begin{pmatrix} x_{{\color{blue}2}1}\\ x_{{\color{blue}2}2}\\ \end{pmatrix}&= \begin{pmatrix} 0\\0\\ \end{pmatrix}\\ \begin{pmatrix} 4 & 3\\ 4 & 3\\ \end{pmatrix} \begin{pmatrix} x_{{\color{blue}2}1}\\ x_{{\color{blue}2}2}\\ \end{pmatrix}&= \begin{pmatrix} 0\\ 0\\ \end{pmatrix} \end{aligned}

Taking the top row gives:

\begin{align*} 4x_{{\color{blue}2}1}+3x_{{\color{blue}2}2}&=0\\ 3x_{{\color{blue}2}2}&=-4x_{{\color{blue}2}1}\\ x_{{\color{blue}2}2}&=-\frac{4}{3}x_{{\color{blue}2}1} \end{align*}

To avoid fractions, let's pick $x_{{\color{blue}2}1}=3$. This means that the eigenvector $\boldsymbol{x}_{\color{blue}2}$ is:

$$\boldsymbol{x}_{{\color{blue}2}}= \begin{pmatrix} x_{{\color{blue}2}1}\\x_{{\color{blue}2}2}\\\end{pmatrix}= \begin{pmatrix} 3\\-4\\ \end{pmatrix}$$

To summarise our results, an eigenvector corresponding to the eigenvalue $\lambda_1=5$ is:

$$\boldsymbol{x}_{\color{blue}1}= \begin{pmatrix}1\\1\\\end{pmatrix}$$

An eigenvector corresponding to the eigenvalue $\lambda_2=-2$ is:

$$\boldsymbol{x}_{\color{blue}2}= \begin{pmatrix}3\\-4\\\end{pmatrix}$$

Again, keep in mind that there is an infinite number of eigenvectors and that these are just one of them.

We have so far covered examples of computing eigenvalues and eigenvectors of $2\times{2}$ matrices. The same logic applies to larger square matrices except that we typically use Gaussian elimination to solve for the eigenvectors. Let's go through a $3\times3$ case.

Example.

# Computing eigenvalues and eigenvectors of a 3x3 matrix

Compute the eigenvalues and eigenvectors of the following matrix:

$$\boldsymbol{A} =\begin{pmatrix} -2 & 3 & 1 \\ 1 & 0 & 3 \\ 3 & -3 & 2 \\ \end{pmatrix}$$

Solution. The flow is to find the eigenvalues first and then find a corresponding eigenvector for each eigenvalue.

## Computing eigenvalues

The first step is to find the eigenvalues $\lambda$ of $\boldsymbol{A}$ using the characteristic equation of $\boldsymbol{A}$, that is:

$$\mathrm{det}(\boldsymbol{A}-\lambda\boldsymbol{I})=0$$

In matrix form, this translates to:

$$\begin{vmatrix} -2-\lambda & 3 & 1 \\ 1 & 0-\lambda & 3 \\ 3 & -3 & 2-\lambda \\ \end{vmatrix}=0$$

From our guide on determinants, we know how to compute the determinant of a $3\times3$ matrix:

\begin{align*} (-2-\lambda) \begin{vmatrix} -\lambda & 3 \\ -3 & 2-\lambda \\ \end{vmatrix} -3\begin{vmatrix} 1 & 3 \\ 3 & 2-\lambda \\ \end{vmatrix} +\begin{vmatrix} 1 & -\lambda \\ 3 & -3 \\ \end{vmatrix}&=0\\ (-2-\lambda ){(-\lambda )(2-\lambda)+9}-3{(2-\lambda)-9}+(-3+3\lambda)&=0\\ (-2-\lambda)(\lambda^{2}-2\lambda+9)-3(-\lambda-7)+(3\lambda-3)&=0\\ -2\lambda^{2}+4\lambda-18-\lambda^{3}+2\lambda^{2}-9\lambda+3\lambda+21+3\lambda-3&=0\\ -\lambda^{3}+\lambda&=0\\ -\lambda(\lambda^{2}-1)&=0\\ -\lambda(\lambda+1)(\lambda-1)&=0\\ \lambda&=0,\;1,\;-1 \end{align*}

We now compute the eigenvectors for each of these eigenvalues.

## Computing 1st eigenvector

Recall that to compute the eigenvectors of an eigenvalue, we use the following equation:

$$(\boldsymbol{A}-\lambda \boldsymbol{I})\boldsymbol{x}=\boldsymbol{0}$$

When $\lambda=0$, we have that:

\label{eq:f3yQwaeNIcsf7vUnmiw} \begin{aligned}[b] \begin{pmatrix} -2-0 & 3 & 1 \\ 1 & 0-0 & 3 \\ 3 & -3 & 2-0 \\ \end{pmatrix} \begin{pmatrix} x_{1}\\ x_{2}\\ x_{3}\\ \end{pmatrix} = \begin{pmatrix} 0\\ 0\\ 0\\ \end{pmatrix}\\ \begin{pmatrix}-2&3&1\\1&0&3\\3&-3&2\end{pmatrix} \begin{pmatrix} x_{1}\\ x_{2}\\ x_{3}\\ \end{pmatrix} =\begin{pmatrix} 0\\ 0\\ 0\\ \end{pmatrix} \end{aligned}

Let's perform Gaussian elimination to solve the system:

$$$$\label{eq:lZTrA1wS9HFxYZxJBCq} \begin{pmatrix}-2&3&1\\1&0&3\\3&-3&2\end{pmatrix}\sim \begin{pmatrix}1&0&3\\-2&3&1\\3&-3&2\end{pmatrix}\sim \begin{pmatrix}1&0&3\\0&3&7\\0&3&7\end{pmatrix}\sim \begin{pmatrix}1&0&3\\0&3&7\\0&0&0\end{pmatrix}$$$$

Because the last row is all zeros, we have that $x_3$ is a free variable, which means that we are free to give it any value.

The second row of \eqref{eq:lZTrA1wS9HFxYZxJBCq} gives us:

\label{eq:vxmu9gszAPneeCSls57} \begin{aligned}[b] 3x_2+7x_3&=0\\ x_{2}&=-\frac{7}{3}x_{3} \end{aligned}

Finally, the first row of \eqref{eq:lZTrA1wS9HFxYZxJBCq} gives us:

\label{eq:gbOpsLxZ4292OAIQRhe} \begin{aligned}[b] x_1+3x_3&=0\\ x_{1}&=-3x_3 \end{aligned}

Therefore, the eigenvector corresponding to eigenvalue $\lambda=0$ is:

$$\boldsymbol{x}_1= \begin{pmatrix} x_{1}\\ x_{2}\\ x_{3} \end{pmatrix} = \begin{pmatrix} -3x_{3}\\ -\frac{7}{3}x_{3}\\ x_{3} \end{pmatrix}$$

Just as we did for the $2\times2$ case, we can pick any value for $x_3$ and the values of the other variables are automatically determined. To avoid fractions, let's set $x_3=3$, which gives us the following eigenvector:

$$\boldsymbol{x}_1= \begin{pmatrix} -9\\ -7\\ 3\\ \end{pmatrix}$$

We've managed to find an eigenvector corresponding to the first eigenvalue! To find the eigenvectors corresponding to the other eigenvalues, we just repeat the same process.

## Computing 2nd eigenvector

When $\lambda=1$, we have that:

\begin{align*} \begin{pmatrix} -2-1 & 3 & 1 \\ 1 & 0-1 & 3 \\ 3 & -3 & 2-1 \\ \end{pmatrix} \begin{pmatrix} x_{1}\\ x_{2}\\ x_{3}\\ \end{pmatrix} &= \begin{pmatrix} 0\\ 0\\ 0\\ \end{pmatrix}\\ \begin{pmatrix} -3 & 3 & 1 \\ 1 & -1 & 3 \\ 3 & -3 & 1 \\ \end{pmatrix} \begin{pmatrix} x_{1}\\ x_{2}\\ x_{3}\\ \end{pmatrix}&= \begin{pmatrix} 0\\ 0\\ 0\\ \end{pmatrix} \end{align*}

Performing Gaussian elimination gives:

$$\begin{pmatrix} -3 & 3 & 1 \\1&-1&3\\3&-3&1\end{pmatrix}\sim \begin{pmatrix}1&-1&3\\-3&3&1\\3&-3&1\end{pmatrix}\sim \begin{pmatrix}1&-1&3\\0&0&10\\0&0&2\end{pmatrix}\sim \begin{pmatrix}1&-1&3\\0&0&1\\0&0&0\end{pmatrix}$$

From the second row, we have that $x_3=0$. Substituting this into the first row gives:

\begin{align*} x_1-x_2&=0\\ x_1&=x_2 \end{align*}

Therefore, the eigenvector corresponding to the second eigenvalue is:

$$\boldsymbol{x}_2=\begin{pmatrix} x_{1}\\ x_{2}\\ x_{3} \end{pmatrix} = \begin{pmatrix} x_{1}\\ x_{1}\\ 0 \end{pmatrix}$$

For simplicity, let's set $x_1=1$. This gives us the eigenvector:

$$\boldsymbol{x}_2= \begin{pmatrix} 1\\ 1\\ 0\\ \end{pmatrix}$$

Now on to the eigenvectors of the 3rd eigenvalue!

Computing thse 3rd eigenvector
expand_more

Once again, we first obtain the system of linear equations:

\begin{align*} \begin{pmatrix} -2-(-1) & 3 & 1 \\ 1 & 0-(-1) & 3 \\ 3 & -3 & 2-(-1) \\ \end{pmatrix} \begin{pmatrix} x_{1}\\ x_{2}\\ x_{3}\\ \end{pmatrix} &= \begin{pmatrix} 0\\ 0\\ 0\\ \end{pmatrix}\\ \begin{pmatrix}-1&3&1\\1&1&3\\3&-3&3\end{pmatrix} \begin{pmatrix} x_{1}\\ x_{2}\\ x_{3}\\ \end{pmatrix} &= \begin{pmatrix} 0\\ 0\\ 0\\ \end{pmatrix} \end{align*}

Performing Gaussian elimination gives:

$$\begin{pmatrix}-1&3&1\\1&1&3\\3&-3&3\end{pmatrix}\sim \begin{pmatrix}1&1&3\\-1&3&1\\3&-3&3\end{pmatrix}\sim \begin{pmatrix}1&1&3\\0&4&4\\0&6&6\end{pmatrix}\sim \begin{pmatrix}1&1&3\\0&1&1\\0&1&1\end{pmatrix}\sim \begin{pmatrix}1&1&3\\0&1&1\\0&0&0\end{pmatrix}\sim \begin{pmatrix}1&0&2\\0&1&1\\0&0&0\end{pmatrix}$$

Again, because the third row contains all zeros, $x_3$ is free to vary. The second row gives us:

$$x_2=-x_3$$

The first row gives us:

$$x_1=-2x_3$$

Therefore, the general form of the eigenvector is:

$$\boldsymbol{x}_3= \begin{pmatrix} x_{1}\\ x_{2}\\ x_{3} \end{pmatrix} = \begin{pmatrix} -2x_{3}\\ -x_{3}\\ x_{3} \end{pmatrix}$$

Let's set $x_3=1$ to obtain a particular eigenvector:

$$\boldsymbol{x}_3= \begin{pmatrix} -2\\ -1\\ 1\\ \end{pmatrix}$$

## Summary of results

The eigenvalues of our matrix $\boldsymbol{A}$ are:

$$\lambda_1=0, \;\;\;\lambda_2=1, \;\;\;\lambda_3=-1$$

The corresponding eigenvectors are:

$$\boldsymbol{x}_1= \begin{pmatrix} -9\\-7\\3 \end{pmatrix},\;\;\; \boldsymbol{x}_2= \begin{pmatrix} 1\\1\\0 \end{pmatrix},\;\;\; \boldsymbol{x}_3= \begin{pmatrix} -2\\-1\\1 \end{pmatrix}$$

Again, be reminded there are infinitely many eigenvectors that correspond to a single eigenvalue - we just chose simple eigenvectors here.

# What is the importance of eigenvalues and eigenvectors in machine learning?

Eigenvalues and eigenvectors crop up in several machine learning topics such as:

• principal component analysis (PCA) - a popular technique for dimensionality reduction. It turns out that projecting data points onto the eigenvectors of the variance-covariance matrix of the features preserves the most amount of information.

• spectral clustering - a clustering technique that can handle data points with a non-convex layout. Similar to PCA, eigenvalues and eigenvectors are computed to perform dimensionality reduction before the actual clustering takes place.

Edited by 0 others
thumb_up
thumb_down
Comment
Citation
Ask a question or leave a feedback...
thumb_up
0
thumb_down
0
chat_bubble_outline
0
settings
Enjoy our search
Hit / to insta-search docs and recipes!