Given a square matrix and a vector, we can easily verify whether or not the vector is an eigenvector of the matrix. Let's now go through some examples.

Solution. By definition, for $\boldsymbol{x}$ to be an eigenvector of $\boldsymbol{A}$, the following equation must be satisfied:

Where $\lambda$ is some scalar. Substituting our matrix and vector gives:

Clearly, the left vector is not a scalar multiple of the right vector, which means that a $\lambda$ satisfying the equation does not exist. Therefore, $\boldsymbol{x}$ is not an eigenvector of $\boldsymbol{A}$.

Solution. Again, we check whether $\boldsymbol{Ax}=\lambda\boldsymbol{x}$ holds for some scalar $\lambda$ like so:

We see that if $\lambda=5$, then the equation holds. Therefore, $\boldsymbol{x}$ is an eigenvector of $\boldsymbol{A}$ and the corresponding eigenvalue is $\lambda=5$.

Interestingly, $\boldsymbol{A}$ may have other eigenvectors and eigenvalues as well. For instance, consider the same matrix $\boldsymbol{A}$ but a different vector:

Let's check whether $\boldsymbol{x}'$ is an eigenvector of $\boldsymbol{A}$ or not:

This equation holds if $\lambda=-2$. Therefore, $\boldsymbol{x}'$ is indeed also an eigenvector of $\boldsymbol{A}$. We will later prove that an $n\times{n}$ matrix can have at most $n$ eigenvalues and corresponding eigenvectors.

As we have just demonstrated, verifying whether or not the given vector is an eigenvector of a matrix is easy. We will later introduce a procedure by which we can derive the eigenvalues and eigenvectors of a given matrix, but let's first go through their geometric intuition.

Geometric intuition behind eigenvalues and eigenvectors

Recall from this sectionlink that multiplying vectors by a scalar $\lambda$ geometrically means that we either shrink or stretch the vector by a factor of $\lambda$. This is illustrated below:

The matrix-vector product $\boldsymbol{Ax}$ can be thought of as a transformation by which $\boldsymbol{A}$ converts $\boldsymbol{x}$ into some other vector. In most cases, $\boldsymbol{Ax}$ results in a vector that has a different direction to $\boldsymbol{x}$, but eigenvectors are a special type of vectors that do not change direction but instead shrinks or stretches after the transformation.

We will now derive a method to compute the eigenvalues and eigenvectors of given a matrix.

Proof. By definition, for a given square matrix $\boldsymbol{A}$, its eigenvectors $\boldsymbol{x}$ and eigenvalues $\lambda$ satisfy the following equation:

Remember, eigenvectors are defined to be non-zero, that is, not all elements in the vector are zero.

Let's rearrange \eqref{eq:HhKD2Nu4PiVGZ0yRh9E} to derive a formula that allows us to compute the eigenvalues:

We are now going to assume that the matrix $\boldsymbol{A}-\lambda\boldsymbol{I}$ is invertible and thus has an inverse $(\boldsymbol{A}-\lambda\boldsymbol{I})^{-1}$. Multiplying this inverse on both sides gives:

This means that the eigenvector $\boldsymbol{x}$ equals the zero vector. However, we started out by specifying that the eigenvector must be non-zero. Therefore, our assumption that $\boldsymbol{A}-\boldsymbol{I}\lambda$ is invertible is false. Since $\boldsymbol{A}-\boldsymbol{I}\lambda$ is not invertible, we know from theoremlink that the determinant of $\boldsymbol{A}-\lambda\boldsymbol{I}$ is zero:

This equation is called the characteristic equation of $\boldsymbol{A}$. This completes the proof.

Computing eigenvalues

Observe how there is only one unknown value in \eqref{eq:tsUuh7krQb7LFI3XLo8} - the eigenvalue $\lambda$. Therefore, this is the equation that allows us to find the eigenvalues of a given matrix! Let's now go through an example.

Solution. The characteristic equation of $\boldsymbol{A}$ is:

By theoremlink, we can compute the determinant of the $2\times2$ matrix:

This means that the eigenvalues of $\boldsymbol{A}$ are $5$ and $-2$. Typically, we label the eigenvalues using a subscript like so:

Computing eigenvectors

Now that we have found the eigenvalues of a given matrix $\boldsymbol{A}$, we must find the corresponding eigenvectors. Recall from \eqref{eq:H24LtfkZTQpa7h9Vpr5} that:

Here, $\boldsymbol{x}$ is the eigenvectors that we are trying to find. Remember, \eqref{eq:dD8bvE1bboBaLwyStHP} holds for eigenvalues $\lambda_1$ and $\lambda_2$. Let's substitute $\lambda=\lambda_{\color{blue}1}$ to get:

Here, we've added a subscript to $\boldsymbol{x}_{\color{blue}1}$ to indicate that it is an eigenvector corresponding to the eigenvalue $\lambda_{\color{blue}1}$. If $\boldsymbol{A}$ is an $2\times2$ matrix, then the matrix form of \eqref{eq:LilXrzPYgI8IiLQtaEV} is:

Since $\boldsymbol{A}$ is given, the left matrix in \eqref{eq:Reg09RZlyP1tf0jWFav} is fully defined.

Let's focus on the first row of \eqref{eq:Reg09RZlyP1tf0jWFav} - the corresponding linear equation is:

The only unknown here is the components of the eigenvectors $x_{{\color{blue}1}1}$ and $x_{{\color{blue}1}2}$. This means that we can express $x_{{\color{blue}1}1}$ in terms of $x_{{\color{blue}1}2}$ (or vice versa) like so:

For any value $x_{{\color{blue}1}2}$, equation \eqref{eq:vYtNJD1t9mOqz33vzYH} automatically determines the value of $x_{{\color{blue}1}1}$. For instance, if we set $x_{{\color{blue}1}2}=1$, then $x_{{\color{blue}1}1}$ becomes:

The right-hand side is fully defined and is equal to some scalar value, say $c$. Therefore, the eigenvector corresponding to eigenvalue $\lambda_1$ is:

Note that we found this particular eigenvector by setting $x_{{\color{blue}1}2}=1$. If we had set a different value for $x_{{\color{blue}1}2}$, then we would end up with a different eigenvector for eigenvalue $\lambda_1$. This means that there exist infinitely many eigenvectors that correspond to a single eigenvalue!

Now that we have found the eigenvectors $\boldsymbol{x}_{{\color{blue}1}}$ corresponding to eigenvalue $\lambda_1$, we must repeat the same process to find eigenvectors $\boldsymbol{x}_{{\color{blue}2}}$ corresponding to eigenvalue $\lambda_2$. This involves:

• substituting $\lambda_2$ into $(\boldsymbol{A}-\lambda \boldsymbol{I})\boldsymbol{x}=\boldsymbol{0}$.

• expressing $x_{{\color{blue}2}1}$ in terms of $x_{{\color{blue}2}2}$ or vice versa.

• setting some value for $x_{{\color{blue}2}2}$ to find $x_{{\color{blue}2}1}$.

Don't worry if you find this step confusing - we will now go through a concrete example of computing the eigenvectors of a $2\times2$ matrix.

Solution. Recall that the eigenvalues of $\boldsymbol{A}$ were:

Since each eigenvalue has corresponding eigenvectors, we need to compute a set of eigenvectors for $\lambda_1$ and another set for $\lambda_2$ separately.

When ​$\lambda_1=5$, we have that:

Notice how dividing the top row by $-3$ and dividing the bottom row by $4$ would make the rows identical. This means that one of the rows is redundant. Again, this is guaranteed to happen because $\boldsymbol{A}-\lambda\boldsymbol{I}$ is not invertible and by theoremlink, we have that one of the rows is redundant.

Let's now focus on the top row:

This tells us that components of eigenvector $\boldsymbol{x}_{{\color{blue}1}}$ must be equal. As a general rule of thumb, we normally pick simple values such as $1$ like so:

Note that because eigenvectors are defined to be non-zero vectors, we cannot set $x_{{\color{blue}1}1}=0$, which leads to $x_{{\color{blue}1}2}=0$.

Now that we've computed an eigenvector for $\lambda_1$, let's find an eigenvector for $\lambda_2$. We repeat the same process:

Taking the top row gives:

To avoid fractions, let's pick $x_{{\color{blue}2}1}=3$. This means that the eigenvector $\boldsymbol{x}_{\color{blue}2}$ is:

To summarise our results, an eigenvector corresponding to the eigenvalue $\lambda_1=5$ is:

An eigenvector corresponding to the eigenvalue $\lambda_2=-2$ is:

Again, keep in mind that there is an infinite number of eigenvectors and that these are just one of them.

We have so far covered examples of computing eigenvalues and eigenvectors of $2\times{2}$ matrices. The same logic applies to larger square matrices except that we typically use Gaussian elimination to solve for the eigenvectors. Let's go through a $3\times3$ case.

Solution. The flow is to find the eigenvalues first and then find a corresponding eigenvector for each eigenvalue.

Computing eigenvalues

The first step is to find the eigenvalues $\lambda$ of $\boldsymbol{A}$ using the characteristic equation of $\boldsymbol{A}$, that is:

In matrix form, this translates to:

From our guide on determinants, we know how to compute the determinant of a $3\times3$ matrix:

We now compute the eigenvectors for each of these eigenvalues.

Computing 1st eigenvector

Recall that to compute the eigenvectors of an eigenvalue, we use the following equation:

When $\lambda=0$, we have that:

Let's perform Gaussian elimination to solve the system:

Because the last row is all zeros, we have that $x_3$ is a free variable, which means that we are free to give it any value.

The second row of \eqref{eq:lZTrA1wS9HFxYZxJBCq} gives us:

Finally, the first row of \eqref{eq:lZTrA1wS9HFxYZxJBCq} gives us:

Therefore, the eigenvector corresponding to eigenvalue $\lambda=0$ is:

Just as we did for the $2\times2$ case, we can pick any value for $x_3$ and the values of the other variables are automatically determined. To avoid fractions, let's set $x_3=3$, which gives us the following eigenvector:

We've managed to find an eigenvector corresponding to the first eigenvalue! To find the eigenvectors corresponding to the other eigenvalues, we just repeat the same process.

Computing 2nd eigenvector

When $\lambda=1$, we have that:

Performing Gaussian elimination gives:

From the second row, we have that $x_3=0$. Substituting this into the first row gives:

Therefore, the eigenvector corresponding to the second eigenvalue is:

For simplicity, let's set $x_1=1$. This gives us the eigenvector:

Now on to the eigenvectors of the 3rd eigenvalue!

Summary of results

The eigenvalues of our matrix $\boldsymbol{A}$ are:

The corresponding eigenvectors are:

Again, be reminded there are infinitely many eigenvectors that correspond to a single eigenvalue - we just chose simple eigenvectors here.

What is the importance of eigenvalues and eigenvectors in machine learning?

Eigenvalues and eigenvectors crop up in several machine learning topics such as:

• principal component analysis (PCA) - a popular technique for dimensionality reduction. It turns out that projecting data points onto the eigenvectors of the variance-covariance matrix of the features preserves the most amount of information.

• spectral clustering - a clustering technique that can handle data points with a non-convex layout. Similar to PCA, eigenvalues and eigenvectors are computed to perform dimensionality reduction before the actual clustering takes place.