Definition

The eigenvector of a linear transformation \(\mathbf{A}_{n\times n}\) is a non-zero vector \(\mathbf{v}\) such as the linear transformation of this vector is proportional to itself: \[\mathbf{A}\mathbf{v} = \lambda \mathbf{v} \iff (\mathbf{A}-\lambda\mathbf{I})\mathbf{v} = \mathbf{0},\] where \(\lambda\) is the eigenvalue associated to the eigenvector \(\mathbf{v}\). The equation above has non-zero solution if and only if \[\det(\mathbf{A}-\lambda\mathbf{I}) = 0.\] Then, all the eigenvalues \(\lambda\) of \(\mathbf{A}\) hold the condition above.

There is an equivalence between the linear transformation \(f(\mathbf{x}) = \mathbf{A}\mathbf{x}\), and the eigenvalues \(\lambda_1, \lambda_2, \dots, \lambda_n\) and eigenvectors \(\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_n\) of itself. This relationship provide more useful interpretation of the eigenvalues and eigenvectors, we will use the change of basis concept to describe it.

dfd

Eigendecomposition and geometric interpretation

Considering a vector space \(V\) with basis \(\{\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_n\}\), any vector \(\mathbf{x} \in \mathbb{R}^n\) can be represented as \(\mathbf{V}\mathbf{x}_v\), where \(\mathbf{x}_v\) is the representation of \(\mathbf{x}\) using the matrix of basis \(\mathbf{V}=(\mathbf{v}_1, \dots, \mathbf{v}_n)\) of the vector space \(V\). Then, the linear transformation can be expressed as \[f(\mathbf{x}) = \mathbf{A}\mathbf{x} = \mathbf{A}\mathbf{V}\mathbf{x}_v = \mathbf{V}\mathbf{D}\mathbf{x}_v,\] where the diagonal matrix \(D=\text{diag}(\lambda_1, \dots, \lambda_n)\) and the last equivalence hold because \(\mathbf{A}\mathbf{v}_i=\mathbf{v}_i\lambda_i\). Finally, expressing \(\mathbf{x}_v\) in terms of the vector \(\mathbf{x}\) defined on the standard basis, we obtain that \[f(\mathbf{x}) = \mathbf{V}\mathbf{D}\mathbf{V}^{-1}\mathbf{x},\] the equality \(\mathbf{A}=\mathbf{V}\mathbf{D}\mathbf{V}^{-1}\) is called eigendecomposition. Hence, the linear transformation is equivalent to the following: change the basis of \(\mathbf{x}\) to the vector space \(V\) , apply the diagonal linear transformation \(D\) and return to the space with standard basis. Geometrically, you can think of \(\{\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_n\}\) as the basis of vectorial space \(V\) where the transformation \(\mathbf{A}\) becomes only an scaling transformation \(\mathbf{D}\) and the eigenvalues \(\lambda_1, \lambda_2, \dots, \lambda_n\) are the scaling factor in direction of the corresponding eigenvector \(\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_n\).

Basis properties

There are certain properties the are useful for statistical modelling such as:

• Trace of \(\mathbf{A}\) is equals to the sum of the eigenvalues.
• Determinant of \(\mathbf{A}\) is equals to the sum of the eigenvalues.
• If \(\mathbf{A}\) is symmetric, then all eigenvalues are real.
• If \(\mathbf{A}\) is positive definite, then all eigenvalues are positive.

Note that, some of these properties can be explained using the eigendecomposition \(\mathbf{A} = \mathbf{V}\mathbf{D}\mathbf{V}^{-1}\).