How SVD Factors Matrices into Rotation and Scaling
Singular Value Decomposition (SVD) is a fundamental technique in applied mathematics that breaks down any rectangular matrix into three simpler geometric transformations. This article provides a clear breakdown of how SVD mathematically and geometrically isolates a complex linear transformation into an initial rotation, a coordinate scaling, and a final rotation. By examining the three distinct component matrices generated during this process, you will understand how SVD successfully generalizes matrix factorization to non-square matrices of any dimension.
The Core Equation of SVD
In linear algebra, a matrix represents a linear transformation that maps vectors from one space to another. If you have a rectangular matrix \(A\) of dimensions \(m \times n\), it takes an \(n\)-dimensional vector and transforms it into an \(m\)-dimensional vector.
SVD states that any \(m \times n\) matrix \(A\) can be factored into the product of three specific matrices:
\[A = U \Sigma V^T\]
Each of these three matrices represents a distinct geometric action:
- \(V^T\) (V-transpose): An \(n \times n\) orthogonal matrix that performs the first rotation.
- \(\Sigma\) (Sigma): An \(m \times n\) diagonal matrix that performs the scaling.
- \(U\): An \(m \times m\) orthogonal matrix that performs the final rotation.
Note: Because \(U\) and \(V^T\) are orthogonal matrices, their geometric actions are technically "proper rotations" or "reflections" (improper rotations). In applied mathematics, these length-preserving transformations are broadly referred to as rotations.
The Three-Step Geometric Transformation
When you multiply a vector \(x\) by the matrix \(A\) (computing \(Ax\)), SVD allows you to view the operation as a sequence of three step-by-step geometric adjustments: \(U(\Sigma(V^T x))\).
Step 1: The Initial Rotation (\(V^T\))
The process begins by multiplying the input vector \(x\) by \(V^T\). Because \(V^T\) is an orthogonal matrix, it does not change the length of the vector or the angles between vectors. Instead, it rotates the vector in the original \(n\)-dimensional space. The purpose of this rotation is to align the input space's principal axes with the standard coordinate axes, preparing the vector to be scaled cleanly.
Step 2: Scaling and Dimension Shifting (\(\Sigma\))
Next, the rotated vector is multiplied by \(\Sigma\). This matrix contains values only along its main diagonal, known as singular values (\(\sigma_1, \sigma_2, ...\)), which are always non-negative and ordered from largest to smallest. All other entries in \(\Sigma\) are zero.
This step stretches or shrinks the vector along the newly aligned coordinate axes. A singular value greater than 1 stretches the axis, a value between 0 and 1 shrinks it, and a value of 0 collapses that dimension entirely.
Because \(A\) is a rectangular matrix (\(m \times n\)), \(\Sigma\) is also \(m \times n\). This means \(\Sigma\) is responsible for changing the dimension of the vector.
- If \(m > n\), \(\Sigma\) pads the vector with zeros, embedding it into a higher-dimensional space.
- If \(m < n\), \(\Sigma\) drops coordinates, projecting the vector into a lower-dimensional space.
Step 3: The Final Rotation (\(U\))
Finally, the scaled vector is multiplied by \(U\). Like \(V^T\), \(U\) is an orthogonal matrix, meaning it only rotates (or reflects) the vector without altering its newly scaled lengths. This operation rotates the stretched vector into its final orientation within the \(m\)-dimensional output space.
How SVD Calculates the Components
A rectangular matrix \(A\) cannot be directly decomposed using standard eigenvalues because it is not square. SVD bypasses this limitation by utilizing two symmetric, square matrices derived from \(A\): the covariance-like matrices \(A^T A\) and \(A A^T\).
- Finding \(V\): You compute \(A^T A\), which yields a square \(n \times n\) symmetric matrix. The normalized eigenvectors of \(A^T A\) form the columns of \(V\). Transposing this gives the initial rotation matrix \(V^T\).
- Finding \(U\): You compute \(A A^T\), which yields a square \(m \times m\) symmetric matrix. The normalized eigenvectors of \(A A^T\) form the columns of the final rotation matrix \(U\).
- Finding \(\Sigma\): Both \(A^T A\) and \(A A^T\) share the exact same positive eigenvalues. Taking the square root of these eigenvalues gives the singular values. Placing these values on the main diagonal of an \(m \times n\) zero matrix constructs the scaling matrix \(\Sigma\).
By bridging the input space (\(A^T A\)) and the output space (\(A A^T\)), SVD guarantees that any matrix, regardless of its shape or whether it is invertible, can be cleanly broken down into pure rotation and scaling components.