Positive Definite Matrices: Generalizing Positive Numbers

Positive definite matrices serve as the multidimensional equivalent of positive real numbers in linear algebra. Just as positive numbers possess unique properties regarding multiplication, square roots, and calculus, positive definite matrices extend these exact characteristics into higher-dimensional vector spaces. This article explores how this generalization works by comparing real numbers and matrices across quadratic forms, eigenvalues, invertibility, square roots, and optimization.

The Quadratic Form Connection

To understand how a matrix can be "positive," we first look at how positivity is defined for a real number \(a\). For any non-zero real number \(x\), the expression \(ax^2\) is strictly greater than zero if and only if \(a\) is positive.

In matrix mathematics, the scalar \(x\) becomes a non-zero vector \(x\), and the number \(a\) becomes a symmetric matrix \(A\). The expression \(ax^2\) generalizes to the quadratic form \(x^T A x\) (where \(x^T\) is the transpose of the vector \(x\)). A symmetric matrix \(A\) is defined as positive definite if, for every non-zero vector \(x\), the result of \(x^T A x\) is strictly greater than zero.

Geometrically, multiplying a vector \(x\) by a positive definite matrix \(A\) never points the resulting vector \(Ax\) in the opposite direction of \(x\). The angle between \(x\) and \(Ax\) is always less than 90 degrees, mimicking how multiplying a number by a positive scalar never flips its sign on the number line.

Positivity of Eigenvalues

A single real number scales space in one dimension. A matrix, however, transforms space in multiple directions simultaneously. The primary directions of this transformation are called eigenvectors, and the factors by which the matrix stretches or shrinks space along these directions are called eigenvalues.

Because a positive definite matrix is the generalization of a positive number, it must stretch space positively in all principal directions. Therefore, a fundamental property of a positive definite matrix is that all of its eigenvalues are strictly positive real numbers. Consequently, its determinant (the product of its eigenvalues) and its trace (the sum of its eigenvalues) are also always positive.

Guaranteed Invertibility

In basic arithmetic, you can divide by any positive real number because it is strictly greater than zero. In linear algebra, the equivalent of division is multiplying by an inverse matrix.

A matrix is only invertible if it does not collapse space into a lower dimension, which mathematically means none of its eigenvalues can be zero. Because a positive definite matrix has strictly positive eigenvalues, it is guaranteed to be non-singular. Just as you can always divide by a positive number, you can always invert a positive definite matrix. Furthermore, the inverse of a positive definite matrix is also positive definite, just as the reciprocal of a positive number remains positive.

Existence of Square Roots

A defining trait of positive real numbers is that they have real square roots. Negative numbers require imaginary numbers to express their square roots, while positive numbers stay within the realm of real numbers.

Positive definite matrices share this exact property. Every positive definite matrix \(A\) has a unique positive definite matrix \(B\) such that \(B \times B = A\). Matrix \(B\) acts as the true square root of \(A\). Additionally, positive definite matrices allow for Cholesky decomposition, where the matrix is factored into \(A = L L^T\) (with \(L\) being a lower triangular matrix). This decomposition is widely used in computer science and engineering as a highly efficient way to solve linear equations, acting as another form of a matrix square root.

Convexity and Optimization

In single-variable calculus, you can find the minimum of a function \(f(x)\) by finding where its first derivative is zero and checking if its second derivative is positive (\(f''(x) > 0\)). A positive second derivative means the function curves upward like a bowl, guaranteeing a local minimum.

In multivariable calculus, the second derivative of a function with many variables is not a single number, but a grid of partial derivatives called the Hessian matrix. To guarantee that a multivariable function has a local minimum at a specific point, the Hessian matrix must be positive definite. When it is, the function forms a multidimensional bowl, allowing algorithms in machine learning and physics to reliably find the lowest point, or optimal solution, of a complex system.