Linear Regression and the Method of Least Squares Explained

This article explores the mathematical mechanics of linear regression and how it utilizes the method of least squares to model relationships between variables. By defining the foundational linear equation, examining prediction errors known as residuals, and demonstrating how squaring and minimizing these errors yields the optimal line of best fit, this guide explains how raw data is transformed into a reliable, predictive mathematical model.

The Goal of Linear Regression

In mathematics and statistics, linear regression is used to model the relationship between two variables:

  • The Independent Variable (\(x\)): The input or cause.
  • The Dependent Variable (\(y\)): The output or effect.

When you plot these variables on a graph as a scatter plot, linear regression attempts to draw a single straight line through the data that best represents the overall trend. The equation for this line is written in slope-intercept form:

\[\hat{y} = mx + b\]

  • \(\hat{y}\) is the predicted value of the dependent variable.
  • \(m\) is the slope of the line (how much \(y\) changes for every one-unit change in \(x\)).
  • \(x\) is the independent variable.
  • \(b\) is the y-intercept (the value of \(\hat{y}\) when \(x\) is zero).

The mathematical challenge is determining the exact values of \(m\) and \(b\) that place the line as close to all the data points as possible at the same time.

Understanding Residuals

Real-world data rarely falls into a perfectly straight line. Because of this, almost every actual data point will sit slightly above or below the regression line.

The vertical distance between an actual data point (\(y\)) and the point predicted by the line (\(\hat{y}\)) is called a residual (or error). The formula for a single residual is:

\[\text{Residual} = y - \hat{y}\]

If a data point is above the line, the residual is positive. If the data point is below the line, the residual is negative. If the line passes directly through the point, the residual is zero.

The Method of Least Squares

To find the "line of best fit," you must minimize the total distance between the line and all the data points. However, you cannot simply add all the residuals together and try to make that sum as small as possible. Because some residuals are positive and others are negative, they would cancel each other out, making a terrible line look mathematically accurate.

The method of least squares solves this problem in two steps:

  1. Square the residuals: Before adding the errors together, the method squares each individual residual: \((y - \hat{y})^2\).
  2. Minimize the sum: It then adds all of these squared values together and finds the specific line that makes this total sum as small as possible.

Squaring the residuals serves two vital mathematical purposes. First, squaring any real number always results in a positive number, which prevents positive and negative errors from canceling each other out. Second, squaring heavily penalizes large errors. A data point that is 2 units away from the line adds 4 to the sum, while a point that is 10 units away adds 100 to the sum. This forces the line to stay balanced and prevents it from straying too far from any single point.

Calculating the Line of Best Fit

The method of least squares uses calculus to find the exact minimum of the sum of squared residuals. Fortunately, this calculus simplifies into two algebraic formulas that directly calculate the optimal slope (\(m\)) and y-intercept (\(b\)).

First, you must find the mean (average) of all your \(x\) values (denoted as \(\bar{x}\)) and the mean of all your \(y\) values (denoted as \(\bar{y}\)).

1. Calculating the Slope (\(m\))

The slope is calculated using the following formula:

\[m = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}\]

In plain terms, for every data point (\(i\)), you subtract the mean of \(x\) from the current \(x\), and the mean of \(y\) from the current \(y\), then multiply those two results together. You add those products up for all data points to get your numerator. For the denominator, you subtract the mean of \(x\) from each \(x\) value, square it, and add them all together. Dividing the numerator by the denominator gives you the exact slope that minimizes the squared error.

2. Calculating the Y-Intercept (\(b\))

Once you have the slope, finding the y-intercept is straightforward because the least squares regression line is mathematically guaranteed to pass directly through the center of your data—the point \((\bar{x}, \bar{y})\). You can find \(b\) by rearranging the line equation and plugging in the means:

\[b = \bar{y} - m\bar{x}\]

Putting the Model to Use

Once the method of least squares has provided the optimal values for \(m\) and \(b\), the relationship between the variables is officially modeled. You can now plug any new \(x\) value into your finalized \(\hat{y} = mx + b\) equation to predict the corresponding \(y\) value with the highest possible mathematical accuracy for a linear trend.