The Law of Large Numbers and Expected Value Convergence

This article explains the mathematical mechanisms behind the Law of Large Numbers (LLN) and how it guarantees that sample averages converge to the theoretical expected value as the sample size grows. We will explore the foundational concepts of independent and identically distributed variables, the role of variance reduction in averaging, Chebyshev's Inequality, and the distinct mathematical assurances provided by both the Weak and Strong Laws of Large Numbers.

The Core Concept: Sample Average vs. Expected Value

To understand how the Law of Large Numbers works, we must first define the two values it connects:

  • Expected Value (\(\mu\)): The theoretical long-run average of a random variable, determined by its probability distribution.
  • Sample Average (\(\bar{X}_n\)): The actual arithmetic mean obtained from a specific number of observed trials (\(n\)).

In a small number of trials, randomness creates significant noise, causing the sample average to fluctuate wildly. The LLN guarantees that as the number of trials (\(n\)) approaches infinity, this noise cancels out, and \(\bar{X}_n\) becomes virtually indistinguishable from \(\mu\).

The Prerequisite: I.I.D. Variables

The mathematical guarantee of the standard LLN relies on the trials being Independent and Identically Distributed (i.i.d.).

  1. Independent: The outcome of one trial does not influence the outcome of another. For example, flipping heads on a coin does not change the probability of flipping heads on the next toss.
  2. Identically Distributed: Every trial is governed by the exact same probability distribution. Consequently, every individual trial shares the same theoretical expected value (\(\mu\)) and the same finite variance (\(\sigma^2\)).

The Mechanism: Variance Reduction

The primary mathematical engine driving the Law of Large Numbers is the behavior of variance when you take an average. Variance measures how far a set of numbers is spread out from their average value.

While a single random trial has a variance of \(\sigma^2\), the variance of the sample average across \(n\) independent trials is calculated by dividing the original variance by the number of trials:

\[\text{Var}(\bar{X}_n) = \frac{\sigma^2}{n}\]

This equation is the heart of the guarantee. Because the original variance (\(\sigma^2\)) is a fixed constant, increasing the sample size (\(n\)) makes the denominator larger, which forces the overall fraction toward zero.

As \(n\) approaches infinity, the variance of the sample average shrinks to zero. A distribution with zero variance has no spread at all; it collapses onto a single point. Because the expected value of the sample average is simply \(\mu\), the distribution of the sample average collapses directly onto the expected value. Extreme outliers still occur, but their mathematical weight is diluted to irrelevance by the sheer volume of typical outcomes.

The Proof: Chebyshev’s Inequality

To prove that a shrinking variance guarantees convergence, probability mathematics uses Chebyshev’s Inequality. This theorem sets an upper limit on the probability that a random variable will deviate from its expected value by more than a certain margin of error.

Let \(\epsilon\) (epsilon) represent an arbitrarily small positive number—your chosen margin of error. Applied to the sample average, Chebyshev’s Inequality states:

\[P(|\bar{X}_n - \mu| \geq \epsilon) \leq \frac{\sigma^2}{n\epsilon^2}\]

In plain terms, the left side of the inequality represents the probability that the difference between your sample average (\(\bar{X}_n\)) and the true expected value (\(\mu\)) is greater than or equal to your tiny margin of error (\(\epsilon\)).

The right side caps this probability at \(\frac{\sigma^2}{n\epsilon^2}\). No matter how microscopic you make your margin of error \(\epsilon\), it remains a fixed number, just like \(\sigma^2\). Therefore, as you increase the sample size \(n\) toward infinity, the right side of the equation is driven down to zero. Consequently, the probability that the sample average deviates from the expected value by even a microscopic amount becomes zero.

Weak Law vs. Strong Law of Large Numbers

Probability mathematics actually provides the guarantee of convergence in two distinct ways, depending on how rigorously the limits are applied.

The Weak Law of Large Numbers (WLLN) The Weak Law relies directly on the Chebyshev proof above, guaranteeing convergence in probability. It states that for any specified margin of error, no matter how small, a sufficiently large sample size will ensure a near-100% probability that the sample average will fall within that margin. However, it leaves open the mathematical possibility that if you keep conducting trials forever, the average might occasionally wander outside the margin before returning.

The Strong Law of Large Numbers (SLLN) The Strong Law provides a stricter guarantee known as almost sure convergence. Instead of looking at the probability of deviation at a specific large \(n\), it looks at the entire infinite sequence of trials as a whole. It states that the probability that the sample average eventually converges to the expected value—and stays there permanently without ever oscillating away again—is exactly 1. This requires more complex proofs than Chebyshev's Inequality (such as Kolmogorov's inequalities), cementing the fact that divergence in an infinite sequence is a mathematical impossibility.