Understanding the Central Limit Theorem
The Central Limit Theorem in the wild
On many occasions, you’re trying to ask questions about the average of some population of unknown distribution to find a confidence interval or test a hypothesis. Imagine you’ve taken some relatively large IID random sample, $X_1, \dots, X_n$, from an unknown distribution but you know that distribution has a finite positive variance and finite mean.
So what can you say about the average, $\bar{X}$, of $X_1, \dots, X_n$? Well, $\bar{X}$ is a random variable itself and like any other random variable there’s a distribution that describes it. Because $\bar{X}$ is a random variable that’s a statistic, you call the distribution that describes it a sampling distribution. So imagine you take a random sample and calculate its mean, $\bar{X}$, and you repeat this process over and over so that you have a set of means, $\bar{X}_1, \dots, \bar{X}_n$.
Notice what happens when you take the value of these means and plot their percentage frequencies; an interesting shape starts to form which you’ll quickly identify to be reminiscent of a normal distribution. And if you do this experiment again but with even larger random sample sizes, you’ll notice that the plot of percentage frequencies looks even more like a normal distribution. Setosa has a great visual example of this experiment in action.
The expected value and standard deviation of the sample mean
Although you haven’t proved that the sampling distribution for $\bar{X}$ approaches a normal distribution, you’ve noticed it experimentally. What other questions can you ask about the statistic $\bar{X}$? What is its expected value? Well, you know that $\bar{X} = \frac{X_1 + \dots + X_n}{n}$. So it follows that
$$ \begin{aligned} \E(\bar{X}) &= \E\bigg(\frac{X_1 + \dots + X_n}{n}\bigg) \\\ &= \frac{1}{n} \E(X_1 + \dots + X_n) \\\ &= \frac{1}{n} \big(\E(X_1) + \dots + \E(X_n)\big) \\\ &= \frac{1}{n} (\mu + \dots + \mu) \\\ &= \frac{1}{n} \cdot n \mu \\\ &= \mu \end{aligned} $$
Notice the significance of this result; the expected value of the mean from a random sample is equal to the population mean that the random sample comes from.
Can you deduce the standard deviation of $\bar{X}$?
$$ \begin{aligned} \var(\bar{X}) &= \var\bigg(\frac{X_1 + \dots + X_n}{n}\bigg) \\\ &= \bigg(\frac{1}{n}\bigg)^2 \var(X_1 + \dots + X_n) \\\ &= \frac{1}{n^2} (\var(X_1) + \dots + \var(X_n)) \\\ &= \frac{1}{n^2} (\sigma^2 + \dots + \sigma^2) \\\ &= \frac{1}{n^2} \cdot n \sigma^2 \\\ &= \frac{\sigma^2}{n} \end{aligned} $$
Since the variance is $\frac{\sigma^2}{n}$, the standard deviation is $\frac{\sigma}{\sqrt{n}}$.
Forming the Central Limit Theorem
To recap, you’ve observed that the distribution of $\bar{X}$ seems to be normal and have proved that $\E(\bar{X}) = \mu$ and $\text{std}(\bar{X}) = \frac{\sigma}{\sqrt{n}}$. You know that an easier way to describe the standard normal distribution, $N(0, 1)$, is with its standard normal form (also sometimes called the standard score or z-score), $\frac{X - \mu}{\sigma}$.
This standard score is a random variable that indicates how many standard deviations a random variable, $X$, is from the mean. Since you suspect the distribution of $\bar{X}$ to be normal, then you might infer its standard normal form to be $\frac{\bar{X} - \mu}{\sigma / \sqrt{n}}$ since $\mu = \E(\bar{X})$ and $\frac{\sigma}{\sqrt{n}}$ is the standard deviation of $\bar{X}$.
Now all you need to do is prove that the sampling distribution for $\bar{X}$ approaches a standard normal distribution, $N(0, 1)$, as the sample size approaches infinity. Mathematically you want to prove:
$$ \frac{\bar{X} - \mu}{\sigma / \sqrt{n}} = \frac{\summation{i=1}{n} X_i - n \mu}{\sqrt{n} \sigma} \approaches N(0, 1) \text{ as } n \approaches \infinity $$
The relationship above is famously known as the Central Limit Theorem and indeed has been proven utilizing moment generating functions and a bit of calculus. Fortunately for you, we won’t go through that proof here, but you should relish the fact that your experimental results holds up against theoretical rigor.
At the end of the day, what does the Central Limit Theorem mean for you? Well, it allows you to use probabilistic and statistical methods that you already know work for the normal distribution for problems involving other types of distributions.
That random sample you took at the beginning of this article could have been from any distribution. Yet, the distribution of its mean is normal.
This theorem is indispensable because, among many reasons, it allows you to create confidence intervals for means and perform hypothesis tests - topics that will be covered in later posts.
But until then, I hope this guide has been a sufficient introduction into the Central Limit Theorem.