DS 5030 · Understanding Uncertainty · Go deeper · §2
Why averages settle, why their wobble is bell-shaped, and what happens when you transform them.
AoS Ch 5 uses three notions of convergence. The distinction matters because the LLN is a statement about values (convergence in probability) and the CLT is a statement about histograms (convergence in distribution).
| Name | Means | Used for |
|---|---|---|
| in quadratic mean | \(E(X_n - X)^2 \to 0\) | proving consistency via MSE → 0 |
| in probability | \(P(|X_n - X| > \epsilon) \to 0\) for every \(\epsilon\) | LLN, consistency |
| in distribution | \(P(X_n \le t) \to P(X \le t)\) at continuity points | CLT, approximate intervals |
Quadratic mean ⇒ probability ⇒ distribution (AoS Thm 5.4). None of the reverse arrows hold.
Let \(Y_n = n\) with probability \(1/n\), else 0.
The variable is almost always 0, but a rare enormous value keeps its mean at 1 forever. Heavy-tailed data behave like this, which is why the Cauchy distribution breaks averages.
Let \(X \sim N(0,1)\) and \(X_n = -X\) for every \(n\). Each \(X_n\) is also \(N(0,1)\), so \(X_n \rightsquigarrow X\) trivially. But \(|X_n - X| = 2|X|\) never shrinks. Same histogram, different values: that's all the CLT promises.
If \(X_1,\dots,X_n\) are iid with \(E|X_1| < \infty\), then \(\bar X_n \xrightarrow{P} \mu\). With finite variance the proof is Chebyshev: \(P(|\bar X_n - \mu| > \epsilon) \le \sigma^2/(n\epsilon^2)\).
Fair die, \(\sigma^2 = 35/12 = 2.92\), \(n = 100\), \(\epsilon = 0.5\). Chebyshev: \(P(|\bar X - 3.5| > 0.5) \le 2.92/25 = 0.117\). The CLT: \(\text{se} = 0.171\), so \(0.5/0.171 = 2.93\) SEs and \(P \approx 2\Phi(-2.93) = 0.003\). Chebyshev is about 35 times too pessimistic here. It's a guarantee for every distribution, so it has to cover the worst one.
Each observation is truth plus an error. The sum of \(n\) independent errors is a random walk: steps that agree pile up, steps that disagree cancel, and independent steps agree and disagree equally often. After \(n\) steps of typical size \(\sigma\), the walk is typically about \(\sigma\sqrt n\) from home (not \(\sigma n\), which is what you'd get if every step agreed). Divide the sum by \(n\) and the typical error of the mean is \(\sigma\sqrt n/n = \sigma/\sqrt n\).
So \(V(\bar X) = n\sigma^2/n^2 = \sigma^2/n\). Write out the \(3\times 3\) grid of covariances for \(n = 3\): three diagonal cells of \(\sigma^2\) and six off-diagonal cells that independence sets to zero. If every pair had correlation \(\rho\), the six cells would each be \(\rho\sigma^2\) and \(V(\bar X) = \frac{\sigma^2}{3}(1 + 2\rho)\). With \(\rho = 0.5\), that's double. This grid is the whole dependence failure in one picture.
iid, mean \(\mu\), variance \(\sigma^2 < \infty\): \(\sqrt n(\bar X_n - \mu)/\sigma \rightsquigarrow N(0,1)\), and also with \(S_n\) for \(\sigma\). Berry–Esseen: the largest gap between the CDF of the standardized mean and \(\Phi\) is at most a constant times \(E|X - \mu|^3/(\sigma^3\sqrt n)\).
The distribution of a sum of independent pieces is built by convolution: smear the current distribution by the shape of the next piece. Each smear averages away fine detail. After many smears only two numbers survive, the total center and the total spread, and the one shape that reproduces itself under smearing (normal plus normal is normal) is where everything ends up. Try "two bumps" with \(n = 1, 2, 3, 10\) in the widget: the bumps merge, then the hump rounds into a bell.
There is no universal \(n\). Berry–Esseen says the error scales with the third absolute moment over \(\sigma^3\), which is large for skewed or heavy-tailed data. Use the widget: uniform data look normal at \(n = 5\); Bernoulli(0.1) is still lumpy and skewed at \(n = 30\); car prices need around 100; Cauchy never converges, because there is no \(\sigma\) to standardize by.
The readout "P(|Z| > 1.96)" is the miss rate of a nominal 95% interval, and the two tails should each be 0.025. Car prices, with the true \(\sigma\):
| \(n\) | divide by \(\sigma\): low / high misses | divide by \(s\): low / high misses |
|---|---|---|
| 5 | 0.000 / 0.039 | 0.210 / 0.012 |
| 30 | 0.015 / 0.035 | 0.078 / 0.010 |
| 100 | 0.021 / 0.035 | 0.051 / 0.014 |
With the true \(\sigma\), the total is near 5% even at \(n = 5\), but the misses are all on the high side: skew in the data survives in the sampling distribution. Replace \(\sigma\) by \(s\) and the misses flip to the low side and multiply, because samples that miss the expensive cars have both a low mean and a small \(s\). Even at \(n = 100\) the low tail is twice its target. This is page 4's subject.
If \(\sqrt n(Y_n - \mu)/\sigma \rightsquigarrow N(0,1)\) and \(g'(\mu) \ne 0\), then \(g(Y_n) \approx N\big(g(\mu), g'(\mu)^2\sigma^2/n\big)\). In practice \(\widehat{\text{se}}(g(\hat\theta)) = |g'(\hat\theta)|\,\widehat{\text{se}}(\hat\theta)\).
| \(g\) | \(g'(5)\) | se of \(g(\bar x)\) |
|---|---|---|
| \(x^2\) | 10 | 10 |
| \(\sqrt x\) | 0.224 | 0.22 |
| \(\log x\) | 0.2 | 0.20 |
| \(1/x\) | −0.04 | 0.04 |
Then set the widget to \(g = x^2\), \(\mu = 5\), SE = 1: the simulation agrees with 10 to within a few percent. Raise SE to 2 and it doesn't: the tangent line is a bad stand-in for the curve over that wide a range.
\(\hat p = 0.6\), \(n = 100\). \(\hat\psi = \log(0.6/0.4) = 0.405\). \(g'(p) = 1/(p(1-p)) = 1/0.24 = 4.17\); \(\widehat{\text{se}}(\hat p) = \sqrt{0.24/100} = 0.049\). So \(\widehat{\text{se}}(\hat\psi) = 4.17 \times 0.049 = 0.204 = 1/\sqrt{n\hat p(1-\hat p)}\). A 95% interval for the log-odds is \(0.405 \pm 0.40 = (0.005, 0.805)\); exponentiate for the odds: \((1.01, 2.24)\). That's how logistic-regression intervals for odds ratios are made.
No. The population is unchanged; it's the distribution of averages of 200 cars that's close to normal. A single car's price is as skewed as ever.
Because \(s\) is itself random. Samples where \(s\) comes out small produce large \(|Z|\). That's the effect the t-distribution corrects (page 4). With skewed data the effect is lopsided: low means come with low \(s\), so the left tail stretches more.
The tangent is flat, so the first-order approximation says the SE is 0, which is false. The wobble of \(\bar X^2\) near 0 is second order: \(n\bar X^2/\sigma^2 \rightsquigarrow \chi^2_1\), not a normal. Set \(\mu\) to the slider's minimum with \(g = x^2\) and a large SE to see the approximation fail.