DS 5030 · Understanding Uncertainty · Go deeper · §1
What the ± after an estimate is measuring, worked out with samples small enough to list.
An estimator is a rule written down before the data arrive: "add the numbers and divide by \(n\)," "sort them and take the middle one." Applied to one dataset, the rule produces one number, the estimate. Applied to every dataset the world could have handed you, it produces a distribution of numbers: the sampling distribution.
The frequentist move (CASI Ch 2) is to grade the number you have by the behavior of the recipe that produced it. If the recipe usually lands within 0.85 of the truth, you report your number as "± 0.85." Nothing about your particular number is known to be good or bad; only the recipe has a track record.
Estimate \(\hat\theta = t(x)\) (a number); estimator \(\hat\Theta = t(X)\) (a random variable). With \(\mu = E_F[\hat\Theta]\): \(\text{bias} = \mu - \theta\), \(\text{var} = E_F[(\hat\Theta - \mu)^2]\), \(\text{se} = \sqrt{\text{var}}\), \(\text{MSE} = E_F[(\hat\Theta - \theta)^2] = \text{bias}^2 + \text{var}\).
Three things vary in a sampling distribution, and it helps to separate them: the population \(F\) (fixed, unknown), the sample (random, one draw of \(n\) values), and the estimate (random, because it's computed from the sample). If you blur the first two, you will later be tempted to think the CLT makes the data normal.
With a tiny population and tiny samples, there are few enough possible samples to write every one down. Then nothing is approximate: the sampling distribution, its mean, its SE, and the bias of any estimator are exact fractions.
\(\mu = 3\), \(\sigma^2 = 8/3\). The 9 equally likely samples give \(\bar x = 1, 2, 3, 2, 3, 4, 3, 4, 5\). So \(\bar X\) takes values 1, 2, 3, 4, 5 with probabilities \(\tfrac19, \tfrac29, \tfrac39, \tfrac29, \tfrac19\): mean 3, variance \(\tfrac{12}{9} = \tfrac43 = \sigma^2/2\), SE 1.15.
Now 27 samples. The sum of three draws counts like the coefficients of \((1 + z + z^2)^3 = 1 + 3z + 6z^2 + 7z^3 + 6z^4 + 3z^5 + z^6\), so
| \(\bar x\) | 1 | 5/3 | 7/3 | 3 | 11/3 | 13/3 | 5 |
|---|---|---|---|---|---|---|---|
| probability (/27) | 1 | 3 | 6 | 7 | 6 | 3 | 1 |
Variance: squared distances from 3 are \(4, \tfrac{16}{9}, \tfrac49, 0, \tfrac49, \tfrac{16}{9}, 4\); weighted sum \(\big(8 + \tfrac{96}{9} + \tfrac{48}{9}\big)/27 = \tfrac{24}{27} = \tfrac89 = \sigma^2/3\). ✓ Going from \(n = 2\) to \(n = 3\), the distribution got narrower and more bell-shaped (1, 3, 6, 7, 6, 3, 1 is already a hump), even though the population is three flat spikes.
MSE splits into bias² plus variance (AoS Thm 6.9). The split matters because the two parts trade off: shrinking an estimator toward zero adds bias but can remove more variance than it adds. The cleanest example is the divisor in a variance estimator, \(\sum(x_i - \bar x)^2/d\).
Every miss \(\hat\theta - \theta\) is the sum of a fixed leg (from \(\theta\) to the estimator's average, the bias) and a random leg (from the average to this estimate). The random leg averages to zero, so on average it's "perpendicular" to the fixed leg: the cross term \(2\,\text{bias}\cdot E[\hat\theta - E\hat\theta]\) vanishes, and the squared miss averages to bias² + variance.
Let \(SS = (x_1 - \bar x)^2 + (x_2 - \bar x)^2 = (x_1 - x_2)^2/2\), which is 0, 2, or 8 with probabilities \(\tfrac39, \tfrac49, \tfrac29\). So \(E[SS] = \tfrac{8 + 16}{9} = \tfrac83\) and \(E[SS^2] = \tfrac{4\cdot 4 + 2\cdot 64}{9} = 16\). For divisor \(d\): \(\text{MSE}(d) = E[SS^2]/d^2 - 2\sigma^2E[SS]/d + \sigma^4 = \tfrac{16}{d^2} - \tfrac{128}{9d} + \tfrac{64}{9}\).
| divisor \(d\) | 1 (unbiased, \(s^2\)) | 2 (plug-in, \(\hat\sigma^2\)) | 2.25 (best) | 3 |
|---|---|---|---|---|
| MSE | 8.89 | 4.00 | 3.95 | 4.15 |
Minimizing \(16/d^2 - 128/(9d)\): setting the derivative \(-32/d^3 + 128/(9d^2)\) to zero gives \(d = 32 \cdot 9/128 = 2.25\). For normal data the MSE-best divisor is \(n + 1\); here the population is flat, so it's a little different. The unbiased choice is the worst of the four.
A consistent estimator gets close to the truth with high probability as \(n\) grows (AoS Def 6.7). Sufficient condition: bias → 0 and se → 0 (Thm 6.10). Unbiased estimators need not be consistent, and consistent estimators need not be unbiased.
| Setting | SD (of the data) | SE (of the estimate) |
|---|---|---|
| Toy data \(\{2, 4, 5, 6, 8\}\) | \(s = 2.24\) | \(2.24/\sqrt5 = 1.00\) |
| 92 car prices | $8.17k | $0.85k |
| CASI gfr, 211 patients | ≈ 13.8 | 0.95 |
| A poll, \(\hat p = 0.6\), \(n = 100\) | \(\sqrt{0.6 \times 0.4} = 0.49\) | 0.049 |
| Same poll, \(n = 10{,}000\) | 0.49 (unchanged) | 0.0049 |
The SD is how much people's weights differ. The SE is how much the reading of a room's average weight would wobble if you redid the weigh-in with a different room. A bigger room doesn't make people more alike; it makes the average steadier.
It shows the population and one sample. Neither depends on how many studies you run, and the spread of a sample estimates the population SD, which is fixed. Only the bottom panel, the spread of averages, shrinks like \(1/\sqrt n\).
Yes. Unbiasedness is about the center (its mean is exactly \(\mu\)), not the shape. Skew in the sampling distribution means the typical study lands below \(\mu\) (median below mean) and a few land far above.
With \(n = 2\) the median equals the mean, so it's also \(\sqrt{4/3} = 1.15\). With \(n = 3\) they differ: the median takes values 1, 3, 5 with probabilities 7/27, 13/27, 7/27 (same counting as the bootstrap example in the guide), so its SE is \(\sqrt{56/27} = 1.44\), larger than the mean's 0.94.