DS 5030 · Understanding Uncertainty · Go deeper · §3
Plug-in, Taylor series, Fisher information, the bootstrap, the jackknife, and pivots, each worked by hand and live.
Every standard error is the SD of an estimator's sampling distribution, and that SD depends on the unknown population \(F\). CASI §2.1 lists the devices for getting around this. The useful way to compare them is to ask when each one swaps \(F\) for something computable.
| Device | Theory first, then plug in? | Works for any statistic? | Cost |
|---|---|---|---|
| Plug-in formula | yes | no: needs a formula | none |
| Taylor / delta | yes | smooth functions of averages | a derivative |
| Fisher information | yes | MLEs in a parametric model | a second derivative |
| Jackknife | numerical derivatives | smooth statistics | \(n\) recomputations |
| Bootstrap | no: plug in first, then simulate | almost any | \(B\) recomputations |
| Pivot | nothing to plug in | rare special cases | none |
The empirical CDF \(\hat F_n\) is the sample treated as a population: probability \(1/n\) on each point. Every plug-in estimate is "compute the population quantity, but on \(\hat F_n\)." Glivenko–Cantelli says \(\hat F_n\) converges to \(F\) uniformly; DKW says how fast, with a band of half-width \(\epsilon_n = \sqrt{\log(2/\alpha)/(2n)}\).
| quantity | plug-in | SE (plug-in) |
|---|---|---|
| mean | 5 | \(\hat\sigma/\sqrt n = 2/\sqrt5 = 0.89\) (or \(s/\sqrt n = 1\)) |
| \(F(5) = P(X \le 5)\) | 3/5 = 0.6 | \(\sqrt{0.6(0.4)/5} = 0.22\) |
| \(P(X > 5)\) | 0.4 | 0.22 |
| \(E[X^2]\) | 29 | SD of \(\{4, 16, 25, 36, 64\}\) is 20.4 (plug-in), so \(20.4/\sqrt5 = 9.1\) |
| variance | 4 | no simple formula: bootstrap it |
Linear functionals (\(\int r\,dF\)) are averages of \(r(x_i)\), so their SE is the SD of the \(r(x_i)\) over \(\sqrt n\). Everything else needs a delta method or a bootstrap.
Covered in depth on page 2. The one-line summary: a smooth function of an average wobbles by the slope times the average's wobble. CASI's historical point is that before computers, every new statistic needed its own Taylor expansion; the SE of a correlation coefficient takes a page of fourth moments (CASI (10.10)), and the jackknife and bootstrap made that expertise unnecessary.
Patch data style (AoS Ex 8.7): \(\hat\theta = \bar Y/\bar Z\). Delta method with two variables, \(g(y, z) = y/z\), gradient \((1/z, -y/z^2)\):
$$\widehat{\text{se}}^2(\hat\theta) \approx \frac{1}{\bar z^2}\Big(\widehat{\text{se}}^2(\bar y) - 2\hat\theta\,\widehat{\text{cov}}(\bar y, \bar z) + \hat\theta^2\,\widehat{\text{se}}^2(\bar z)\Big).$$Tiny numbers: \(\bar y = 2\), \(\bar z = 4\), so \(\hat\theta = 0.5\); with \(\widehat{\text{se}}(\bar y) = 0.4\), \(\widehat{\text{se}}(\bar z) = 0.8\), and independent \(\bar y, \bar z\): \(\widehat{\text{se}}^2 = \frac{1}{16}(0.16 + 0.25 \times 0.64) = \frac{0.32}{16} = 0.02\), so \(\widehat{\text{se}} = 0.14\). AoS uses the bootstrap instead (0.105 on the real data) precisely to avoid this algebra.
For a parametric model, the MLE's SE comes from the curvature of the log-likelihood at its peak: sharp peak, small SE. The widget shows a Bernoulli log-likelihood. As \(n\) grows with the same success rate, the curve gets sharper and the SE shrinks like \(1/\sqrt n\).
Ten distinct resamples (probabilities out of 27): means 1, 3, 5 (1 each); 5/3, 7/3, 7/3, 11/3, 11/3, 13/3 (3 each); 3 (6). Bootstrap variance of the mean \(= 8/9\), SE \(= 0.94 = \hat\sigma/\sqrt n\). Medians: 1 w.p. 7/27, 3 w.p. 13/27, 5 w.p. 7/27, SE \(= \sqrt{56/27} = 1.44\). Set the widget to {1, 3, 5} and run 1,000 resamples: it should land near 0.94 and 1.44.
With \(B\) resamples the bootstrap SE is itself an SD estimated from \(B\) draws, so its own relative noise is about \(1/\sqrt{2(B-1)}\) (page 4's formula, applied to the simulation). \(B = 200\) gives 5%, \(B = 1000\) gives 2%. Compare that with the statistical uncertainty in the SE from having only \(n\) data points (11% for the 92 cars). Past a few hundred resamples, more \(B\) buys almost nothing.
Mean of {2, 4, 5, 6, 8}: leave-one-out means 5.75, 5.25, 5, 4.75, 4.25; squared deviations sum 1.25; \(\frac45 \times 1.25 = 1\); SE 1 \(= s/\sqrt n\).
Median of {1, 2, 3, 4, 5}: leave-one-out medians 3.5, 3.5, 3, 2.5, 2.5; squared deviations sum 1.0; \(\frac45 \times 1.0 = 0.8\); SE 0.89. Only three distinct values: the median jumps from one data point to the next as points are removed, which is the non-smoothness that breaks the jackknife on larger samples (car prices: jackknife 1.43 vs bootstrap 0.75).
A pivot's distribution doesn't depend on unknown parameters, so no plugging in is needed. The t-statistic is the classic: \(\sqrt n(\bar X - \mu)/S \sim t_{n-1}\) for normal data whatever \(\sigma\) is. Others: \((n-1)S^2/\sigma^2 \sim \chi^2_{n-1}\) (gives an exact interval for \(\sigma^2\)); for exponential data, \(2\lambda\sum X_i \sim \chi^2_{2n}\).
Toy data, \(s^2 = 5\), \(n = 5\). \(4s^2/\sigma^2 = 20/\sigma^2 \sim \chi^2_4\), whose 2.5% and 97.5% points are 0.484 and 11.14. So \(0.484 \le 20/\sigma^2 \le 11.14\), i.e. \(1.80 \le \sigma^2 \le 41.3\). A 95% interval for the variance runs over a factor of 23: five points say almost nothing about spread.
Both are estimates of the same thing. The bootstrap is a plug-in method, so it reproduces \(\hat\sigma/\sqrt n\) with divisor \(n\): \(2/\sqrt5 = 0.89\). The textbook \(s/\sqrt n = 1\) uses divisor \(n - 1\). At \(n = 5\) the difference is 11%; at \(n = 92\) it's 0.5%.
The Wald interval is symmetric by construction (\(\hat p \pm 1.96\,\widehat{\text{se}}\)), but the likelihood is lopsided near the boundary. The quadratic approximation is the delta-method idea applied to the log-likelihood, and it fails where the curve isn't close to a parabola: small \(n\) or \(p\) near 0 or 1.
Pooling and resampling 72 patients would make the group sizes random (sometimes 30 AML, sometimes 20), adding variability the real study didn't have, since the design fixed the group sizes. Resample within each group to mimic the design (CASI §10.3).