DS 5030 · Understanding Uncertainty · Go deeper · §5
What the 95% belongs to, and how normal, t, and bootstrap intervals are built.
A confidence interval is a pair of statistics \((a(X), b(X))\) chosen so that \(P_\theta(a \le \theta \le b) \ge 1 - \alpha\) for every \(\theta\) (AoS (6.9)). The probability is over the data, with \(\theta\) held fixed. Once you've computed \((6.92, 10.26)\), nothing random is left: the interval contains \(\mu\) or it doesn't.
The peg is bolted down. Your throwing arm lands the ring around the peg 95% of the time. After a throw you can see the ring, but not the peg, so you can't tell whether this throw was one of the 95%. The 95% describes the arm.
Wasserman's reading (AoS §6.3.2): over a career of unrelated studies, each with its own 95% interval, 95% of the intervals trap their own true values. CASI prefers hypothetical repetitions of this problem. Both are statements about the procedure.
The nine sample means are 1, 2, 3, 2, 3, 4, 3, 4, 5 and \(\mu = 3\). The rule "\(\bar x \pm h\)" covers whenever \(|\bar x - 3| \le h\).
| half-width \(h\) | covering samples | coverage |
|---|---|---|
| 0.5 | \(\bar x = 3\): 3 samples | 33% |
| 1.5 | \(\bar x \in \{2,3,4\}\): 7 | 78% |
| 2.5 | all 9 | 100% |
No interval rule here hits exactly 95%: with discrete data, coverage jumps. That's why exact intervals for counts are conservative (coverage "at least" \(1-\alpha\)).
Two noisy readings \(Y_i = \theta \pm 1\) (each sign with probability 1/2). Report \(\{Y_1 - 1\}\) if the readings agree, \(\{(Y_1 + Y_2)/2\}\) if they differ. Four equally likely sign patterns: \((+,+)\) gives \(\theta\) ✓; \((-,-)\) gives \(\theta - 2\) ✗; \((+,-)\) and \((-,+)\) give \(\theta\) ✓. Coverage 3/4. Now observe 15 and 17: the readings differ, so the report is \(\{16\}\), and \(\theta\) must be 16. A "75% interval" that is certainly right. Coverage is about the rule, not the realized interval.
If \(\hat\theta \approx N(\theta, \widehat{\text{se}}^2)\), then \(\hat\theta \pm z_{\alpha/2}\widehat{\text{se}}\) has coverage \(\to 1 - \alpha\). Proof: \(P(|\hat\theta - \theta|/\widehat{\text{se}} < z_{\alpha/2}) \to P(|Z| < z_{\alpha/2}) = 1 - \alpha\).
| problem | estimate | \(\widehat{\text{se}}\) | 95% interval |
|---|---|---|---|
| toy data mean | 5 | 1 | (3.04, 6.96); t₄: (2.22, 7.78) |
| proportion: 30 of 100 | 0.30 | \(\sqrt{0.21/100} = 0.046\) | (0.21, 0.39) |
| difference: \(\bar x_1 = 10\), \(\bar x_2 = 7\), SEs 1.2 and 0.9 | 3 | \(\sqrt{1.44 + 0.81} = 1.5\) | (0.06, 5.94) |
| Poisson count 16 | 16 | \(\sqrt{16} = 4\) | (8.2, 23.8) |
| log-odds, \(\hat p = 0.6\), \(n = 100\) | 0.405 | 0.204 | (0.005, 0.805) |
The difference row is the easiest one to get wrong: SEs of independent estimates add in squares, so the SE of the difference is 1.5, not \(1.2 + 0.9 = 2.1\).
Worst case \(p = 0.5\): half-width \(1.96\sqrt{0.25/n} = 0.03\) gives \(\sqrt n = 1.96 \times 0.5/0.03 = 32.7\), so \(n \approx 1{,}068\). Halving the margin to ±1.5 points needs four times as many: about 4,270.
\(\hat\theta = 5\); replicates 3.6, 4.0, 4.4, 4.6, 4.8, 5.0, 5.2, 5.6, 6.2, 7.0 (SD 1.02); take the 10% and 90% quantiles as 4.0 and 6.2. 80% intervals: normal \(5 \pm 1.28(1.02) = (3.69, 6.31)\); percentile \((4.0, 6.2)\); pivotal \((10 - 6.2, 10 - 4.0) = (3.8, 6.0)\).
The bootstrap world's miss \(\hat\theta^* - \hat\theta\) imitates the real world's miss \(\hat\theta - \theta\). If replicates often overshoot \(\hat\theta\) by a lot (long right tail), then \(\hat\theta\) probably overshoots \(\theta\) by a lot, so \(\theta\) sits further left. The interval stretches left: \(\hat\theta - (\theta^*_{.975} - \hat\theta) = 2\hat\theta - \theta^*_{.975}\).
CASI Ch 11's ladder: standard → percentile (handles skew via a hidden transformation) → BC (also median bias, via \(z_0\)) → BCa (also SE that changes with \(\theta\), via the acceleration \(a\)). Standard intervals have coverage error of order \(1/\sqrt n\); BCa of order \(1/n\).
\(p_0\) = share of replicates below \(\hat\theta\); \(z_0 = \Phi^{-1}(p_0)\). The BC interval uses percentiles \(\Phi(2z_0 \pm z_{\alpha/2})\) instead of \(\alpha/2\) and \(1 - \alpha/2\).
| \(p_0\) | \(z_0\) | 95% BC uses percentiles |
|---|---|---|
| 0.50 | 0 | 2.5%, 97.5% (= percentile interval) |
| 0.45 | −0.126 | \(\Phi(-2.212) = 1.3\%\), \(\Phi(1.708) = 95.6\%\) |
| 0.40 | −0.253 | \(\Phi(-2.466) = 0.7\%\), \(\Phi(1.454) = 92.7\%\) |
| 0.55 | 0.126 | \(\Phi(-1.708) = 4.4\%\), \(\Phi(2.212) = 98.7\%\) |
When most replicates sit above \(\hat\theta\) (\(p_0 < 0.5\)), the estimate looks biased upward relative to its bootstrap distribution, and BC slides both ends down.
One Poisson count of 10. Standard: \(10 \pm 1.96\sqrt{10} = (3.8, 16.2)\). Exact: \((5.1, 17.8)\). The Poisson's spread grows with its mean, so plausible values above 10 range further than plausible values below. A square-root transformation nearly fixes it: \(\sqrt{10} \pm 1.96/2 = (2.18, 4.14)\), squared: \((4.8, 17.2)\). The percentile method does this transformation for you without being told.
Mostly ones entirely below \(\mu\). Small samples of car prices usually miss the few expensive cars, giving a low mean and a small SE together, so the interval is short and sits too low. Increase \(n\) and the misses even out.
Not in the frequentist sense: each is either right or wrong. What's higher is the long-run hit rate of the 99% procedure. You pay for it with width: \(2.576/1.96 = 1.31\) times wider.
When the bootstrap distribution is symmetric and close to normal: then its quantiles are \(\hat\theta \pm z\cdot\widehat{\text{se}}_{\text{boot}}\). In the widget, the mean of all 92 cars nearly agrees; the SD of the first 15 cars doesn't.