DS 5030 · Understanding Uncertainty · Go deeper · §4
How noisy an estimated SE is, what that does to intervals, and how the t-distribution fixes it.
\(\widehat{\text{se}} = s/\sqrt n\) is computed from the same \(n\) numbers as \(\bar x\). Draw a different sample and you get a different \(s\), so a different SE. CASI's first chapter notes this regress: the accuracy estimate is "itself an algorithm" with its own accuracy. In practice the SE's own uncertainty matters a lot at small \(n\) and for skewed data, and almost not at all for large, well-behaved samples.
Measuring "how many SEs away" with an estimated SE is measuring with a ruler you also had to estimate. A steel ruler (large \(n\)) gives the same length every time. A rubber ruler (small \(n\)) stretches and shrinks from sample to sample, and every interval inherits the stretch.
\((n-1)S^2/\sigma^2 \sim \chi^2_{n-1}\), independent of \(\bar X\); \(V(S^2) = 2\sigma^4/(n-1)\). By the delta method on \(\sqrt{\cdot}\), \(\text{sd}(S) \approx \sigma/\sqrt{2(n-1)}\), so the relative noise in \(\widehat{\text{se}}\) is about \(1/\sqrt{2(n-1)}\).
\(g(v) = \sqrt v\), \(g'(v) = 1/(2\sqrt v)\). At \(v = \sigma^2\): \(\text{sd}(S) \approx \frac{1}{2\sigma}\,\text{sd}(S^2) = \frac{1}{2\sigma}\cdot\sigma^2\sqrt{\frac{2}{n-1}} = \frac{\sigma}{\sqrt{2(n-1)}}\). Divide by \(\sigma\) for the relative noise. Then the table: \(n = 5 \to 1/\sqrt8 = 0.35\); \(n = 10 \to 0.24\); \(n = 30 \to 0.13\); \(n = 92 \to 0.07\). For the toy data (\(n = 5\), \(\widehat{\text{se}} = 1\)), a true SE of 0.65 or 1.35 would be unsurprising.
Resample \(\{2, 2, 5, 8, 8\}\): \(s^2 = (9+9+0+9+9)/4 = 9\), \(\widehat{\text{se}} = 3/\sqrt5 = 1.34\). Resample \(\{4, 5, 5, 6, 8\}\): mean 5.6, \(s^2 = (2.56 + 0.36 + 0.36 + 0.16 + 5.76)/4 = 2.3\), \(\widehat{\text{se}} = 0.68\). Same population, SEs differing by a factor of two.
If the SE comes out 30% too small, a "±1.96 SE" interval is really ±1.37 true SEs and covers 83%. If it comes out 30% too big, the interval is ±2.55 true SEs and covers 99%. Losing 12 points one way but gaining only 4 the other means a symmetric wobble in the SE produces a net loss in coverage. Equivalently, \(Z = (\bar X - \mu)/\widehat{\text{se}}\) has heavier tails than \(N(0,1)\).
If the SE estimate is \(c\) times the truth, the interval \(\bar x \pm 1.96\,\widehat{\text{se}}\) covers with probability \(2\Phi(1.96c) - 1\).
| \(c\) | 0.5 | 0.7 | 0.85 | 1 | 1.15 | 1.3 | 1.5 |
|---|---|---|---|---|---|---|---|
| coverage | 67% | 83% | 90% | 95% | 98% | 99% | 99.7% |
Average coverage over a symmetric spread of \(c\) values is below 95%: the curve bends down faster on the left than it rises on the right.
\(T = Z/\sqrt{V/k}\) with \(Z \sim N(0,1)\) and \(V \sim \chi^2_k\) independent. Conditional on \(V\), \(T\) is normal with SD \(\sqrt{k/V}\). So \(T\) is a mixture of bells of different widths, weighted by how likely each width is. Mixing narrow and wide bells gives a sharper peak and fatter tails. As \(k\) grows, \(V/k \to 1\) and every bell has width 1.
All three have \(t\) near 2 and all three fail to reject: at these sample sizes, "2 standard errors" isn't enough evidence once you admit the SE is estimated.
The t-interval assumes normal data. The bootstrap-t keeps the idea of dividing by an estimated SE, but learns the distribution of \(t^* = (\bar x^* - \bar x)/\widehat{\text{se}}^*\) by resampling (CASI (11.52)–(11.53)). Each resample brings its own SE, so the noise in the SE is built into \(t^*\), including its lopsidedness for skewed data.
From \(\{2, 4, 5, 6, 8\}\): \(\{2,2,5,8,8\}\) gives \(t^* = 0/1.34 = 0\); \(\{4,5,5,6,8\}\) gives \(t^* = 0.6/0.68 = 0.89\); \(\{2,4,4,5,6\}\) gives \(t^* = -0.8/0.66 = -1.21\). With thousands of these, take the 2.5% and 97.5% points \(t^{*(.025)}, t^{*(.975)}\); the interval is \(\big(\bar x - \widehat{\text{se}}\,t^{*(.975)},\ \bar x - \widehat{\text{se}}\,t^{*(.025)}\big)\). Note the flip, as with the pivotal interval.
CASI's example: AML − ALL means, Student's interval \((0.062, 0.314)\), bootstrap-t \((0.072, 0.323)\). Close, which supported the normality assumption in that case.
Lower left means a low mean and a small SE. Those are samples that happened to miss the few expensive cars: without them, both the average and the spread come out small. A large miss divided by a small SE gives a very negative \(Z\), so the interval misses on the low side. For normal data the two are independent and the red points spread evenly.
It's being accurate. For normal data, \(t_{n-1}\) is the exact distribution of the standardized mean, so the t-interval has exactly 95% coverage and the z-interval has less. Look at the race widget with normal data: t sits on the line at every \(n\).
It's what makes \(T = Z/\sqrt{V/k}\) a ratio of independent pieces, which is the definition of the t-distribution. For skewed data \(\bar X\) and \(S\) are positively correlated, the ratio isn't t-distributed, and its tails are lopsided (the scatter widget's red cluster).