DS 5030 · Understanding Uncertainty · Go deeper · §9
Dependence, peeking, heavy tails, and the other ways an interval quietly lies.
Every interval is \(\hat\theta \pm c\cdot\widehat{\text{se}}\), and every test compares \(\hat\theta/\widehat{\text{se}}\) to \(c\). A failure breaks one of three things: the SE is wrong (dependence, unequal variance, no finite variance), the critical value \(c\) is wrong (small samples, skew, many tests, peeking), or the result is read wrong (coverage read as belief, significance read as importance). This page puts the first two kinds in widgets.
\(V(\bar X) = \sigma^2/n\) used independence to drop every covariance term. With positive correlation those terms are positive, and \(s/\sqrt n\) is too small. For a series where each value is \(\rho\) times the previous plus fresh noise (AR(1)), the variance of the mean is roughly \(\frac{\sigma^2}{n}\cdot\frac{1+\rho}{1-\rho}\) for large \(n\).
With \(\rho = 0.7\): \((1 + 0.7)/(1 - 0.7) = 5.67\). So the real variance of the mean is 5.67 times what \(s/\sqrt n\) assumes, and the real SE is \(\sqrt{5.67} = 2.4\) times larger. 100 correlated observations carry about as much information as \(100/5.67 \approx 18\) independent ones. A "95%" interval built with the naive SE is really \(\pm 1.96/2.4 = \pm 0.82\) true SEs, covering about 59%. (The course simulation: true SD 0.33, naive 0.14, coverage 59%.)
Twins: 50 pairs, each pair's two measurements correlated 0.5. \(V(\bar X) = \frac{\sigma^2}{100}(1 + \rho) = 1.5\,\sigma^2/100\). The naive SE is too small by a factor \(\sqrt{1.5} = 1.22\). Treating 100 correlated people as 100 independent ones overstates the evidence; the effective sample size is 67.
If you test after every batch of data and stop the first time \(p < 0.05\), the overall false-positive rate climbs with every look (CASI §3.3).
One-sided at 1.645, looks at months 20 and 30. If the looks were independent, the false-positive rate would be \(1 - 0.95^2 = 0.0975\). They're positively correlated (month 30 includes months 1–20, correlation \(\sqrt{20/30} = 0.82\)), so it's less: 0.074. Still half again over the promised 0.05.
"Average price $8.59k ± $8.17k" (SD) vs "± $0.85k" (SE). The first says individual cars vary; the second says the average is pinned down. As a 95% statement about the mean, the first is 9.6 times too wide.
10 estimates, each \(N(\mu_i, 1)\), all with true \(\mu_i = 0\). The largest of 10 standard normals averages about 1.54. Report "our best result was 1.5 SEs above zero" and you've reported noise: selecting the max biases it upward. CASI's prostate example: the top gene's 5.29 shrinks to 4.11 under empirical Bayes.
\(n = 1{,}000{,}000\), \(s = 10\): \(\widehat{\text{se}} = 0.01\). An effect of 0.03 has \(z = 3\), \(p = 0.003\), and a 95% interval of \((0.01, 0.05)\): very significant and, on a scale with SD 10, practically nothing. The interval shows that; the p-value hides it.
\(n = 9\), \(s = 6\), \(\widehat{\text{se}} = 2\). A true effect of 2 is 1 SE: power about 0.17 (page 6). So \(p = 0.4\) here is what you'd see most of the time even if the effect is real. The 95% interval, about \(\pm 4.6\), says so honestly.
| failure | breaks | where it's worked |
|---|---|---|
| unequal error variance | SE | page 8 widget |
| small n with z | c | page 4 coverage race |
| skewed data | c | page 4 scatter, page 2 |
| jackknife on a median | SE | page 3 widget |
| bootstrap of a maximum | SE | page 3 widget |
| many tests | c | page 7 widget |
| coverage read as belief | reading | page 5 Berger–Wolpert |
It goes above 95%: with negative correlation, consecutive values tend to cancel, the true variance of the mean is smaller than \(\sigma^2/n\), and the naive SE is too large. Rare in practice, but it shows the formula's error has a sign set by the correlation.
The looks are positively correlated: each later test includes all the earlier data. If look 3 wasn't significant, look 4 shares most of its data and probably isn't either. Correlation makes repeated chances less independent, so the inflation is real but smaller than for independent tests.
The variance is finite but the fourth moment isn't (for \(k \le 4\)), so \(s^2\) itself has infinite variance and converges very slowly. The CLT for the mean still holds, but SE estimates are unreliable at practical sample sizes. "Finite variance" is the minimum for the theory, not a guarantee of good behavior.