DS 5030 · Understanding Uncertainty · Go deeper · §7
Family-wise error, Bonferroni, Holm, false discovery rates, and Benjamini–Hochberg.
With \(m\) independent true nulls each tested at \(\alpha\), \(P(\text{at least one false rejection}) = 1 - (1 - \alpha)^m\).
| \(m\) | 1 | 5 | 10 | 20 | 100 |
|---|---|---|---|---|---|
| \(1 - 0.95^m\) | 0.05 | 0.23 | 0.40 | 0.64 | 0.994 |
| expected false rejections \(0.05m\) | 0.05 | 0.25 | 0.5 | 1 | 5 |
CASI §15.1: "how easy it is to be fooled." Leukemia gene 136 had \(p = 0.0036\), but 400 of 7128 genes beat its t-statistic.
One ticket at 1-in-20 odds rarely wins. Buy twenty and you probably hold a winner. A "significant" result pulled from many tests is a winning ticket from a stack: it says more about the size of the stack than about the ticket.
Bonferroni: reject \(H_{0i}\) if \(p_i < \alpha/m\). FWER \(\le \alpha\) by the union bound, with no independence needed. Holm: sort the p-values; reject \(p_{(1)}, p_{(2)}, \dots\) while \(p_{(i)} \le \alpha/(m - i + 1)\); stop at the first failure. Also FWER \(\le \alpha\), and never rejects fewer than Bonferroni.
Say \(m_0\) of the nulls are true. Walk down the sorted list until the first true null. Everything before it was a false null, so at most \(m - m_0\) hypotheses came first, and this one is at rank \(i \le m - m_0 + 1\). To be rejected it must pass \(\alpha/(m - i + 1) \le \alpha/m_0\). So a false rejection requires the smallest true-null p-value to be below \(\alpha/m_0\), which by the union bound over the \(m_0\) true nulls has probability at most \(m_0 \cdot \alpha/m_0 = \alpha\). Holm earns its extra power because, once false nulls are cleared away, the later thresholds can afford to be looser.
0.001, 0.012, 0.020, 0.040, 0.300; \(\alpha = 0.05\).
| \(i\) | \(p_{(i)}\) | Bonferroni 0.01 | Holm \(0.05/(6-i)\) | BH \(0.01i\) |
|---|---|---|---|---|
| 1 | 0.001 | ✓ | 0.0100 ✓ | 0.01 ✓ |
| 2 | 0.012 | ✗ | 0.0125 ✓ | 0.02 ✓ |
| 3 | 0.020 | ✗ | 0.0167 ✗ stop | 0.03 ✓ |
| 4 | 0.040 | ✗ | — | 0.04 ✓ |
| 5 | 0.300 | ✗ | — | 0.05 ✗ |
Rejections: Bonferroni 1, Holm 2, BH 4, uncorrected 4.
Two-sided Bonferroni at \(\alpha = 0.05\) requires \(|z| > \Phi^{-1}(1 - 0.025/m)\): \(m = 1 \to 1.96\); \(m = 10 \to 2.81\); \(m = 100 \to 3.48\); \(m = 7128 \to 4.49\). CASI's prostate example uses one-sided 0.05/6033, giving 4.31: four genes pass.
FDP = false rejections / rejections (0 if none); FDR = \(E[\text{FDP}]\). BH: find the largest \(i\) with \(p_{(i)} \le i\alpha/m\) and reject the \(i\) smallest. With independent p-values, FDR \(= \pi_0\alpha \le \alpha\), where \(\pi_0\) is the share of true nulls.
Suppose you reject everything with \(p \le t\). True nulls have uniform p-values, so about \(m_0 t \le m t\) of them fall below \(t\): that's the expected number of false discoveries. If \(R\) p-values in total fall below \(t\), the false-discovery share is at most about \(mt/R\). Setting \(mt/R = \alpha\) gives \(t = R\alpha/m\): the BH line. BH picks the largest threshold where the observed count of small p-values justifies it.
0.00017, 0.00448, 0.00671, 0.00907, 0.01220, 0.33626, … BH line \(0.005i\): the first five are below (0.005, 0.010, 0.015, 0.020, 0.025); the sixth (0.336 vs 0.030) is not. BH rejects 5; Bonferroni (0.005) rejects 2.
Uncorrected rejects about \(0.05m\) (all false). Bonferroni and BH usually reject none: when every null is true, FDR control and FWER control coincide, since any rejection makes FDP = 1.
Look at its FDR: with 90 nulls and 10 real effects, about 4.5 false rejections per run against roughly 9 true ones, so a third of its "discoveries" are false. BH keeps that share near 5% while keeping most of the power; Bonferroni keeps even one false discovery rare, at a large cost in power.
The basic version is valid for independent and positively dependent p-values. AoS's \(C_m = \sum_{j=1}^m 1/j\) factor (divide \(\alpha\) by it) makes it valid under any dependence, at a cost of about a factor of \(\log m\) in the threshold. Bonferroni never needs independence.