DS 5030 · Understanding Uncertainty · Go deeper · §6
Size, power, the Wald test, p-values, permutation tests, and Neyman–Pearson, with power you can drag.
A test fixes a rejection region \(R\) in advance and rejects \(H_0\) when the data land in it. Its power function \(\beta(\theta) = P_\theta(X \in R)\) says how often that happens for each possible truth (AoS Def 10.1). The size is the largest rejection probability over \(\theta\) in the null.
| reject if the number of heads is | size (\(p = 0.5\)) | power at \(p = 0.8\) |
|---|---|---|
| 0 or 5 | \(2/32 = 0.0625\) | \(0.8^5 + 0.2^5 = 0.328\) |
| 5 (one-sided) | \(1/32 = 0.031\) | \(0.8^5 = 0.328\) |
| 4 or 5 (one-sided) | \(6/32 = 0.19\) | \(5(0.8)^4(0.2) + 0.8^5 = 0.737\) |
The one-sided rule "5 heads" has the same power at \(p = 0.8\) as the two-sided "0 or 5," at half the size, because 0 heads is almost impossible when \(p = 0.8\). Bigger regions buy power with size. Nothing here gets both a small size and high power: five flips is too little data.
Two-sided, \(\alpha = 0.05\), true effect \(\delta\) SEs: \(\text{power} \approx 1 - \Phi(1.96 - \delta) + \Phi(-1.96 - \delta)\).
| \(\delta\) | 0 | 1 | 1.96 | 2.8 | 3.6 |
|---|---|---|---|---|---|
| power | 0.05 | 0.17 | 0.50 | 0.80 | 0.95 |
The "\(\delta = 2.8\) for 80% power" rule: \(2.8 = 1.96 + 0.84\), where 0.84 is the normal's 80th percentile. A study planned for 80% power needs the true effect to be 2.8 SEs. Example: to detect a 2-point difference with SD 10 per person in each of two groups, the SE is \(10\sqrt{2/n}\), so \(2 = 2.8 \times 10\sqrt{2/n}\) gives \(n = 2(28/2)^2 = 392\) per group.
Reject \(H_0: \theta = \theta_0\) when \(|W| > z_{\alpha/2}\), \(W = (\hat\theta - \theta_0)/\widehat{\text{se}}\). Equivalently, reject exactly when \(\theta_0\) is outside \(\hat\theta \pm z_{\alpha/2}\widehat{\text{se}}\).
Toy data: \(\bar x = 5\), \(\widehat{\text{se}} = 1\), interval \((3.04, 6.96)\). Test \(\mu_0 = 3\): \(W = 2\), reject. \(\mu_0 = 4\): \(W = 1\), retain. \(\mu_0 = 6.9\): \(W = -1.9\), retain. \(\mu_0 = 7\): \(W = -2\), reject. The interval is the list of null values that survive.
A p-value is the smallest \(\alpha\) at which you'd reject (AoS Def 10.11), equivalently the probability under \(H_0\) of a statistic at least as extreme as observed (Thm 10.12). Under \(H_0\) it is uniform (Thm 10.14); under an alternative it piles up near 0, more so with more power.
It is not \(P(H_0\mid\text{data})\). A large one is not evidence for \(H_0\) (it might be low power). And two studies with \(p = 0.04\) and \(p = 0.06\) found nearly identical evidence (\(z = 2.05\) vs \(1.88\)).
AoS: \((1, 9 \mid 3)\): \(T = |5 - 3| = 2\); the 3 distinct splits give \(T = 2, 7, 5\) (each twice among the 6 orderings). \(P(T > 2) = 4/6\) (AoS's convention); \(P(T \ge 2) = 1\).
Four points: A \(\{6, 8\}\), B \(\{1, 3\}\): \(T = 5\). The six splits give \(T = 5, 5, 2, 2, 0, 0\), so \(p = 2/6 = 1/3\). With two per group, no data can reach significance.
For simple \(f_0\) vs \(f_1\), the tests that can't be beaten on both error rates at once reject when \(\log(f_1(x)/f_0(x)) \ge c\).
One \(X\), \(N(0,1)\) vs \(N(1,1)\): \(\log L = x - \tfrac12\); reject when \(x \ge 1.645\) for \(\alpha = 0.05\); \(\beta = \Phi(1.645 - 1) = 0.74\).
Ten \(X\)s, \(N(0,1)\) vs \(N(0.5,1)\) (CASI Fig 2.2): \(\log L = 0.5\sum x_i - 1.25\). Rejecting when \(\sum x_i \ge 4\): \(\alpha = P(N(0,10) \ge 4) = 1 - \Phi(1.26) = 0.10\), \(\beta = P(N(5,10) < 4) = \Phi(-0.32) = 0.38\).
In both cases the "best test" is just "reject when the average is big." Neyman–Pearson justifies the obvious test.
Exactly \(\alpha\). With no true effect, "power" is the false-rejection rate. That's why the power curve starts at \(\alpha\), not 0.
Only if you decided before seeing the data, and only if an effect in the other direction would be treated like no effect. Choosing the side after looking doubles the real false-positive rate, and a one-sided test has essentially no power for effects in the opposite direction (set \(\delta\) negative in your head: the shaded region is on the wrong side).
"\(p < 0.05\)" and "reject at \(\alpha = 0.05\)" are the same event, so the share of p-values below 0.05 is the power, about 0.52 at \(\delta = 2\).