DS 5030 · Understanding Uncertainty · study guide

Standard errors, intervals, and tests: one source

Everything Wasserman's All of Statistics and Efron & Hastie's Computer Age Statistical Inference say about where uncertainty statements come from, rebuilt in one order, with analogies, story proofs, and the course's own numbers.

How to use this guide

Every section follows the same pattern: the idea in plain words, the formal statement with its book reference, then an analogy, a story proof, a worked example, and a real-world case. Read the plain words and the analogy first; the formal box is there when you need to be exact.

Formal statement Analogy Story proof Worked example Work it by hand Pitfall Why it matters Think about it / Check your understanding

Book references look like AoS Thm 6.16 (Wasserman's numbered result) or CASI (10.6) (Efron & Hastie's equation number). Results that appear in neither book but are standard, or that come from this course's own simulations, are marked standard or course. The text paraphrases both books; it never quotes them at length.

Worked examples are meant to be done with a pencil and a normal or t table; where a constant like $\log 40$ is needed, its rounded value is given. Boxes tagged From the books report published results to quote, not to recompute. Why it matters boxes connect a section to a real case where the idea cost someone money, a decision, or worse, or to a scenario from data-science work with illustrative numbers (always labelled as such).

Two kinds of collapsible questions run through the guide. Think about it questions look easy to answer; each one reveals a common mistake and then the real answer. Check your understanding questions ask you to explain or work out a point from the section. Commit to an answer before revealing it.

A story proof, in Blitzstein & Hwang's sense, is an argument that explains why a result must be true by describing what is happening, instead of by pushing symbols. It is not a substitute for the algebra, but it is usually the part you remember.

Topic	AoS	CASI
Estimator, bias, SE, MSE, consistency	§6.3.1	Ch 2 opening
LLN, CLT, delta method	Ch 5	§2.1 (device 2), §4.2
Empirical CDF, plug-in estimators	Ch 7	§2.1 (device 1)
Bootstrap SE and intervals	Ch 8	§10.2, Ch 11
Jackknife	§8.5.1	§10.1, §10.3
MLE, Fisher information, parametric bootstrap	Ch 9	§4.1–4.2, §10.4
Confidence intervals	§6.3.2	§2.1 (device 5), Ch 11
Tests, p-values, Wald, t, permutation, Neyman–Pearson	Ch 10	§1.2, §2.1, §2.2, §3.3
Multiple testing, FDR	§10.7	§1.2, §15.1–15.2
Regression inference	§13.1–13.5	§1.1

The hand-worked datasets

Every formal statement is followed by a Work it by hand example using tiny integer data, small enough to work with a pencil in a couple of minutes. The same few datasets recur, so you meet familiar numbers in new places.

Name	Data	Why it's convenient
Toy data	$\{2, 4, 5, 6, 8\}$	mean 5, deviations $-3, -1, 0, 1, 3$, $s^2 = 5$, SE exactly 1
Tiny population	$\{1, 3, 5\}$, samples of 2	only 9 samples, so the whole sampling distribution fits in a row; $\sigma^2 = 8/3$
Bootstrap toy	$\{1, 3, 5\}$, resamples of 3	only 10 distinct resamples, so the ideal bootstrap can be listed
Two groups	$\{2, 4, 6\}$ vs. $\{5, 7, 9\}$	means 4 and 7, pooled variance 4
Five-point line	$x = \{1, \dots, 5\}$, $y = \{1, 3, 2, 5, 4\}$	slope 0.8, intercept 0.6, $\hat\sigma^2 = 1.2$
Coins	5 or 100 flips	p-values and power by counting

Notation used throughout

Symbol	Meaning
$F$	the unknown distribution that generated the data (the population)
$\hat F_n$	the empirical distribution: probability $1/n$ on each data point
$\theta = T(F)$	the parameter: a fixed number, some feature of $F$
$\hat\theta_n$	an estimate (a number) or estimator (a random variable); CASI writes $\hat\theta$ vs. $\hat\Theta$
$\text{se}$, $\widehat{\text{se}}$	the true standard error, and an estimate of it
$\hat\theta^*_b$	the statistic computed on bootstrap sample $b$
$\hat\theta_{(i)}$	the statistic with observation $i$ left out (jackknife)
$z_{\alpha/2}$	the standard normal value with $\alpha/2$ above it: $z_{.025} = 1.96$
$\Phi$	the standard normal CDF

1 · Sampling distributions: the one idea under everything

Go deeper: expanded page with more worked examples and interactive widgets →

Frequentist inference rests on a single move. You computed one number from one dataset. Imagine rerunning the whole study, with new data from the same source, many times. Your number would come out differently each time. The spread of those imagined numbers is how accurate your one number is.

Analogy · parallel universes

Picture a thousand parallel universes, each with its own research team running your exact study on its own sample of 92 used cars. Each team reports an average price. You live in one universe and see one report. The sampling distribution is the histogram of all thousand reports. You will never see it, but its width tells you how much to trust your own.

Estimate vs. estimator

CASI makes the distinction with capitalization. The estimate $\hat\theta = t(x)$ is the number you got from your data $x$. The estimator $\hat\Theta = t(X)$ is the same recipe applied to random data $X$, so it is a random variable with a distribution. The frequentist principle, in CASI's words paraphrased: the accuracy of the number you have is defined as the accuracy of the recipe that produced it.

Formal · CASI (2.2)–(2.7), AoS §6.3.1

Data $X_1,\dots,X_n \overset{iid}{\sim} F$, written $F \to X$. A point estimator is any function $\hat\theta_n = g(X_1,\dots,X_n)$. Its distribution is the sampling distribution. With $\mu = E_F[\hat\Theta]$:

$$\text{bias} = \mu - \theta, \qquad \text{var} = E_F\big[(\hat\Theta - \mu)^2\big].$$

AoS reminds the reader that $E_\theta$ here means averaging over datasets drawn from $F$, never averaging over values of $\theta$.

Work it by hand · a whole sampling distribution

Population $\{1, 3, 5\}$, so $\mu = 3$ and $\sigma^2 = \frac{(1-3)^2 + 0 + (5-3)^2}{3} = \frac83$. Draw $n = 2$ with replacement. There are only 9 equally likely samples, so we can list the entire sampling distribution of $\bar X$.

sample	(1,1)	(1,3)	(1,5)	(3,1)	(3,3)	(3,5)	(5,1)	(5,3)	(5,5)
$\bar x$	1	2	3	2	3	4	3	4	5

Sampling distribution: $\bar X = 1, 2, 3, 4, 5$ with probabilities $\frac19, \frac29, \frac39, \frac29, \frac19$.

Center. $E[\bar X] = \frac{1(1) + 2(2) + 3(3) + 4(2) + 5(1)}{9} = \frac{27}{9} = 3 = \mu$, so $\bar X$ is unbiased (bias 0).

Spread. Treat the five values of $\bar X$ as a dataset in their own right, weighted by how often they occur, and compute an ordinary standard deviation around their mean of 3:

$\bar x$	1	2	3	4	5	total
probability	$\frac19$	$\frac29$	$\frac39$	$\frac29$	$\frac19$	1
deviation $\bar x - 3$	$-2$	$-1$	0	1	2
squared deviation	4	1	0	1	4
probability × squared deviation	$\frac49$	$\frac29$	0	$\frac29$	$\frac49$	$\frac{12}{9} = \frac43$

$$V(\bar X) = \tfrac43, \qquad \text{SD}(\bar X) = \sqrt{4/3} = 1.15.$$

Read it as "a sample mean of two draws typically misses $\mu = 3$ by about 1.15." This SD of the estimate is called its standard error (se). The standard error subsection, right after the pitfall box, defines it formally and gets the same 1.15 from a formula, without listing samples.

Every later idea (bias, SE, coverage, MSE) can be checked on this one table.

Pitfall · the sampling distribution is not a distribution of $\theta$

Which quantities are random. In the table above, $\theta = \mu = 3$ is a fact about the population $\{1, 3, 5\}$. It is fixed, whether or not you know it. The sample is random, so $\bar X$ is random, with distribution $1, 2, 3, 4, 5$ at probabilities $\frac19, \frac29, \frac39, \frac29, \frac19$. Every frequentist probability is a statement about $\bar X$ or something built from it, such as $P(\bar X = 3) = \frac39$ or $P(|\bar X - \theta| \le 1) = \frac79$. A statement like "$P(\theta = 4)$" has no place here: $\theta$ is 3, so the statement is 0 or 1. The sampling distribution is computed with $\theta$ held fixed. It answers "if the truth is $\theta$, what estimates would I see?" and never "given my estimate, where is the truth?"

The tempting move. Suppose you draw $(3, 5)$, so $\bar x = 4$ and $\text{se} = 1.15$. The natural next thought is "$\theta$ is around 4, give or take 1.15," which pictures a bell curve over $\theta$ centered at 4. That move takes the sampling distribution, re-centers it on your estimate, and reads it as a distribution over the parameter. The sampling distribution is $P(\text{data} \mid \theta)$; the re-centered curve claims to be $P(\theta \mid \text{data})$. Swapping one for the other is the same error as confusing $P(\text{positive test} \mid \text{disease})$ with $P(\text{disease} \mid \text{positive test})$. The only legitimate route to $P(\theta \mid \text{data})$ is Bayes' rule, and that needs a prior over $\theta$, which the sampling distribution does not contain.

Why the mistake survives. It often gives the right numbers. For a normal mean with a flat prior, the Bayesian posterior is centered at $\bar x$ with spread $\text{se}$, so "$\theta$ is probably in $\bar x \pm 1.96\,\text{se}$" agrees numerically with the frequentist interval. The wrong reading goes unpunished until a case where the two answers differ.

What the frequentist claims instead. The claim is about the recipe: "$\bar X$ lands within 1 of $\theta$ with probability $\frac79$, whatever $\theta$ is." After seeing $\bar x = 4$, you attach that recipe's reliability to your one number. That is the frequentist principle from Estimate vs. estimator: the accuracy of the estimate is defined as the accuracy of the estimator.

Where the shock lands. Attaching an average over all possible datasets to the one dataset you have is exactly the step the Bayesian critique attacks. CASI opens Chapter 3 by warning that this critique "may come as a shock." A procedure that is right 75% of the time on average can be obviously right, or obviously shaky, on the data in front of you. The frequentist number cannot tell these apart, because it averages over datasets you never saw. Section 5 gives the knockout case (AoS Ex 6.14): a valid 75% confidence set that, once you see $Y_1 = 15$, $Y_2 = 17$, you know with certainty contains $\theta$.

In short: before the data, $\theta$ is fixed and $\bar X$ is random. After the data, both are fixed numbers and nothing random is left. The probability lives in the procedure, not in $\theta$ and not in your interval.

Summary, using the draw $(3, 5)$, so $\bar x = 4$ and $\text{se} = 1.15$:

WHAT YOU CANNOT SAY	WHAT YOU CAN SAY INSTEAD
"$\theta$ is distributed around 4 with SD 1.15."	"Estimates from this procedure scatter around $\theta$ with SD 1.15, and 4 is one of them."
"There is a $\frac79$ probability that $\theta$ is within 1 of 4."	"The rule $\bar X$ lands within 1 of $\theta$ with probability $\frac79$, whatever $\theta$ is, and my 4 came from that rule."
"The interval $4 \pm 1$ has a $\frac79$ chance of containing $\theta$."	"$4 \pm 1$ came from a rule that covers $\theta$ in $\frac79$ of samples. This interval either contains $\theta$ or it doesn't." (Here it does: $[3, 5]$ contains 3.)
"The SE measures how uncertain $\theta$ is."	"The SE measures how much my estimate would vary from sample to sample."
"$P(\theta = 3) = \frac39$."	"$P(\bar X = 3) = \frac39$." Probabilities attach to $\bar X$, never to $\theta$.

The standard error

Formal · AoS (6.5), CASI §1 and Ch 10 opening

The standard error of $\hat\theta_n$ is the standard deviation of its sampling distribution:

$$\text{se} = \text{se}(\hat\theta_n) = \sqrt{V(\hat\theta_n)}.$$

It usually depends on the unknown $F$, so in practice we report an estimate $\widehat{\text{se}}$. CASI calls the standard error "a central element of frequentist inference" and uses "standard error" and "standard deviation of the estimate" interchangeably.

Work it by hand · the formula, on the same nine samples

Back to the population $\{1, 3, 5\}$ with $n = 2$. Earlier we listed all nine samples and got $\text{SD}(\bar X) = \sqrt{4/3} = 1.15$ by hand. For a sample mean, the formula gets there without the list. Variances of independent draws add, so $V(X_1 + \dots + X_n) = n\sigma^2$; dividing the sum by $n$ divides the variance by $n^2$:

$$V(\bar X) = \frac{n\sigma^2}{n^2} = \frac{\sigma^2}{n}, \qquad \text{se}(\bar X) = \frac{\sigma}{\sqrt n}.$$

With $\sigma^2 = \frac83$: $V(\bar X) = \frac{8/3}{2} = \frac43$ and $\text{se} = \sqrt{8/3}\,/\sqrt 2 = 1.633/1.414 = 1.15$. ✓ Same number as the table.

Two SDs are in play. The population SD $\sigma = 1.63$ is the spread of single draws; the SE, 1.15, is the spread of averages of two draws. Averaging shrinks the spread by $\sqrt 2$.

The formula still needs $\sigma$, which in practice you don't know. From one sample, say $(3, 5)$, you estimate it: $\bar x = 4$, $s^2 = \frac{(3-4)^2 + (5-4)^2}{2 - 1} = 2$, so $\widehat{\text{se}} = \sqrt{s^2/n} = \sqrt{2/2} = 1$. The true se is 1.15; this sample's estimate of it is 1. A different sample gives a different $\widehat{\text{se}}$ (from $(3, 3)$ it is 0), which is the question Section 4 takes up.

Why divide by $n - 1$. Not just because $\sigma$ is estimated, but because the deviations are measured from $\bar x$, which was fit to these same data and so sits closer to them than $\mu$ does. For $(3, 5)$: squared deviations from $\bar x = 4$ sum to $1 + 1 = 2$; from the true $\mu = 3$ they would sum to $0 + 4 = 4$. In general $\sum(x_i - \bar x)^2 = \sum(x_i - \mu)^2 - n(\bar x - \mu)^2$, whose expectation is $n\sigma^2 - \sigma^2 = (n-1)\sigma^2$. Dividing by $n - 1$ makes $s^2$ unbiased: averaged over the nine samples, $s^2$ comes out to exactly $\frac83 = \sigma^2$. (With $\mu$ known, you would divide by $n$. Each estimated center costs one degree of freedom, which is where $n - k$ comes from in regression.)

But $\widehat{\text{se}}$ is still biased low. $s^2$ is unbiased for $\sigma^2$; its square root is not unbiased for $\sigma$. With $n = 2$, $s^2 = (x_1 - x_2)^2/2$, so $\widehat{\text{se}} = \sqrt{s^2/2} = |x_1 - x_2|/2$:

$\|x_1 - x_2\|$	0	2	4	average
samples (of 9)	3	4	2
$s^2$	0	2	8	$\frac{0 + 8 + 16}{9} = \frac83$ ✓ unbiased
$\widehat{\text{se}}$	0	1	2	$\frac{0 + 4 + 4}{9} = \frac89 = 0.89$, not 1.15

The square root pulls the average down: the average of square roots is smaller than the square root of the average (Jensen's inequality). So the $n - 1$ fixes the variance, not the SE. The gap shrinks as $n$ grows, and it's one more way the SE is itself uncertain at small $n$.

Work it by hand · the toy dataset

Data $\{2, 4, 5, 6, 8\}$. Mean $\bar x = 25/5 = 5$. Deviations $-3, -1, 0, 1, 3$; squares $9, 1, 0, 1, 9$, sum 20.

$s^2 = 20/4 = 5$, $s = 2.24$: how much the data spread.
$\widehat{\text{se}} = \sqrt{s^2/n} = \sqrt{5/5} = 1$: how much the mean would wobble. (CASI (1.2) directly: $\sqrt{20/(5 \cdot 4)} = 1$.)
Plug-in version: $\hat\sigma^2 = 20/5 = 4$, $\hat\sigma/\sqrt n = 2/\sqrt5 = 0.89$.

Report: $5 \pm 1$.

CASI's opening example is the whole subject in two lines. The algorithm is the average, $\bar x = \sum x_i/n$. The inference is its accuracy:

$$\widehat{\text{se}} = \Big[\sum_{i=1}^n (x_i - \bar x)^2 \big/ \big(n(n-1)\big)\Big]^{1/2} \qquad \text{CASI (1.2)}$$

Efron & Hastie call it surprising and crucial that the same data that produce an estimate can also grade it. They also point out that $\widehat{\text{se}}$ is itself an algorithm whose accuracy could be questioned. Section 4 of this guide takes them up on that.

Analogy · SD vs. SE

The SD is how much people's weights differ. The SE is how much a bathroom scale's reading of the average weight in a room would wobble if you redid the weigh-in with a new room of people. Weighing more people doesn't make people more alike; it makes your average steadier.

Worked example · course and CASI

Car prices (92 Charlottesville listings, in $1000s): $s = 8.17$, $\widehat{\text{se}} = 8.17/\sqrt{92} = 0.85$. CASI's gfr data (211 kidney patients): $54.25 \pm 0.95$ CASI (2.1). CASI's reading: don't trust the ".25", and even the "4" is shaky. Working backwards, $s \approx 0.95\sqrt{211} \approx 13.8$.

Why it matters · real case · the small-schools bet

Around 2000 the Gates Foundation acted on data showing that small high schools were overrepresented among the top scorers on state tests. It funded the creation of small schools and the splitting of large ones: hundreds of millions of dollars, and by some counts well over a billion. Wainer pointed out that small schools are also overrepresented among the worst scorers. Size wasn't improving teaching; small schools simply have noisier averages. He called $\sigma/\sqrt n$ "the most dangerous equation" for this reason, and the foundation later dropped the strategy. Wainer, American Scientist 2007

By hand. Suppose individual scores have SD 100. A school testing 25 students has $\text{se} = 100/5 = 20$; a school testing 400 has $\text{se} = 100/20 = 5$. A school average 40 points above the state's is 2 SEs, ordinary luck, for the small school, and 8 SEs, essentially impossible by luck, for the large one. Rank schools by raw average and both ends of the list fill up with small schools.

In data work: any leaderboard of averages over groups of unequal size. Stores by conversion rate, sales reps by close rate, hospitals by mortality, products by star rating.

Think about itIf I double the sample size, what happens to the SD of the data, and what happens to the SE of the mean?

Common mistake: Both get smaller, because more data means less uncertainty.

The real answer: The SD stays about the same: it estimates a fixed feature of the population (how much cars differ). The SE shrinks by a factor of $1/\sqrt 2 \approx 0.71$, because it measures how much the average wobbles. More data doesn't make cars more alike.

Check your understandingIs the standard error a property of the data or of the estimator?

Of the procedure: the estimator, the sampling design, and the population $F$ together. The same data give different SEs for the mean and the median, and the same estimator gives a different SE under a different sampling plan (paired vs. independent, clustered vs. iid). What the data supply is an estimate of it.

Think about itEach of these is "8.59 ± 0.85". Which one is the SE: the mean car price is $8.59k ± $0.85k, or cars cost $8.59k ± $0.85k?

Common mistake: They mean the same thing.

The real answer: Only the first. The SE describes the average; individual cars spread with SD $8.17k. The second sentence claims cars are nearly all priced within a thousand dollars of each other, which is false.

Bias, variance, and mean squared error

An estimator can be wrong in two ways: it can be aimed at the wrong place on average (bias), or it can scatter (variance). Mean squared error adds the two.

Formal · AoS Thm 6.9 $$\text{MSE} = E_\theta\big[(\hat\theta_n - \theta)^2\big] = \text{bias}^2(\hat\theta_n) + V_\theta(\hat\theta_n).$$

Proof (AoS). Let $\bar\theta_n = E_\theta(\hat\theta_n)$. Expand $(\hat\theta_n - \bar\theta_n + \bar\theta_n - \theta)^2$. The cross term is $2(\bar\theta_n - \theta)E_\theta(\hat\theta_n - \bar\theta_n) = 0$, leaving $(\bar\theta_n-\theta)^2 + E_\theta(\hat\theta_n - \bar\theta_n)^2$.

Work it by hand · a biased estimator that beats an unbiased one

Same nine samples from $\{1, 3, 5\}$. Estimate $\sigma^2 = 8/3$ two ways. With $n = 2$, the plug-in variance is $\hat\sigma^2 = \frac{(x_1 - x_2)^2}{4}$, and the unbiased one is $s^2 = 2\hat\sigma^2$.

$\|x_1 - x_2\|$	how many samples	$\hat\sigma^2$	$s^2$
0	3	0	0
2	4	1	2
4	2	4	8

	mean	bias	variance	MSE = bias² + var
$\hat\sigma^2$ (divide by $n$)	$\frac{0+4+8}{9} = \frac43$	$-\frac43$	$\frac{20}{9}$	$\frac{16}{9} + \frac{20}{9} = 4$
$s^2$ (divide by $n-1$)	$\frac83$	0	$\frac{80}{9}$	$\frac{80}{9} \approx 8.9$

The unbiased estimator has more than twice the MSE. (Variance of $\hat\sigma^2$: $E[\hat\sigma^4] = \frac{0 + 4(1) + 2(16)}{9} = 4$, minus $(4/3)^2$, gives $20/9$; doubling the estimator quadruples the variance.) This is AoS's "unbiasedness is less important than it used to be" and CASI (3.29) in one table.

Analogy · archery

Bias is how far the center of your arrow cluster sits from the bullseye. Variance is how spread out the cluster is. A tight cluster in the wrong place and a loose cluster centered on the bullseye can have the same MSE.

Why it matters · real case · 2.4 million wrong answers

In 1936 the Literary Digest mailed straw-poll ballots to millions of people, drawn largely from telephone directories, automobile registrations and its own subscriber lists. About 2.4 million came back, and the poll predicted that Alf Landon would beat Franklin Roosevelt with about 57% of the vote. Roosevelt won with about 61%; Landon got about 37%. George Gallup, with a far smaller sample, called the election correctly.

By hand. The SE of a proportion from 2.4 million responses is $\sqrt{0.25/2{,}400{,}000} \approx 0.0003$: three hundredths of a percentage point. The miss on Landon's share was about 20 points. In MSE terms, $\text{bias}^2 = 0.2^2 = 0.04$ against $\text{se}^2 \approx 10^{-7}$: the bias term is hundreds of thousands of times larger. Collecting more responses the same way shrinks the SE and leaves the bias exactly where it was.

In data work: a survey of active users, a churn model trained only on customers who stayed long enough to be logged, an A/B test run only on desktop. A tiny SE on any of them describes the noise, not whether the sample resembles the population you will act on.

Why it matters · real case and everyday version · biased estimates that predict better

Efron and Morris took the batting averages of 18 major-league players after their first 45 at-bats of the 1970 season and asked how well each predicted that player's average for the rest of the season. Shrinking every player's average part of the way toward the group mean (the James–Stein estimator, CASI Ch 7) predicted better overall than each player's own average. The shrunken estimates are biased; they win on MSE because they cut variance by more than they add in bias. Efron & Morris 1975; CASI Table 7.1

Everyday version, illustrative numbers. Rank products by star rating. Product A has one review, 5 stars. Product B has 200 reviews averaging 4.7. The site-wide average is 4.0. Raw averages put A first. A common fix treats every product as if it also had $k = 10$ reviews at the site average:

$$\text{estimate} = 4.0 + \frac{n}{n + 10}\,(\text{raw average} - 4.0).$$

A: $4.0 + \frac{1}{11}(1.0) = 4.09$. B: $4.0 + \frac{200}{210}(0.7) = 4.67$. B now ranks first. The estimate for A is biased toward 4.0 and is much closer to where A's average will end up after 200 reviews.

Story proof · why the cross term vanishes

Split every miss into two legs: from the bullseye to the center of the cluster (always the same leg, the bias) and from the center of the cluster to this arrow (the scatter). The scatter leg points in every direction equally often, since by definition the cluster's center is its average. So on average it is perpendicular to the fixed bias leg, and Pythagoras applies: squared miss = squared bias + squared scatter.

Wasserman notes that unbiasedness "used to receive much attention" but is now considered less important; many good estimators are biased. CASI §3.4 makes the same point with variance estimators $\sum(x_i-\bar x)^2/(n-c)$: $c = 1$ is unbiased, but a different $c$ gives smaller MSE CASI (3.29).

Think about itAn estimator is unbiased. Does that mean it's usually close to the truth?

Common mistake: Yes, unbiased means it's correct.

The real answer: No. Unbiased only means it's right on average over many samples. "Use only the first observation $X_1$" is an unbiased estimator of $\mu$, and it never improves no matter how much data you collect: its MSE is $\sigma^2$ forever. Closeness is MSE (bias² + variance), and improving with $n$ is consistency.

Check your understandingWhy divide by $n-1$ in the sample variance?

Deviations measured from $\bar x$ are systematically smaller than deviations from $\mu$, because $\bar x$ is fit to the same data. Exactly: $\sum(x_i - \bar x)^2 = \sum(x_i - \mu)^2 - n(\bar x - \mu)^2$. Take expectations: $n\sigma^2 - n\cdot\sigma^2/n = (n-1)\sigma^2$. Dividing by $n-1$ makes $s^2$ unbiased. Two follow-ups worth knowing: $s$ itself is still biased for $\sigma$ (Jensen), and the unbiased divisor isn't the minimum-MSE one (CASI (3.29)).

Consistency

Formal · AoS Def 6.7, Thm 6.10

$\hat\theta_n$ is consistent if $\hat\theta_n \xrightarrow{P} \theta$. If bias $\to 0$ and se $\to 0$ as $n \to \infty$, then $\hat\theta_n$ is consistent. (Proof: MSE $\to 0$, which is convergence in quadratic mean, which implies convergence in probability, AoS Thm 5.4.)

Work it by hand · watching consistency happen

Data with $\sigma = 2$, estimator $\bar X_n$: bias 0, $\text{se} = 2/\sqrt n$. Chebyshev: $P(|\bar X_n - \mu| > 1) \le \sigma^2/(n\cdot 1^2) = 4/n$.

$n$	4	16	100	400
se	1	0.5	0.2	0.1
bound on $P(\text{miss by} > 1)$	1 (useless)	0.25	0.04	0.01

Bias stays 0 and se shrinks to 0, so the probability of missing by any fixed amount goes to 0: that is consistency.

Worked example · AoS Ex 6.8, 6.11 (coin)

$X_i \sim \text{Bernoulli}(p)$, $\hat p_n = \bar X_n$. Then $E(\hat p_n) = p$ (unbiased) and $\text{se} = \sqrt{p(1-p)/n} \to 0$, so $\hat p_n$ is consistent. The estimated standard error is $\widehat{\text{se}} = \sqrt{\hat p(1-\hat p)/n}$. With $\hat p = 0.6$, $n = 100$: $\widehat{\text{se}} = \sqrt{0.24/100} = 0.049$.

2 · The LLN, the CLT, and the delta method

Go deeper: expanded page with more worked examples and interactive widgets →

Three limit theorems do almost all the work. The law of large numbers says averages settle down. The central limit theorem says how the remaining wobble is shaped. The delta method says what happens to that wobble when you transform the average.

Kinds of convergence

Formal · AoS Defs 5.1–5.2, Thm 5.4

In probability, $X_n \xrightarrow{P} X$: for every $\epsilon > 0$, $P(|X_n - X| > \epsilon) \to 0$.
In distribution, $X_n \rightsquigarrow X$: $F_n(t) \to F(t)$ at every $t$ where $F$ is continuous.
In quadratic mean, $X_n \xrightarrow{qm} X$: $E(X_n - X)^2 \to 0$.

Quadratic mean implies probability, which implies distribution. The reverse implications fail in general. (AoS Ex 5.3: $X_n \sim N(0, 1/n)$ concentrates at 0 and converges to 0 in all three senses.)

Work it by hand · why the implications only go one way

$X_n = 1$ with probability $1/n$, else 0. Then $P(|X_n| > \epsilon) = 1/n \to 0$ (in probability) and $E[X_n^2] = 1/n \to 0$ (in quadratic mean).

$Y_n = n$ with probability $1/n$, else 0. Still $P(|Y_n| > \epsilon) = 1/n \to 0$, so $Y_n \xrightarrow{P} 0$. But $E[Y_n^2] = n^2 \cdot \frac1n = n \to \infty$, so no convergence in quadratic mean, and $E[Y_n] = 1$ for every $n$ even though $Y_n$ is almost always 0. A rare, huge value keeps the mean alive. (This is the Cauchy problem in miniature.)

Analogy

Convergence in probability is about the values getting close: the darts land near the bullseye. Convergence in distribution is only about the histogram of where darts land matching a target histogram, even if no individual dart is close to anything. The CLT is a statement of the second kind.

What each kind of convergence says, in words

In all three, the target $X$ is often a constant: "$\bar X_n$ converges to $\mu$." Read each one as an answer to a different question.

Kind	The question it answers	Typical use
in quadratic mean	Does the average squared miss go to zero?	MSE → 0, the easy route to consistency (AoS Thm 6.10)
in probability	For any tolerance $\epsilon$, does the chance of missing by more than $\epsilon$ go to zero?	the LLN; consistency of an estimator
in distribution	Does the histogram (the CDF) approach a target shape?	the CLT; approximate intervals and p-values

The first two are about the random variable's values getting close to the target. The third is only about the shape of its distribution, which is why the CLT needs the rescaling $\sqrt n(\bar X_n - \mu)$: without it, $\bar X_n - \mu$ just shrinks to 0 and the "shape" is a spike.

Work it by hand · all three at once for $X_n \sim N(0, 1/n)$ (AoS Ex 5.3)

In probability. $P(|X_n| > \epsilon) = 2\Phi(-\epsilon\sqrt n)$. With $\epsilon = 0.1$:

$n$	1	100	400	10,000
$P(\|X_n\| > 0.1)$	$2\Phi(-0.1) = 0.92$	$2\Phi(-1) = 0.32$	$2\Phi(-2) = 0.046$	$2\Phi(-10) \approx 0$

In quadratic mean. $E[X_n^2] = V(X_n) = 1/n \to 0$.

In distribution. $F_n(t) = \Phi(t\sqrt n)$, which tends to 0 for $t < 0$ and to 1 for $t > 0$: the CDF of a point mass at 0. But at $t = 0$ itself, $F_n(0) = \Phi(0) = \tfrac12$ for every $n$, while the limit's CDF is $F(0) = 1$. They never agree at 0. That's exactly why the definition says "at every $t$ where $F$ is continuous": the limit's CDF jumps at 0, and the definition forgives that one point.

Work it by hand · the counterexamples AoS uses

Probability without quadratic mean. $U \sim \text{Uniform}(0, 1)$, $X_n = \sqrt n$ if $U < 1/n$, else 0. $P(|X_n| > \epsilon) = 1/n \to 0$, so $X_n \xrightarrow{P} 0$. But $E[X_n^2] = n \cdot \tfrac1n = 1$ for every $n$: the average squared miss never shrinks.

Distribution without probability. $X \sim N(0, 1)$ and $X_n = -X$ for every $n$. Every $X_n$ has exactly the distribution of $X$, so $X_n \rightsquigarrow X$ trivially. But $|X_n - X| = 2|X|$, which never gets small. Same histogram, different values. (A simpler version: flip a fresh fair coin at every $n$. The distribution never changes; the value never settles.)

Converging in probability doesn't make the mean converge. $X_n = n^2$ with probability $1/n$, else 0. $X_n \xrightarrow{P} 0$, yet $E[X_n] = n \to \infty$. One rare, enormous value keeps the expectation alive.

Story proof · why the arrows point the way they do (AoS Thm 5.4)

Quadratic mean ⇒ probability. A miss bigger than $\epsilon$ contributes at least $\epsilon^2$ to the squared error. So if misses like that happened with probability $p$, the average squared miss would be at least $p\epsilon^2$. That's Markov's inequality: $P(|X_n - X| > \epsilon) \le E(X_n - X)^2/\epsilon^2$. If the right side goes to 0, so does the left.

Probability ⇒ distribution. If $X_n$ is within $\epsilon$ of $X$ except with a tiny probability, then "$X_n \le t$" almost forces "$X \le t + \epsilon$", and "$X \le t - \epsilon$" almost forces "$X_n \le t$". So $F(t - \epsilon) - \text{small} \le F_n(t) \le F(t + \epsilon) + \text{small}$. Shrink $\epsilon$; if $F$ is continuous at $t$, both bounds close in on $F(t)$.

The one reverse arrow: distribution to a constant ⇒ probability. If the limit is a point mass at $c$, then $P(|X_n - c| > \epsilon) \le F_n(c - \epsilon) + 1 - F_n(c + \epsilon) \to 0 + 1 - 1 = 0$. When the target is a single number, "the histogram becomes a spike at $c$" and "the values get close to $c$" are the same statement.

Combining limits: the continuous mapping theorem and Slutsky

Formal · AoS Thm 5.5

Let $g$ be continuous.

If $X_n \xrightarrow{P} X$ and $Y_n \xrightarrow{P} Y$, then $X_n + Y_n \xrightarrow{P} X + Y$ and $X_nY_n \xrightarrow{P} XY$.
Continuous mapping: $X_n \xrightarrow{P} X \Rightarrow g(X_n) \xrightarrow{P} g(X)$, and $X_n \rightsquigarrow X \Rightarrow g(X_n) \rightsquigarrow g(X)$.
Slutsky: if $X_n \rightsquigarrow X$ and $Y_n \rightsquigarrow c$ (a constant), then $X_n + Y_n \rightsquigarrow X + c$ and $X_nY_n \rightsquigarrow cX$.

Warning (AoS): $X_n \rightsquigarrow X$ and $Y_n \rightsquigarrow Y$ do not in general give $X_n + Y_n \rightsquigarrow X + Y$. Slutsky needs one of the two limits to be a constant.

Work it by hand · why the CLT still works with $s$ in place of $\sigma$ (AoS Thm 5.10) $$\frac{\sqrt n(\bar X_n - \mu)}{S_n} = \underbrace{\frac{\sqrt n(\bar X_n - \mu)}{\sigma}}_{\rightsquigarrow\ N(0,1)\ \text{(CLT)}} \times \underbrace{\frac{\sigma}{S_n}}_{\xrightarrow{P}\ 1}$$

The right factor: $S_n^2 \xrightarrow{P} \sigma^2$ by the LLN (applied to the $X_i^2$ and to $\bar X_n$), and $g(v) = \sigma/\sqrt v$ is continuous, so $\sigma/S_n \xrightarrow{P} 1$ by continuous mapping. Slutsky multiplies the limits: $N(0,1) \times 1 = N(0,1)$. This one line licenses every "estimate ± 1.96 SE" interval in the course, and it also says what it doesn't promise: nothing about how fast $\sigma/S_n$ gets close to 1, which is the small-$n$ problem of Section 4.

Work it by hand · Slutsky's warning, in one line

Let $X_n \sim N(0,1)$ and $Y_n = -X_n$. Each is $\rightsquigarrow N(0,1)$, so you might expect $X_n + Y_n \rightsquigarrow N(0, 2)$. But $X_n + Y_n = 0$ exactly. Convergence in distribution says nothing about how two sequences move together, which is why Slutsky insists one limit be a constant.

Result in this guide	Kind of convergence
MSE → 0 (bias → 0 and se → 0)	quadratic mean, which gives consistency
Weak LLN, $\bar X_n \to \mu$	probability
Consistency of an estimator; the MLE	probability
CLT, $\sqrt n(\bar X_n - \mu)/\sigma$	distribution
CLT with $S_n$ (Thm 5.10)	distribution, via Slutsky
Delta method	distribution, via a Taylor step plus Slutsky
MLE asymptotic normality	distribution (CLT on the score, LLN on the curvature, Slutsky to combine)

Think about itThe CLT is only convergence in distribution. So can we still say $\bar X_n$ gets close to $\mu$?

Common mistake: No, the CLT only talks about the shape.

The real answer: Yes, and the CLT even implies it. $\sqrt n(\bar X_n - \mu) \rightsquigarrow N(0, \sigma^2)$ and $1/\sqrt n \to 0$, so by Slutsky $\bar X_n - \mu = \frac{1}{\sqrt n}\cdot\sqrt n(\bar X_n - \mu) \rightsquigarrow 0 \cdot N(0, \sigma^2) = 0$. Convergence in distribution to a constant is convergence in probability (AoS Thm 5.4(c)). The CLT says more than the LLN: not just that the miss goes to 0, but that it does so at rate $1/\sqrt n$ with a bell shape.

Check your understandingWhy does it matter that the definition of convergence in distribution skips discontinuity points?

Because limits are often constants, whose CDFs jump. For $X_n \sim N(0, 1/n)$, $F_n(0) = \tfrac12$ forever while the limit's $F(0) = 1$. Requiring agreement at every point would say this obviously-converging sequence doesn't converge. An even simpler case: the constants $X_n = 1/n$ obviously converge to 0, yet $F_n(0) = P(1/n \le 0) = 0$ for every $n$ while $F(0) = 1$. Which side of the jump the sequence approaches from shouldn't decide whether it converges, so the jump point is excluded.

The law of large numbers

Formal · AoS Thm 5.6 (WLLN)

If $X_1, \dots, X_n$ are iid with finite mean $\mu$, then $\bar X_n \xrightarrow{P} \mu$.

When the variance $\sigma^2$ is also finite, the proof is one line of Chebyshev: $P(|\bar X_n - \mu| > \epsilon) \le V(\bar X_n)/\epsilon^2 = \sigma^2/(n\epsilon^2) \to 0$.

Work it by hand · Chebyshev on a die

A fair die: $\mu = 3.5$, $\sigma^2 = 35/12 \approx 2.92$. Roll 100 times. Chebyshev: $P(|\bar X - 3.5| > 0.5) \le \frac{2.92}{100 \times 0.25} = 0.117$.

The CLT gives the real answer: $\text{se} = \sqrt{2.92/100} = 0.171$, so $0.5$ is 2.93 SEs and $P \approx 2\Phi(-2.93) = 0.003$. Chebyshev is always true and usually very loose, which is why it proves the LLN but nobody uses it for intervals.

The LLN is also what makes simulation work. AoS §8.1: draw $Y_1,\dots,Y_B$ from any distribution $G$; then $\frac1B\sum h(Y_j) \to E[h(Y)]$, so the sample variance of simulated values approximates the true variance. This is the license for the bootstrap's Monte Carlo step.

Story proof · why the variance of a mean is σ²/n

Think of each observation as truth plus its own error. The sum of $n$ errors is a random walk: each new error is a step left or right, independent of the others. Steps that agree add up, steps that disagree cancel, and independent steps agree and disagree equally often. A random walk of $n$ steps of size $\sigma$ typically ends about $\sigma\sqrt n$ from where it started (the variances of independent steps add, so the total variance is $n\sigma^2$). Dividing the sum by $n$ to get the mean shrinks that to $\sigma\sqrt n / n = \sigma/\sqrt n$.

The algebra is three lines: $V(\bar X_n) = \frac{1}{n^2}V(\sum X_i) = \frac{1}{n^2}\sum V(X_i) = \sigma^2/n$. The middle step is the one that uses independence, and it is the step that fails for correlated data (Section 9).

Why it matters · real case · pooling that didn't diversify

A mortgage-backed security pools thousands of home loans. If defaults were independent, the LLN would make the pool's default rate nearly predictable, and that predictability is what made the safest slices of these pools look safe. Before 2007, many rating and pricing models treated defaults in different regions as only weakly correlated. When house prices fell across the country at once, defaults moved together, and pooled losses were far larger than those models allowed.

By hand. For $n$ draws with SD $\sigma$ and the same correlation $\rho$ between every pair,

$$V(\bar X) = \frac{1}{n^2}\Big[n\sigma^2 + n(n-1)\rho\sigma^2\Big] = \sigma^2\Big[\frac1n + \rho\Big(1 - \frac1n\Big)\Big].$$

With $n = 10{,}000$ and $\rho = 0$, the SD of the average is $\sigma/100$. With $\rho = 0.1$ it is about $\sqrt{0.1}\,\sigma = 0.32\sigma$: 32 times larger, and no number of extra loans brings it below $0.32\sigma$. The middle step of the three-line proof above, the one that uses independence, is the step that broke.

Think about itThe law of large numbers says that if a coin has come up heads 10 times in a row, tails is now "due." True?

Common mistake: Yes, things have to even out.

The real answer: No. The coin has no memory, and the LLN never says the count of heads and tails evens out. It says the proportion converges: after 10 extra heads, the next 1,000 flips still average about half heads, and 10 extra heads out of 1,010 barely moves the proportion. The excess is diluted, not corrected.

The central limit theorem

Formal · AoS Thm 5.8, 5.10

Let $X_1, \dots, X_n$ be iid with mean $\mu$ and variance $\sigma^2 < \infty$. Then

$$Z_n = \frac{\bar X_n - \mu}{\sqrt{V(\bar X_n)}} = \frac{\sqrt n(\bar X_n - \mu)}{\sigma} \rightsquigarrow N(0,1), \qquad \text{i.e. } \lim_{n\to\infty} P(Z_n \le z) = \Phi(z).$$

Thm 5.10: the same holds with the sample SD $S_n$ in place of $\sigma$: $\sqrt n(\bar X_n - \mu)/S_n \rightsquigarrow N(0,1)$.

AoS lists equivalent shorthands: $\bar X_n \approx N(\mu, \sigma^2/n)$, $\sqrt n(\bar X_n - \mu) \approx N(0, \sigma^2)$, and so on.

Work it by hand · 100 coin flips

Fair coin, 100 flips. What's $P(\text{at least 60 heads})$? $\hat p$ has mean 0.5 and $\text{se} = \sqrt{0.25/100} = 0.05$. 60 heads is $\hat p = 0.6$, which is $(0.6 - 0.5)/0.05 = 2$ SEs up. CLT: $P(Z \ge 2) = 0.023$. The exact binomial answer is 0.028 (0.029 with a continuity correction). Close, and every step can be done by hand.

Wasserman adds an interpretive note: the CLT approximates probability statements about $\bar X_n$; it does not say $\bar X_n$ itself turns into a normal random variable.

How good is the approximation? The Berry–Esseen inequality AoS Thm 5.11 bounds the worst-case gap between $P(Z_n \le z)$ and $\Phi(z)$ by a constant times $E|X_1 - \mu|^3/(\sigma^3\sqrt n)$. Two lessons: the error shrinks like $1/\sqrt n$, and it is larger when the data have heavy tails or skew (big third moment). That is exactly what the course's car-price simulations show (Section 4).

Worked example · AoS Ex 5.9

Errors per program are Poisson with mean 5, so $\mu = \sigma^2 = 5$. With 125 programs, $P(\bar X_n < 5.5) \approx P\big(Z < \sqrt{125}(5.5 - 5)/\sqrt5\big) = P(Z < 2.5) = 0.9938$. Hand-workable: $\sqrt{125/5} = 5$, times 0.5.

Analogy · crowd noise

One person in a stadium can shout anything. Fifty thousand people produce a roar whose loudness barely changes from second to second, and whose small fluctuations have the same bell shape whatever the individuals are shouting. Individual quirks cancel; only the mean and spread of the quirks survive.

Story proof · why a bell, and why it's universal

Add independent errors one at a time. Each addition smears the current histogram by convolving it with the error's shape. Smearing removes detail: skew and bumps are averaged away, and after many smears only two features of the error survive, its center and its spread. The bell curve is the one shape that reproduces itself under this smearing (a sum of independent normals is normal), so it is where repeated smearing ends up. That is why the limit doesn't depend on what the errors looked like.

Pitfall

The CLT needs finite variance. For Cauchy data (a t with 1 degree of freedom, AoS §2.4) the sample mean never settles and has the same Cauchy distribution for every $n$. The course simulation shows $s/\sqrt n$ bouncing between 0.47 and 0.92 as $n$ grows from 100 to 100,000 course.

Why it matters · scenario · one customer moves the mean

Illustrative numbers. An A/B test on revenue per user has 1,000 users per arm. Almost everyone spends about $5, and one user in the whole experiment spends $10,000. Without that user, each arm averages $5. The arm that gets the big spender averages $(999 \times 5 + 10{,}000)/1{,}000 \approx \$15$. The "treatment effect" is plus or minus $10 per user depending on which arm one person was randomized into.

Revenue is heavy-tailed. The CLT still applies eventually, but the Berry–Esseen error term grows with the third moment, so "eventually" can mean far more users than the test has. Experimentation teams usually cap (winsorize) revenue at a high percentile, test a less heavy-tailed metric, or check the result with a bootstrap, and they say in the report which of these they did.

When the CLT fails

The CLT needs an average of many independent pieces, none of which dominates. It fails when the pieces are linked, when one piece can dominate, or when the estimate isn't an average at all. A useful check for any estimate: is there really a sum of iid pieces underneath it?

Missing ingredient	Example	What goes wrong	Fix
Independence	time series; repeated measures; many rows per person	covariances don't vanish, so $\sigma/\sqrt n$ is too small. Lag-1 correlation 0.7, $n = 100$: true SE ≈ 2.4 × $s/\sqrt n$; "95%" covers about 59%	model the dependence; bootstrap blocks or clusters
Finite variance	Cauchy; Pareto-type tails (incomes, losses, views)	the mean of $n$ Cauchy draws is Cauchy for every $n$; with very heavy tails one value can move the mean arbitrarily	robust statistics (median, trimmed mean); model the tail
Enough data for the skew	car prices at small $n$; rare events	Berry–Esseen error grows with skew. Cars, $n = 5$, estimated SE: 21% of "95%" intervals miss low, 1% high	more data; transform (log); bootstrap-t or BCa
A sum underneath	sample max or min; Uniform(0, θ) via the max; parameters at a boundary	maxima of normals tend to a Gumbel, not a normal; the max's error shrinks like $1/n$ with an exponential limit	extreme-value theory; exact or likelihood methods
A correct SE	small $n$; a failing bootstrap; classical regression SE under unequal variance	standardizing by a wrong or noisy SE: t-like tails (Section 4); Ames area SE a third of the truth (Section 8)	t or bootstrap-t; robust SE; block bootstrap
Random sampling	convenience samples; selecting the largest of many estimates	the CLT controls variance, not bias: a precise normal interval around the wrong number (CASI's winner's curse: 5.29 vs ≈ 4.1)	better design; shrinkage

Work it by hand · a rare event breaks the normal interval

$p = 0.01$, $n = 100$. The expected number of successes is 1. The chance of seeing none is $0.99^{100} = 0.37$. In that case $\hat p = 0$, $\widehat{\text{se}} = \sqrt{0 \times 1/100} = 0$, and the "95%" interval $\hat p \pm 1.96\,\widehat{\text{se}}$ is $[0, 0]$: zero width, around an estimate that is certainly wrong. The sum of 100 Bernoulli(0.01) draws is closer to Poisson(1) than to normal. Use an exact (Clopper–Pearson) or Wilson interval; for zero successes the "rule of three" gives an approximate 95% upper bound of $3/n = 0.03$.

Work it by hand · a statistic that isn't a sum

Data Uniform(0, θ); estimate θ by the sample max $M$. $P(M \le m) = (m/\theta)^n$, so $P\big(n(\theta - M)/\theta > t\big) = (1 - t/n)^n \to e^{-t}$. The scaled error is exponential (one-sided, never negative), and it shrinks like $1/n$ rather than $1/\sqrt n$. With $n = 20$ and $\theta = 10$, the typical miss is about $\theta/n = 0.5$, always below the truth. A symmetric $M \pm 1.96\,\widehat{\text{se}}$ interval is the wrong shape entirely.

Story proof · why "none of the pieces dominates" is the real condition

The bell shape comes from many small, independent nudges canceling and compounding. If one nudge is as big as all the others combined (a single $38k truck in a sample of five cars, one Cauchy draw of size 10,000), the total looks like that one nudge, whatever its shape. If the nudges move together (dependence), they don't cancel, so there are effectively far fewer of them. And a maximum isn't built from nudges at all: it's decided by a single observation. The CLT is a statement about crowds, and each failure is a crowd that isn't really one.

Think about itThe CLT says that with enough data, the data become normally distributed. True?

Common mistake: Yes, that's why everything is a bell curve.

The real answer: No. The data stay whatever shape they are; their histogram converges to $F$ (Glivenko–Cantelli), and car prices stay right-skewed at any $n$. The CLT is about the sampling distribution of the average (or a sum). It's the averages from many samples that pile up in a bell.

Think about itIs $n \ge 30$ enough for the CLT?

Common mistake: Yes, 30 is the rule.

The real answer: There is no universal $n$. Berry–Esseen says the error depends on $E|X-\mu|^3/\sigma^3$, the skew and heaviness of the tails. For car prices at $n = 30$, a nominal 95% interval covers 92%; for nearly symmetric data, $n = 10$ is plenty; for Cauchy data, no $n$ is enough.

Check your understandingDoes the CLT apply to the sample median?

Not the CLT for sums directly, but the median is still asymptotically normal: $\hat m \approx N\big(m, 1/(4nf(m)^2)\big)$, where $f(m)$ is the density at the true median (a standard result, not in AoS Ch 5). The SE depends on an unknown density, which is exactly why AoS and CASI reach for the bootstrap to get it.

The delta method

You often care about a smooth function of an average: a log, a ratio, a square. Zoom in far enough on a smooth curve and it looks like a straight line. Near the truth, $g(\bar X)$ is approximately a linear function of $\bar X$, and a linear function just rescales the SE by the slope.

Step 1: a linear function rescales the SE. If $Y = a + bX$, adding $a$ shifts every value by the same amount, which changes no distances, and multiplying by $b$ stretches every distance by $|b|$. So $\text{SD}(Y) = |b|\,\text{SD}(X)$. On the car prices, $\text{se}(2\bar x + 3) = 2 \times 0.85 = 1.70$; the $+3$ does nothing to the spread.

Step 2: a smooth curve is nearly a straight line over a short distance. The slope of that line at $\mu$ is the derivative $g'(\mu)$: how much $g$ changes per unit change in $x$, right at $\mu$. Near the truth,

$$g(\bar X) \approx g(\mu) + g'(\mu)\,(\bar X - \mu),$$

which is a linear function of $\bar X$ with slope $g'(\mu)$. Step 1 then gives $\text{se}\big(g(\bar X)\big) \approx |g'(\mu)|\,\text{se}(\bar X)$, and in practice you plug in $\bar x$ for $\mu$. Check the straight-line approximation for $g(x) = x^2$ near 5, where the slope is 10:

$x$	exact $x^2$	straight line $25 + 10(x - 5)$	error
5.1	26.01	26.0	0.01
5.5	30.25	30.0	0.25
6	36	35	1

For small wobbles the line is nearly exact; the error grows with the square of the distance from $\mu$. The delta method is only as good as the straight line is over the range where $\bar X$ actually wobbles.

Formal · AoS Thm 5.13, Thm 9.24; CASI (2.10) and §2.3 note 1

If $\sqrt n(Y_n - \mu)/\sigma \rightsquigarrow N(0,1)$ and $g$ is differentiable with $g'(\mu) \ne 0$, then

$$\frac{\sqrt n\,\big(g(Y_n) - g(\mu)\big)}{|g'(\mu)|\,\sigma} \rightsquigarrow N(0,1), \qquad \text{i.e. } g(Y_n) \approx N\!\Big(g(\mu),\ \frac{g'(\mu)^2\sigma^2}{n}\Big).$$

In SE form: $\widehat{\text{se}}\big(g(\hat\theta)\big) \approx |g'(\hat\theta)|\,\widehat{\text{se}}(\hat\theta)$. CASI's version: $\text{se}(\bar x^2) \doteq 2|\bar x|\,\widehat{\text{se}}$.

Work it by hand · four transformations of the toy-data mean

Toy data: $\bar x = 5$, $\widehat{\text{se}} = 1$. Multiply the SE by $|g'(5)|$.

$g(\bar x)$	$g'(x)$	$\|g'(5)\|$	$\widehat{\text{se}}(g(\bar x))$
$\bar x^2 = 25$	$2x$	10	10
$\sqrt{\bar x} = 2.24$	$\frac{1}{2\sqrt x}$	0.224	0.22
$\log \bar x = 1.61$	$1/x$	0.2	0.20
$1/\bar x = 0.2$	$-1/x^2$	0.04	0.04

Squaring stretches the wobble tenfold; log and reciprocal shrink it. (The log row says: an SE of 1 on a mean of 5 is a 20% relative error.)

Work it by hand · checking the delta method against the exact answer

The tiny population $\{1, 3, 5\}$ with samples of 2 has a sampling distribution we know exactly (Section 1): $\bar X = 1, 2, 3, 4, 5$ with probabilities $\frac19, \frac29, \frac39, \frac29, \frac19$, $\mu = 3$, and $\text{se}(\bar X) = \sqrt{4/3} = 1.15$. Push each of the five values through $g$, take the SD of the results with the same weights, and compare with the delta method.

$g(\bar x)$	values at $\bar x = 1, \dots, 5$	slope at $\mu = 3$	delta-method SE	exact SE
$\bar x^2$	1, 4, 9, 16, 25	6	$6 \times 1.15 = 6.93$	7.09
$\log \bar x$	0, 0.69, 1.10, 1.39, 1.61	$1/3$	$1.15/3 = 0.38$	0.46
$1/\bar x$	1, 0.50, 0.33, 0.25, 0.20	$-1/9$	$1.15/9 = 0.13$	0.23

The square is 2% off. The log is 16% low, and the reciprocal is nearly 2× too small. Here $\bar X$ wobbles $\pm 1.15$ around 3, which is a wide range compared with how sharply $\log x$ and $1/x$ bend near small $x$. Look at the $1/\bar x$ row: the drop from $\bar x = 1$ to 2 is 0.50, while the drop from 4 to 5 is 0.05. A single straight line can't follow that, so the delta method misses the big values at the left. With more data the wobble shrinks, the straight line fits the curve over a shorter range, and the approximation improves. The delta method is a large-$n$ result.

(For the square, the exact SE is $\sqrt{E[\bar X^4] - (E[\bar X^2])^2}$: $E[\bar X^2] = \frac{1 + 8 + 27 + 32 + 25}{9} = \frac{93}{9}$, $E[\bar X^4] = \frac{1 + 32 + 243 + 512 + 625}{9} = 157$, so $\sqrt{157 - 106.8} = \sqrt{50.2} = 7.09$.)

Pitfall · when the delta method breaks

The SE is large compared with the curvature. That is the $1/\bar x$ row above. A quick check: compute $g(\bar x - \widehat{\text{se}})$ and $g(\bar x + \widehat{\text{se}})$. If the two changes from $g(\bar x)$ are far from equal in size, the curve isn't straight over the range that matters. For $1/\bar x$ at 3 with $\text{se} = 1.15$: $1/1.85 - 1/3 = 0.21$ up, $1/3 - 1/4.15 = 0.09$ down. That's lopsided by more than a factor of 2, so don't trust the delta method here.

The slope at the truth is zero. For $g(x) = x^2$ with $\mu = 0$, the slope is 0, so the formula says $\text{se} = 0$. That is false: $\bar X^2$ still varies. The straight line is flat, and the variation comes entirely from the curvature that the line ignores. This is why the formal statement requires $g'(\mu) \ne 0$. (The fix is a second-order expansion; AoS doesn't cover it.)

$g$ isn't smooth at the truth. $|x|$ at 0, a maximum of several means, a threshold rule: there's no single slope to use. Use the bootstrap (Section 3), which needs no derivative.

Worked examples

AoS Ex 5.14: $W_n = e^{\bar X_n}$. Since $g'(s) = e^s$, $W_n \approx N(e^\mu, e^{2\mu}\sigma^2/n)$.

AoS Ex 9.25 (log-odds): $\psi = \log(p/(1-p))$, $g'(p) = 1/(p(1-p))$, so $\widehat{\text{se}}(\hat\psi) = \frac{1}{\hat p(1-\hat p)}\sqrt{\hat p(1-\hat p)/n} = 1/\sqrt{n\hat p(1-\hat p)}$.

CASI gfr: $\bar x = 54.25$, $\widehat{\text{se}} = 0.95$: $\text{se}(\bar x^2) \approx 2(54.25)(0.95) \approx 103$ for an estimate of about 2943.

Analogy · the funhouse mirror

Push a small wobble through a function. Where the function is steep, the wobble gets stretched; where it's flat, the wobble gets squashed. The slope at the truth is the stretch factor. That's the whole delta method.

CASI's point: the delta method is a plug-in at the end. You derive $|g'(\theta)|\sigma/\sqrt n$ with theory, then substitute $\hat\theta$ for $\theta$ and $s$ for $\sigma$. Keep that in mind for the table in Section 3.

Why it matters · real case · ratio metrics in A/B tests

Click-through rate is total clicks divided by total page views. Online experiments randomize users, not page views, and one user's page views are correlated: the same person tends to click a lot or a little. Treating every page view as an independent Bernoulli trial gives an SE that is too small, so tests "find" effects too often. Deng, Knoblich and Lu at Microsoft describe the standard fix: write the metric as a ratio of two per-user averages, $R = \bar Y/\bar X$ (clicks per user over views per user), and get its SE from the two-variable delta method. Deng et al., KDD 2018

$$V(R) \approx \frac{1}{n\,\mu_X^2}\Big(\sigma_Y^2 - 2R\,\sigma_{XY} + R^2\sigma_X^2\Big).$$

Every quantity in it is a per-user mean, variance or covariance, so the unit of analysis matches the unit of randomization. It is the same move as the paired comparison in Section 6: compute the SE at the level where the observations are independent.

Think about itThe car prices have $\bar x = 8.59$ with SE 0.85. What's the SE of $2\bar x + 3$? Of $\bar x^2$?

Common mistake: Both are 0.85, or both are complicated.

The real answer: $2\bar x + 3$: the +3 shifts without spreading, the ×2 doubles the spread, so $2(0.85) = 1.70$. $\bar x^2$: the delta method gives $2|\bar x| \times 0.85 = 2(8.59)(0.852) \approx 14.6$. The square stretches wobble by its slope at 8.59, which is about 17.

3 · Five ways to get a standard error

Go deeper: expanded page with more worked examples and interactive widgets →

Every SE formula has a gap: the true SE depends on the unknown $F$. CASI §2.1 lists the "ingenious devices" frequentists use to fill it. This section takes them in CASI's order, with AoS's treatment folded in. The summary table comes first so the section has a map.

Device	What it computes	Where the unknown $F$ is replaced	Book
1 · Plug-in	a formula like $\sqrt{\text{var}_F(X)/n}$	at the end: $\text{var}_F$ → sample variance	CASI (2.8)–(2.9); AoS Ch 7
2 · Taylor / delta	$\|g'(\theta)\|\,\sigma/\sqrt n$	at the end: $\theta \to \hat\theta$	CASI (2.10); AoS 5.13, 9.24
3 · MLE theory	$1/\sqrt{n\,I(\theta)}$	at the end: $\theta \to \hat\theta$	CASI (4.27); AoS 9.18
4 · Bootstrap	SD of $t(X^)$, $X^ \sim \hat F$	first: simulate from $\hat F$	CASI §10.2; AoS Ch 8
5 · Pivotal statistic	an exact distribution, e.g. $t_{n-1}$	nowhere: the unknown cancels	CASI (2.17)–(2.19)

The jackknife, which CASI calls "a first step toward a computation-based, non-formulaic approach," sits between devices 2 and 4: it computes the derivatives of device 2 numerically.

Device 1 · The plug-in principle and the empirical CDF

The parameter is a feature of the population: $\theta = T(F)$. The plug-in estimate computes the same feature of the sample: $\hat\theta = T(\hat F_n)$. The sample is treated as a miniature population.

Formal · AoS Def 7.1, Thms 7.3–7.5

The empirical distribution function puts mass $1/n$ on each data point:

$$\hat F_n(x) = \frac{1}{n}\sum_{i=1}^n I(X_i \le x).$$

Thm 7.3: at each fixed $x$, $E[\hat F_n(x)] = F(x)$, $V[\hat F_n(x)] = F(x)(1-F(x))/n$, so $\hat F_n(x) \xrightarrow{P} F(x)$. (Each $I(X_i \le x)$ is a Bernoulli($F(x)$); $\hat F_n(x)$ is a sample proportion.)
Thm 7.4 (Glivenko–Cantelli): $\sup_x |\hat F_n(x) - F(x)| \xrightarrow{P} 0$ (in fact almost surely). The whole curve converges, not just each point.
Thm 7.5 (DKW): for any $\epsilon > 0$, $P\big(\sup_x |\hat F_n(x) - F(x)| > \epsilon\big) \le 2e^{-2n\epsilon^2}$. So $\hat F_n \pm \epsilon_n$ with $\epsilon_n = \sqrt{\log(2/\alpha)/(2n)}$ is a $1-\alpha$ confidence band for the whole CDF.

Work it by hand · the ECDF of five points

Toy data $\{2, 4, 5, 6, 8\}$. $\hat F_5(x)$ jumps by $1/5$ at each point: 0.2 at 2, 0.4 at 4, 0.6 at 5, 0.8 at 6, 1.0 at 8.

Thm 7.3 at $x = 5$: $\hat F(5) = 3/5 = 0.6$ (three points are $\le 5$), with $\widehat{\text{se}} = \sqrt{0.6 \times 0.4/5} = 0.22$. It's just a proportion.
DKW band: with $\log 40 \approx 3.7$ (natural log), $\epsilon_5 = \sqrt{\log 40/10} \approx \sqrt{0.37} = 0.61$. With five points, the 95% band for the whole CDF is ±0.61, nearly useless. For ±0.05 you need $n = \log 40/(2 \times 0.05^2) \approx 3.7/0.005 = 740$.

Worked example · DKW band for the cars

$n = 92$, $\alpha = 0.05$: $\epsilon_n = \sqrt{\log 40 / 184} \approx \sqrt{3.7/184} \approx \sqrt{0.020} = 0.14$. The true CDF of car prices lies within ±0.14 of the ECDF, everywhere at once, with 95% confidence. (AoS Ex 7.6 does this for the 799 nerve-pulse waiting times.)

Formal · AoS Defs 7.7–7.8, Thm 7.9

The plug-in estimator of $\theta = T(F)$ is $\hat\theta_n = T(\hat F_n)$. A functional is linear if $T(F) = \int r(x)\,dF(x)$; its plug-in estimate is just an average, $T(\hat F_n) = \frac1n\sum r(X_i)$.

AoS then adds: if you can show (or assume) $\widehat{\text{se}}$ is a reasonable estimate and $T(\hat F_n) \approx N(T(F), \widehat{\text{se}}^2)$, the normal-based interval $T(\hat F_n) \pm z_{\alpha/2}\widehat{\text{se}}$ follows. For linear functionals, $\widehat{\text{se}}$ is the sample SD of the $r(X_i)$ divided by $\sqrt n$.

Work it by hand · plug-in everything

Treat $\{2, 4, 5, 6, 8\}$ as the whole population (each value with probability 1/5) and compute the population feature.

Parameter	Plug-in calculation	Value
mean	$25/5$	5
$E[X^2]$ (linear functional, $r(x) = x^2$)	$(4 + 16 + 25 + 36 + 64)/5$	29
variance	$29 - 5^2$	4 (divisor $n$)
$P(X > 5)$	$r(x) = I(x > 5)$: two of five	0.4
median	middle value	5
skewness	cubed deviations $-27, -1, 0, 1, 27$ sum to 0	0

For a linear functional, the plug-in SE is the sample SD of the $r(x_i)$ over $\sqrt n$. For $P(X > 5)$: $r = (0, 0, 0, 1, 1)$, giving $\sqrt{0.4 \times 0.6/5} = 0.22$.

Parameter	$T(F)$	Plug-in $T(\hat F_n)$	AoS
Mean	$\int x\,dF$	$\bar X_n$	Ex 7.10
Variance	$\int x^2 dF - (\int x\,dF)^2$	$\frac1n\sum(X_i - \bar X_n)^2$ (divisor $n$)	Ex 7.11
Skewness	$E(X-\mu)^3/\sigma^3$	$\frac1n\sum(X_i-\bar X)^3/\hat\sigma^3$	Ex 7.12
Correlation	$\text{Cov}(X,Y)/(\sigma_X\sigma_Y)$	sample correlation	Ex 7.13
Quantile	$F^{-1}(p)$	$\hat F_n^{-1}(p) = \inf\{x : \hat F_n(x) \ge p\}$	Ex 7.14

Analogy · the scale model

An architect can't wind-tunnel a real skyscraper, so she tests a scale model and assumes the real building behaves the same way. The sample is a scale model of the population. Plug-in means: whatever you would compute on the population, compute on the model.

For the mean, plug-in gives $\widehat{\text{se}} = \hat\sigma/\sqrt n$ with divisor $n$; textbooks usually swap in $s$ with divisor $n-1$. CASI's version uses the unbiased variance CASI (2.9). At $n = 92$ the difference is 0.5%. CASI's coin footnote is a nice concrete case: the true SE is $\sqrt{\theta(1-\theta)/n}$, and all we can ever report is $\sqrt{p(1-p)/n}$.

Check your understandingWhy does the plug-in principle need $n$ to be large, if $\hat F_n$ is unbiased for $F$ at every point?

Being right on average at every point doesn't make $T(\hat F_n)$ close to $T(F)$ for a given sample. The plug-in SE replaces $F$ by one noisy copy of it, and functionals that depend on the tails (variance, skewness, extremes) are sensitive to exactly the parts of $F$ a small sample sees least. DKW quantifies the curve's error at $\sqrt{\log(2/\alpha)/(2n)}$, which is ±0.14 even at $n = 92$.

Device 2 · Taylor series

See the delta method. CASI's historical aside: before computers, applied statisticians had to be "Taylor series experts," laboriously differentiating every new statistic. The formula for the SE of a correlation coefficient CASI (10.10) is a page of moments; the jackknife and bootstrap made that expertise unnecessary.

Device 3 · Maximum likelihood and Fisher information

In a parametric model, maximum likelihood comes with its own SE formula. The SE depends on how sharply the log-likelihood peaks: a sharp peak pins the parameter down, a flat one doesn't.

Formal · AoS Defs 9.7–9.8, 9.16; Thms 9.17–9.18; CASI (4.14)–(4.27)

Likelihood $\mathcal L_n(\theta) = \prod_i f(X_i;\theta)$; log-likelihood $\ell_n(\theta) = \sum_i \log f(X_i;\theta)$; MLE $\hat\theta_n = \arg\max_\theta \ell_n(\theta)$.

Score: $s(X;\theta) = \partial \log f(X;\theta)/\partial\theta$, which has mean 0. Fisher information:

$$I(\theta) = V_\theta\big(s(X;\theta)\big) = -E_\theta\Big[\frac{\partial^2 \log f(X;\theta)}{\partial\theta^2}\Big], \qquad I_n(\theta) = n\,I(\theta).$$

Asymptotic normality (AoS Thm 9.18; CASI (4.27)):

$$\frac{\hat\theta_n - \theta}{\text{se}} \rightsquigarrow N(0,1), \quad \text{se} \approx \sqrt{1/I_n(\theta)}; \qquad \text{and also with } \widehat{\text{se}} = \sqrt{1/I_n(\hat\theta_n)}.$$

So $\hat\theta_n \pm z_{\alpha/2}\,\widehat{\text{se}}$ is an approximate $1-\alpha$ interval (AoS Thm 9.19).

Work it by hand · Fisher information on five counts

Treat $\{2, 4, 5, 6, 8\}$ as Poisson counts. Log-likelihood $\ell(\lambda) = (\sum x_i)\log\lambda - n\lambda = 25\log\lambda - 5\lambda$ (plus a constant).

Slope $\ell'(\lambda) = 25/\lambda - 5 = 0 \Rightarrow \hat\lambda = 5$.
Curvature $\ell''(\lambda) = -25/\lambda^2$, which is $-1$ at $\lambda = 5$. Observed information 1.
$\widehat{\text{se}} = 1/\sqrt{1} = 1$. Same as $\sqrt{\hat\lambda/n} = \sqrt{5/5}$ (AoS Ex 9.22), and here the same as $s/\sqrt n$, because these data happen to have $s^2 = \bar x$.

Bernoulli version: $\{1, 0, 1, 1, 0\}$: $\hat p = 0.6$, $I(p) = 1/(p(1-p))$, $\widehat{\text{se}} = \sqrt{0.6 \times 0.4/5} = 0.22$.

Story proof · CASI's two-step heuristic for (4.27)

At the MLE the slope of the log-likelihood is zero. Expand the slope around the truth: $0 = \dot\ell(\hat\theta) \approx \dot\ell(\theta) + \ddot\ell(\theta)(\hat\theta - \theta)$, so $\hat\theta - \theta \approx \dot\ell(\theta)/(-\ddot\ell(\theta))$. The numerator is a sum of $n$ independent mean-zero scores, so the CLT makes it about $N(0, nI)$. The denominator is a sum of $n$ curvatures, so the LLN makes it about $nI$. Normal over constant: $\hat\theta \approx N(\theta, 1/(nI))$. This one derivation uses both limit theorems.

Worked examples · all hand-workable

Normal mean, σ known (CASI (4.28)–(4.31)): $\ell(\theta) = -\frac12\sum(x_i - \theta)^2/\sigma^2 + \text{const}$; $\dot\ell = \sum(x_i - \theta)/\sigma^2$, $\ddot\ell = -n/\sigma^2$. So $\hat\theta = \bar x$ and $I = 1/\sigma^2$, giving $\text{se} = \sigma/\sqrt n$. Fisher's machinery reproduces the plug-in answer.

Bernoulli (AoS Ex 9.20): $I(p) = 1/(p(1-p))$, so $\widehat{\text{se}} = \sqrt{\hat p(1-\hat p)/n}$, the same as Section 1.

Poisson (AoS Ex 9.22): $\hat\lambda = \bar X$, $I(\lambda) = 1/\lambda$, $\widehat{\text{se}} = \sqrt{\hat\lambda/n}$.

Analogy · finding a mountaintop in fog

The log-likelihood is a mountain and the MLE is its summit. On a sharp peak, a small step off the top drops you a long way, so you know exactly where the top is. On a broad plateau you could be anywhere near the top. Fisher information is the peak's sharpness (its curvature); the SE is roughly one over the square root of that sharpness.

Further results to know exist: the MLE is consistent (AoS Thm 9.13), equivariant (Thm 9.14: the MLE of $g(\theta)$ is $g(\hat\theta)$, which CASI calls the plug-in rule (4.3)), and efficient (Thm 9.23; CASI's Cramér–Rao bound (4.33): no unbiased estimator has variance below $1/(nI)$). The multiparameter version replaces $1/I$ with the inverse of the Fisher information matrix (AoS §9.10). That is where a regression table's standard errors come from when the model is fit by maximum likelihood.

Pitfall · many parameters

CASI §4.1: with thousands of parameters, each may be well estimated while a summary of them is badly biased. For the prostate data, the MLE of the largest of 6033 gene effects is 5.29, "almost certainly a flagrant overestimate" (the winner's curse). Plug-in of a maximum is biased upward.

Think about itTwo log-likelihood curves peak at the same $\hat\theta$. One is a sharp spike, one a broad hill. Which estimate has the smaller SE?

Common mistake: The same, since they have the same peak.

The real answer: The sharp one. Sharpness is curvature, curvature is Fisher information, and $\text{se} \approx 1/\sqrt{\text{information}}$. A sharp peak means nearby parameter values make the data much less likely, so the data rule them out.

Check your understandingWork it: the SE of the MLE for an exponential rate $\lambda$.

$\log f = \log\lambda - \lambda x$; second derivative $-1/\lambda^2$; so $I(\lambda) = 1/\lambda^2$ and $\text{se} = \lambda/\sqrt n$. The MLE is $\hat\lambda = 1/\bar x$, so $\widehat{\text{se}} = \hat\lambda/\sqrt n$. Cross-check with the delta method on $g(\bar x) = 1/\bar x$: $|g'| = 1/\bar x^2$, and $\text{se}(\bar x) = \sigma/\sqrt n = 1/(\lambda\sqrt n)$, giving $\lambda^2 \cdot 1/(\lambda\sqrt n) = \lambda/\sqrt n$. Two devices, same answer.

Device 4 · The bootstrap

The bootstrap estimates the SE of any statistic by simulation. Pretend the sample is the population, draw new samples from it, recompute the statistic on each, and take the SD of the results. In CASI's phrase, the bootstrap "substitutes an estimate $\hat F$ for $F$ and then estimates the frequentist standard error by direct simulation."

Formal · AoS §8.2 (8.1); CASI (10.13)–(10.20)

Real world: $F \to X_1,\dots,X_n \to \hat\theta = s(X)$. Bootstrap world: $\hat F_n \to X_1^*,\dots,X_n^* \to \hat\theta^* = s(X^*)$.

Draw $X_1^*,\dots,X_n^*$ from $\hat F_n$: that is, draw $n$ points with replacement from the data.
Compute $\hat\theta^* = s(X^*)$.
Repeat $B$ times to get $\hat\theta^*_1,\dots,\hat\theta^*_B$.
$\widehat{\text{se}}_{\text{boot}} = \Big[\sum_{b=1}^B(\hat\theta^*_b - \hat\theta^*_\cdot)^2/(B-1)\Big]^{1/2}$, with $\hat\theta^*_\cdot = \frac1B\sum_b\hat\theta^*_b$.

CASI frames the ideal bootstrap SE as a plug-in: if $\text{Sd}(F)$ is the true SD of $\hat\theta$ as a function of $F$, then $\widehat{\text{se}}_{\text{boot}} = \text{Sd}(\hat F)$ CASI (10.20). AoS states the same in two steps: Step 1 estimate $V_F(T_n)$ by $V_{\hat F_n}(T_n)$; Step 2 approximate $V_{\hat F_n}(T_n)$ by simulation.

Work it by hand · the ideal bootstrap of {1, 3, 5}

With $n = 3$ there are only 10 distinct resamples, so the bootstrap distribution can be listed exactly. Counts are how many copies of (1, 3, 5) the resample contains. Probabilities are multinomial: $3!/(N_1!N_2!N_3!)$ out of 27.

counts of (1,3,5)	resample	mean	median	probability
(3,0,0)	1,1,1	1	1	1/27
(0,3,0)	3,3,3	3	3	1/27
(0,0,3)	5,5,5	5	5	1/27
(2,1,0)	1,1,3	5/3	1	3/27
(2,0,1)	1,1,5	7/3	1	3/27
(1,2,0)	1,3,3	7/3	3	3/27
(0,2,1)	3,3,5	11/3	3	3/27
(1,0,2)	1,5,5	11/3	5	3/27
(0,1,2)	3,5,5	13/3	5	3/27
(1,1,1)	1,3,5	3	3	6/27

Mean: the bootstrap variance is $8/9$, so $\widehat{\text{se}}_{\text{boot}} = 0.94$. That's exactly the plug-in $\hat\sigma/\sqrt n = \sqrt{(8/3)/3}$: for the mean, the ideal bootstrap is the plug-in formula.
Median: $P(1) = 7/27$, $P(3) = 13/27$, $P(5) = 7/27$. Variance $\frac{4 \cdot 7 + 0 + 4 \cdot 7}{27} = \frac{56}{27}$, so $\widehat{\text{se}}_{\text{boot}} = 1.44$. No formula needed.

There are two separate approximations, and it helps to name them. Replacing $F$ by $\hat F_n$ is the statistical error; it shrinks only as $n$ grows. Using finite $B$ is simulation error; it shrinks as $B$ grows and costs only computer time. CASI's guidance: $B = 200$ is usually enough for a standard error; 1000–2000 for confidence intervals.

CASI also notes that $\hat F_n$ is the nonparametric MLE of $F$: of all distributions, it makes the observed sample most probable. So the bootstrap's plug-in step is itself a maximum-likelihood move.

Analogy · tasting soup

You can't cook a hundred pots of soup to see how much the saltiness varies from batch to batch. But you have one pot. Ladle out a bowl, taste it, pour it back, stir, ladle again. Because each ladle draws from the same pot with replacement, the bowls vary the way bowls from that pot vary. The bootstrap assumes your pot is a fair stand-in for every pot the recipe could have produced.

Sampling distribution vs. bootstrap distribution

They're the same kind of object built from different sources. The sampling distribution draws new samples from the population; the bootstrap distribution re-draws from your one sample. The bootstrap copies the sampling distribution's spread and shape, never its location, and even the spread is only as good as your sample is representative.

	Sampling distribution	Bootstrap distribution
Samples come from	the population $F$ (unknown)	your sample, treated as the population ($\hat F_n$)
How many exist	one, fixed by $F$ and $n$	one per dataset: every sample gives a different one
Centered at	the true $\theta$ (if unbiased)	your estimate $\hat\theta$
Spread	the true SE	an estimate of the SE
Values it can take	anything the population can produce	only combinations of your data points
Error	none: it is the truth	statistical ($\hat F_n \ne F$, shrinks with $n$) plus Monte Carlo (finite $B$, shrinks with $B$)

What's the same. The bootstrap bets that the bootstrap miss $\hat\theta^* - \hat\theta$ behaves like the real miss $\hat\theta - \theta$. That's why its SD estimates the SE and its shape estimates the shape of your error (the basis of the pivotal interval). It never tells you where $\theta$ is: recentering on $\hat\theta$ does not move your estimate toward the truth.

Work it by hand · one sampling distribution, three bootstrap distributions

Population $\{1, 3, 5\}$, samples of size 3. The sampling distribution of the mean is fixed: the 27 equally likely samples give means 1, 5/3, 7/3, 3, 11/3, 13/3, 5 with counts 1, 3, 6, 7, 6, 3, 1, centered at 3 with SE $\sqrt{8/9} = 0.94$.

Three students each draw one sample and bootstrap it. For the mean, the ideal bootstrap SE is the plug-in $\hat\sigma/\sqrt n$ of their own sample:

student's sample	bootstrap center $\bar x$	plug-in $\hat\sigma^2$	bootstrap SE	vs. truth (3, 0.94)
$\{1, 3, 5\}$	3	$(4 + 0 + 4)/3 = 8/3$	$\sqrt{8/9} = 0.94$	exact match
$\{3, 5, 5\}$	13/3 = 4.33	$(16/9 + 4/9 + 4/9)/3 = 8/9$	$\sqrt{8/27} = 0.54$	center off by 1.33; SE 40% too small
$\{3, 3, 3\}$	3	0	0	every resample is $\{3, 3, 3\}$: a single spike

Each bootstrap distribution estimates the one sampling distribution; how well depends on how representative the sample is. The $\{3, 3, 3\}$ row is the extreme case.

Pitfall · where the two come apart

Small $n$: $\hat F_n$ is a rough copy of $F$, so the bootstrap SE is itself noisy (Section 4).
Missing tails: if your sample lacks the rare $38k truck, no resample will ever contain it.
Extremes: the bootstrap distribution of the maximum can't exceed your largest data point, so it piles up there and fails.
Dependence: resampling single points destroys correlation that the sampling distribution has; resample blocks instead (moving blocks).
Divisor $n$: for the mean, the bootstrap SE is $\hat\sigma/\sqrt n$, slightly small at small $n$: 0.89 vs. 1 on the toy data.

Analogy · many pots vs. many ladles

The sampling distribution is cooking a thousand pots from the same recipe and tasting each. The bootstrap is tasting many ladles from the one pot you made. If your pot is typical, ladles vary the way pots vary. If you oversalted your pot, every ladle is oversalted, and no amount of ladling reveals it.

Think about itTwo students bootstrap the mean of their own samples from the same population. One gets SE 0.94, the other 0.54. Did one of them make a mistake?

Common mistake: Yes, the bootstrap should give the same answer for the same population.

The real answer: Neither. Each bootstrap distribution is built from that student's sample, so it is an estimate of the one true sampling distribution, and estimates vary. Here the population is $\{1, 3, 5\}$: the sample $\{1, 3, 5\}$ gives 0.94 (exactly right), and $\{3, 5, 5\}$ gives 0.54 (too small, because it happens to be less spread out). The disagreement is the bootstrap's own sampling variability.

Story proof · each resample leaves out about 37% of the data

Fix one data point. On each of the $n$ draws, the chance of not picking it is $1 - 1/n$. The draws are independent, so the chance it is never picked is $(1 - 1/n)^n$, which tends to $e^{-1} \approx 0.368$. For the 92 cars, $(1 - 1/92)^{92} = 0.366$. So every bootstrap sample is missing about a third of the original points, with others appearing two or three times. That is exactly why resamples vary, and why they vary by about the right amount. (CASI (10.34): the count $N_i$ of copies of point $i$ is Binomial$(n, 1/n)$, mean 1, variance $(n-1)/n$.)

Story proof · how many different bootstrap samples are there?

A bootstrap sample is determined by how many copies of each of the $n$ points it contains: $n$ indistinguishable draws placed into $n$ labeled bins. Stars and bars: $\binom{2n-1}{n}$ ways. For $n = 3$ that's 10 (CASI Fig 10.3); for $n = 10$, 92,378; for $n = 20$, $6.9 \times 10^{10}$ CASI (10.38). You could never enumerate them, which is why we sample $B$ of them at random. For $n = 3$, the resample $\{x_1, x_2, x_3\}$ itself has probability $3!/3^3 = 6/27$, each "all one point" resample has $1/27$, and the six mixed ones $3/27$ each.

From the books · results to quote

Law school (AoS Ex 8.6, Efron's original example): 15 schools, correlation of LSAT and GPA $\hat\theta = 0.776$; bootstrap $\widehat{\text{se}} = 0.137$ ($B = 1000$). Normal interval $0.78 \pm 2(0.137) = (0.51, 1.00)$; percentile interval $(0.46, 0.96)$.
Nerve data skewness (AoS Ex 8.2): bootstrap $\widehat{\text{se}} = 0.16$ for the plug-in skewness.
Patch bioequivalence (AoS Ex 8.7): ratio $\hat\theta = \bar Y/\bar Z = -452.3/6342 = -0.0713$, bootstrap $\widehat{\text{se}} = 0.105$, 95% interval $(-0.24, 0.15)$, not inside the required $(-0.20, 0.20)$, so bioequivalence is not demonstrated.
Kidney lowess (CASI §1.1): no formula exists for the accuracy of a lowess curve; $B = 250$ bootstrap curves give SEs roughly double the linear-regression SEs (0.71 vs 0.21 at age 20). Flexibility costs precision.
Course, car prices: bootstrap SE of the mean 0.849 vs formula 0.852; bootstrap SE of the median 0.746, where no formula exists.

CASI's summary of the bootstrap's virtues: it is completely automatic (one master program works for any statistic), it "shakes the data more violently" than the jackknife, so it doesn't rely on smoothness, and nothing restricts it to standard errors. The same replicates estimate bias, absolute error, or quantiles.

Pitfall · what the bootstrap can't fix

The bootstrap inherits every flaw of $\hat F_n$. If the sample isn't representative, resampling it faithfully reproduces the bias. If the data are dependent, resampling individual points destroys the dependence and understates the SE (see moving blocks). It also fails for statistics that depend on extreme order statistics, such as the sample maximum.

Why it matters · scenario · an SLA on the 95th percentile

Illustrative numbers. A service promises that 95% of requests finish within 300 ms. From 200 logged requests, the 95th percentile (the 190th smallest value) is 280 ms. Is that comfortably under target? No textbook SE formula for a sample percentile avoids the unknown density, the same problem as the median in Section 2. The bootstrap gives an SE directly: resample the 200 requests, recompute the 190th value, repeat.

By hand, the intuition. The 190th value has only 10 requests above it, so the estimate is governed by a handful of slow requests. A resample that happens to include two extra copies of one slow outlier moves it a lot. That is why tail percentiles need far more data than means for the same precision.

And the limit. If all 200 requests came from a quiet Sunday, resampling them reproduces Sunday faithfully. The bootstrap measures sampling noise, not whether Monday looks like Sunday.

Think about itWhy does the bootstrap sample with replacement? What would happen without it?

Common mistake: To get more data.

The real answer: A sample of size $n$ drawn without replacement from $n$ points is just the original data in a different order, so every "resample" gives the same statistic and the SE comes out 0. Replacement is what makes resamples differ, by omitting about 37% of points and repeating others.

Think about itIf I use $B$ = 1,000,000 bootstrap samples instead of 1,000, is my SE now essentially exact?

Common mistake: Yes, more resamples means more accuracy.

The real answer: Only the simulation error goes away. The statistical error, from using $\hat F_n$ in place of $F$, is fixed by $n$, not $B$. A million resamples of 10 data points still describe those 10 points. CASI: $B = 200$ is usually plenty for an SE.

Think about itThe bootstrap distribution is centered at $\hat\theta$, not at $\theta$. So how can it tell us anything about $\theta$?

Common mistake: It can't, or: it moves our estimate toward $\theta$.

The real answer: It isn't used for location. It estimates the shape and spread of the miss $\hat\theta - \theta$ by the shape and spread of $\hat\theta^* - \hat\theta$. That's all an SE or an interval needs. The bootstrap never improves the estimate itself.

Check your understandingMy bootstrap SE came out 0.849 and my partner's came out 0.853. Which is right?

Both; the difference is Monte Carlo error from different random resamples. It shrinks like $1/\sqrt B$ and is tiny next to the SE's own statistical uncertainty (about ±11% here, Section 4). Set a seed so results are reproducible, and raise $B$ if the digits matter.

The jackknife

The jackknife is the bootstrap's older, cheaper cousin (AoS credits Quenouille, 1949). Instead of random resamples, it uses the $n$ datasets you get by leaving out one point at a time.

Formal · CASI (10.6); AoS §8.5.1

Let $\hat\theta_{(i)}$ be the statistic computed without $x_i$, and $\hat\theta_{(\cdot)} = \frac1n\sum_i\hat\theta_{(i)}$. Then

$$\widehat{\text{se}}_{\text{jack}} = \Big[\frac{n-1}{n}\sum_{i=1}^n\big(\hat\theta_{(i)} - \hat\theta_{(\cdot)}\big)^2\Big]^{1/2}.$$

AoS: under suitable conditions $v_{\text{jack}}/V(T_n) \xrightarrow{P} 1$, but unlike the bootstrap, the jackknife does not give consistent SEs for sample quantiles.

Work it by hand · jackknife of the toy data

$\{2, 4, 5, 6, 8\}$, $\bar x = 5$. Leave-one-out means: drop 2 → 23/4 = 5.75; drop 4 → 5.25; drop 5 → 5.00; drop 6 → 4.75; drop 8 → 4.25. Their average is 5. Deviations $\pm 0.75, \pm 0.25, 0$; squares sum to $2(0.5625) + 2(0.0625) = 1.25$.

$\widehat{\text{se}}^2_{\text{jack}} = \frac45 \times 1.25 = 1$, so $\widehat{\text{se}}_{\text{jack}} = 1$, exactly $s/\sqrt n$. Each leave-one-out mean moves by a quarter of the dropped point's deviation, $(\bar x - x_i)/(n-1)$.

Story proof · why the strange factor $(n-1)/n$

Two leave-one-out samples share $n - 2$ of their points, so the $\hat\theta_{(i)}$ are much closer together than independent replications would be. For the mean, $\bar x_{(i)} - \bar x = (\bar x - x_i)/(n-1)$: each leave-one-out mean differs from the full mean by only $1/(n-1)$ of that point's deviation. Squaring and summing gives $\sum(x_i - \bar x)^2/(n-1)^2$, which is too small by a factor of about $n$. The $(n-1)/n$ factor in front, combined with summing (not averaging), inflates it back exactly: $\widehat{\text{se}}^2_{\text{jack}} = \frac{n-1}{n}\cdot\frac{\sum(x_i-\bar x)^2}{(n-1)^2} = \frac{s^2}{n}$. CASI says plainly that the "fudge factor" was chosen to make this agree with (1.2) CASI (10.8).

Worked example · jackknife of {2, 4, 9} by hand

Leave-one-out means: drop 2 → 6.5; drop 4 → 5.5; drop 9 → 3.0. Their average is 5.0. Squared deviations: 2.25, 0.25, 4.00; sum 6.5. $\widehat{\text{se}}^2_{\text{jack}} = \frac{2}{3}(6.5) = 4.33$, so $\widehat{\text{se}}_{\text{jack}} = 2.08$. Check: $s^2 = (9 + 1 + 16)/2 = 13$, $s^2/3 = 4.33$. ✓

CASI's examples: for the kidney-data correlation $\hat\theta = 0.572$, the jackknife and bootstrap both gave $\widehat{\text{se}} = 0.058$ and the page-long Taylor formula gave 0.057. For the student-score "eigenratio" $\hat\theta = 0.693$, bootstrap 0.075 vs jackknife 0.083.

CASI lists the jackknife's properties. It is nonparametric and automatic. It works with samples of size $n-1$, which hides an assumption of smooth behavior across sample sizes (worrying for the median, whose definition differs for odd and even $n$). It is biased upward as an estimate of the true SE. And it is secretly a Taylor-series method: $\widehat{\text{se}}_{\text{jack}}$ is proportional to the root-mean-square of numerical derivatives of the statistic in each data point's direction CASI (10.12).

Pitfall · the jackknife needs a smooth statistic

Because it only nudges the data a little, the jackknife sees local slopes. A statistic that jumps fools it. On the car prices, every leave-one-out median is either $5.5k or $5.8k, so the jackknife SE (1.43) is 1.9× the bootstrap's (0.75) course. CASI Fig 10.1 shows the same failure for the lowess curve at age 25, where the jackknife greatly overstates the variability.

Analogy · testing a table's wobble

The jackknife checks how sturdy a table is by removing one leg at a time and seeing how much the top tilts. That works for a solid wooden table. It fails for a table with a hinge: remove most legs and nothing happens, remove one and the top snaps to a new angle. A median is a hinge.

Check your understandingWork it: the jackknife SE of the median of {1, 2, 3, 4, 5}.

Leave-one-out medians: drop 1 → 3.5; drop 2 → 3.5; drop 3 → 3; drop 4 → 2.5; drop 5 → 2.5. Their mean is 3.0; squared deviations sum to 0.25 + 0.25 + 0 + 0.25 + 0.25 = 1.0. $\widehat{\text{se}}^2_{\text{jack}} = \frac45(1.0) = 0.8$, $\widehat{\text{se}} = 0.89$. Notice only three distinct values appear: the median jumps between neighboring data points, which is the non-smoothness that breaks the jackknife.

Resampling plans and variations

CASI §10.3 unifies the jackknife and bootstrap. Keep the data fixed and change only the weights $P = (P_1,\dots,P_n)$ on the points. The original estimate uses equal weights $1/n$. The jackknife uses weights with one zero. The bootstrap uses random weights $N_i/n$ with multinomial counts. The jackknife points sit about $\sqrt n$ times closer to the center than typical bootstrap points CASI (10.33), (10.35), which is why the jackknife sees only local behavior. For a linear statistic such as the mean, the two agree (up to the fudge factor).

Multisample bootstrap (CASI §10.3): with two groups, resample each group separately. For the AML/ALL median difference 0.235, $B = 500$ gives $\widehat{\text{se}} = 0.074$, so the estimate sits 3.18 SEs above zero, close to the t-statistic of 3.01. Resampling the pooled 72 patients would make the group sizes random and add spurious variability.
Moving blocks bootstrap (CASI §10.3): for a time series, resample contiguous blocks of length $m$ so that short-range correlation survives. This is the fix for the dependence failure in Section 9.
Bayesian bootstrap (CASI §10.3): smooth random weights from normalized exponentials instead of multinomial counts.
Parametric bootstrap (AoS §9.11; CASI §10.4): fit a parametric model, then simulate from the fitted model $f(x;\hat\theta)$ instead of from $\hat F_n$. AoS Ex 9.30 uses it for $\tau = \sigma/\mu$ and compares with the delta method.
Regression: resample pairs course. Resampling whole rows $(x_i, y_i)$ makes no assumption that the error variance is constant, which is why on the Ames data the pairs-bootstrap SE agrees with the heteroskedasticity-robust formula and not with the classical one (Section 8). Resampling residuals instead would bake the constant-variance assumption back in.

Device 5 · Pivotal statistics

A pivotal statistic has a distribution that doesn't depend on the unknown parameters at all. When you have one, no plugging in is needed: its distribution applies exactly.

Formal · CASI (2.11)–(2.19)

Two normal samples with a common unknown variance $\sigma^2$. Under $H_0: \mu_1 = \mu_2$, with pooled $\hat\sigma^2 = \big[\sum(x_{1i}-\bar x_1)^2 + \sum(x_{2i}-\bar x_2)^2\big]/(n_1+n_2-2)$,

$$t = \frac{\bar x_2 - \bar x_1}{\hat\sigma\sqrt{1/n_1 + 1/n_2}} \sim t_{n_1+n_2-2}$$

whatever $\sigma$ is. For df = 70, $P_{H_0}(-1.99 \le t \le 1.99) = 0.95$, so $\bar x_2 - \bar x_1 \pm 1.99\,\widehat{\text{sd}}$ is an exact 95% interval for $\mu_2 - \mu_1$.

Work it by hand · a two-sample t

Group 1 $\{2, 4, 6\}$, group 2 $\{5, 7, 9\}$. Means 4 and 7; each group's squared deviations sum to 8.

Pooled $\hat\sigma^2 = (8 + 8)/(3 + 3 - 2) = 4$, $\hat\sigma = 2$.
$\widehat{\text{sd}} = 2\sqrt{1/3 + 1/3} = 1.63$; $t = (7 - 4)/1.63 = 1.84$ on 4 df.
$t_{4,.025} = 2.776$: retain $H_0$ (p = 0.14). Exact 95% interval: $3 \pm 2.776(1.63) = (-1.53, 7.53)$.

Whatever $\sigma$ really is, this interval has exactly 95% coverage under the normal model: that's what pivotal buys.

Analogy · measuring in body lengths

Say a lizard ran 3 meters. Whether that's fast depends on how big the lizard is, which you don't know. Measure the distance in the lizard's own body lengths instead and the unknown size cancels: "it ran 20 body lengths" means the same thing for every lizard. A pivotal statistic measures the estimate in units of its own estimated spread, so the unknown $\sigma$ cancels.

CASI's "strong frequentism" demands exact correctness like this; pivots make it possible, but "unfortunately" they are unavailable in most problems. The other four devices are the realistic, approximate route.

4 · Is the standard error itself uncertain?

Go deeper: expanded page with more worked examples and interactive widgets →

Yes. $\widehat{\text{se}}$ is computed from the same sample as $\hat\theta$, so it is a statistic with its own sampling distribution. CASI says this in its first chapter: $\widehat{\text{se}}$ "is itself an algorithm" and could be subjected to further inference. Neither book makes this a section of its own, but it explains why the t-distribution exists, why small samples need special care, and why bootstrap-t intervals work.

Analogy · a rubber ruler

An interval $\hat\theta \pm 1.96\,\widehat{\text{se}}$ measures uncertainty with a ruler whose length you also estimated from the data. With lots of data the ruler is steel. With five data points it's rubber: sometimes it comes out stretched, sometimes shrunk, and when it's shrunk your interval is too short and you miss.

How noisy is ŝe?

Formal · standard result (normal data)

If $X_i \overset{iid}{\sim} N(\mu, \sigma^2)$, then $(n-1)S^2/\sigma^2 \sim \chi^2_{n-1}$, independent of $\bar X$. A $\chi^2_k$ has variance $2k$, so

$$V(S^2) = \frac{2\sigma^4}{n-1}, \qquad \frac{\text{sd}(\widehat{\text{se}})}{\text{se}} \approx \frac{1}{\sqrt{2(n-1)}}.$$

(The last step uses the delta method on $\sqrt{\cdot}$: $\text{sd}(S) \approx \text{sd}(S^2)/(2\sigma)$.) AoS defines the $\chi^2$ distribution in §10.3; the variance fact is standard but not stated in either book.

Work it by hand · how wrong could our SE of 1 be?

Toy data: $n = 5$, $\widehat{\text{se}} = 1$. Relative noise $\approx 1/\sqrt{2 \times 4} = 0.35$. So a true SE anywhere from about 0.65 to 1.35 would be unsurprising. With five points you don't know your SE to better than about a third, which is exactly why $t_4$'s critical value (2.776) is so much bigger than 1.96.

$n$	$1/\sqrt{2(n-1)}$	Simulated, normal data	Simulated, car prices
5	35%	36%	58%
10	24%	24%	39%
30	13%	13%	21%
92	7%	7%	12%
2930	1.3%	1.3%	2%

Typical relative error in $\widehat{\text{se}}$ course. Skewed, heavy-tailed data make the SE noisier: whether a small sample happens to include the one $38k truck swings $s$ a lot. On the full 92 cars, bootstrapping the SE itself gives 90% of resampled SEs between 0.68 and 1.00 around our 0.852, about ±11%.

Story proof · why a noisy denominator fattens the tails

Write $Z = (\bar X - \mu)/\widehat{\text{se}}$. If $\widehat{\text{se}}$ were exact, $Z$ would be standard normal. But $\widehat{\text{se}}$ sometimes comes out too small, and dividing an ordinary miss by a too-small number produces an extreme $Z$. Too-large SEs pull $Z$ toward zero, but they can only shrink it modestly, while too-small SEs can blow it up without limit. The net effect is more mass in the tails than $N(0,1)$ has.

For skewed data there is a second effect. A sample that misses the expensive cars has a low mean and a small spread, so its large negative miss is divided by a small SE. In the course simulation with 10 cars, $\text{corr}(\bar x, \widehat{\text{se}}) = 0.80$, and with 5 cars, 21% of "95%" intervals miss low while only 1% miss high.

$n$	Normal data, $\bar x \pm 1.96\,\widehat{\text{se}}$	Normal data, t interval	Car prices, $\pm 1.96$	Car prices, t interval
5	88%	95%	78%	86%
10	92%	95%	85%	88%
30	94%	95%	92%	93%
92	95%	95%	94%	94%

Actual coverage of a nominal 95% interval course. The t interval exactly repairs the noisy-SE problem for normal data. It cannot repair skew.

Think about itAt $n = 10$, which hurts a 95% interval more: an SE that came out 30% too small, or one that came out 30% too big?

Common mistake: Equally bad, they're symmetric.

The real answer: Too small, by far. Too small: the interval is really $\pm 1.37$ true SEs and covers 83%. Too big: $\pm 2.55$ true SEs, covering 99%. You lose 12 points one way and gain 4 the other. That asymmetry is why a noisy SE produces too many misses on average, and why the t critical value is larger than 1.96.

The t-distribution: the exact fix for normal data

Formal · AoS §2.4 and §10.10.2; CASI (2.17), (11.47)

$T$ has a t-distribution with $k$ degrees of freedom if its density is

$$f(t) = \frac{\Gamma\big(\frac{k+1}{2}\big)}{\sqrt{k\pi}\,\Gamma\big(\frac k2\big)}\Big(1 + \frac{t^2}{k}\Big)^{-(k+1)/2}.$$

It has thicker tails than the normal; as $k \to \infty$ it becomes $N(0,1)$; with $k = 1$ it is the Cauchy. If $X_1,\dots,X_n \sim N(\mu,\sigma^2)$ with both unknown,

$$T = \frac{\sqrt n\,(\bar X_n - \mu_0)}{S_n} \sim t_{n-1} \text{ under } H_0: \mu = \mu_0.$$

Rejecting when $|T| > t_{n-1,\alpha/2}$ gives an exact size-$\alpha$ test. AoS: "when $n$ is moderately large, the t-test is essentially identical to the Wald test."

Work it by hand · z vs. t on the toy data

$\{2, 4, 5, 6, 8\}$, $H_0: \mu = 3$. $t = (5 - 3)/1 = 2$, df = 4.

	critical value	decision at 0.05	p-value	95% interval
normal (z)	1.96	reject	0.046	$5 \pm 1.96 = (3.04, 6.96)$
$t_4$	2.776	retain	0.116	$5 \pm 2.776 = (2.22, 7.78)$

Same data, opposite conclusions. The t answer is the honest one at $n = 5$.

Story proof · a t is a normal with a random scale

Standard construction: $T = Z/\sqrt{V/k}$, with $Z \sim N(0,1)$ and $V \sim \chi^2_k$ independent. In the one-sample case, $Z = \sqrt n(\bar X - \mu)/\sigma$ and $V/k = S^2/\sigma^2$, and the unknown $\sigma$ cancels, which is why $T$ is pivotal. So a t random variable is a normal draw divided by an independently estimated scale. That is a mixture of normals with different widths, and mixing narrow and wide bells gives a peak plus heavy shoulders. As $k$ grows, $V/k$ concentrates at 1 (LLN) and the mixture collapses to a single bell.

Confidence	df 3	df 5	df 8	df 10	df 30	df 70	df 91	normal
90%	2.353	2.015	1.860	1.812	1.697	1.667	1.662	1.645
95%	3.182	2.571	2.306	2.228	2.042	1.994	1.986	1.960
99%	5.841	4.032	3.355	3.169	2.750	2.648	2.631	2.576

CASI's leukemia example uses df = 70, where the cutoff is 1.99 instead of 1.96: barely different. Degrees of freedom = sample size minus the number of parameters estimated before the residual spread could be computed: $n-1$ for one mean, $n_1 + n_2 - 2$ for two, $n - k$ for a regression with $k$ coefficients.

Worked example · when t changes the decision

$n = 9$, $\bar x = 54$, $s = 6$, $H_0: \mu = 50$. $\widehat{\text{se}} = 6/3 = 2$, $t = 2.0$. Normal: $2.0 > 1.96$, reject (p = 0.046). $t_8$: $2.0 < 2.306$, fail to reject (p = 0.081). With 9 points the SE is itself uncertain by about ±25%, and the t answer accounts for it.

Why it matters · real case · why the t-distribution exists

William Sealy Gosset worked at the Guinness brewery in Dublin, where experiments on barley and brewing came in tiny batches: a handful of measurements, not hundreds. The large-sample normal tables gave misleading answers at those sizes. He worked out the distribution of $(\bar x - \mu)/(s/\sqrt n)$ for normal data and published it in 1908 under the name "Student," because Guinness restricted what its employees could publish.

By hand. With $n = 4$, the 95% critical value is $t_{3,.025} = 3.182$ instead of 1.96, so the honest interval is $3.182/1.96 = 1.62$ times as wide. A rule built on 1.96 would have been overconfident on every small batch.

In data work: an experiment across 5 regions, 8 stores or 6 hospitals is a Guinness problem even when each unit contributes millions of rows. The number of independent units sets the degrees of freedom.

Think about itIf the SE is estimated, why doesn't the t-test have the same problem as the z-test?

Common mistake: Because t is more conservative.

The real answer: Because its critical value is computed knowing the SE is estimated. For normal data, $T = (\bar X - \mu)/(S/\sqrt n)$ has exactly the $t_{n-1}$ distribution, with the randomness of $S$ built in. The z-test pretends $S = \sigma$. Being wider isn't the point; being correctly calibrated is.

Check your understandingWhy is it $n-1$ degrees of freedom and not $n$?

The residuals $x_i - \bar x$ must sum to zero, so once you know $n-1$ of them the last is determined: only $n-1$ carry independent information about the spread. In general, each parameter estimated before computing the spread costs one degree of freedom, giving $n - k$ in a regression with $k$ coefficients.

Bootstrap-t: studentize, then resample

The t-distribution is exact only for normal data. The bootstrap-t keeps Student's idea of dividing by an estimated SE, but learns the distribution of the ratio from the data instead of from a table.

Formal · CASI (11.52)–(11.53)

For each bootstrap sample compute $\hat\theta^*$ and its own $\widehat{\text{se}}^*$, and form $t^* = (\hat\theta^* - \hat\theta)/\widehat{\text{se}}^*$. With $t^{*(\alpha)}$ the $\alpha$ quantile of the $B$ replicates, the level-$\alpha$ limit is

$$\hat\theta_t[\alpha] = \hat\theta - \widehat{\text{se}}\cdot t^{*(1-\alpha)}.$$

Work it by hand · three bootstrap-t replicates

From $\{2, 4, 5, 6, 8\}$ ($\hat\theta = 5$, $\widehat{\text{se}} = 1$), three resamples:

resample	$\bar x^*$	$\widehat{\text{se}}^* = s^*/\sqrt5$	$t^* = (\bar x^* - 5)/\widehat{\text{se}}^*$
2, 2, 5, 8, 8	5.0	1.34	0.00
4, 5, 5, 6, 8	5.6	0.68	0.89
2, 4, 4, 5, 6	4.2	0.66	−1.21

Each resample brings its own SE, and they range from 0.66 to 1.34: "is the SE uncertain?" happening in front of you. The $t^*$ values carry that extra noise into the interval: with many replicates, the limits are $5 - 1 \times t^{*(0.975)}$ and $5 - 1 \times t^{*(0.025)}$.

From the books · CASI §11.5

AML minus ALL means, $n_x = 47$, $n_y = 25$: Student's 95% interval $(0.062, 0.314)$; bootstrap-t $(0.072, 0.323)$. The $t^*$ distribution was close to $t_{70}$, which supports the original normality assumption. Course version: the "is the SE uncertain?" effect is exactly what $t^*$ captures, because each resample brings its own SE.

5 · Confidence intervals

Go deeper: expanded page with more worked examples and interactive widgets →

A confidence interval turns an estimate and its SE into a range. The hard part is not the arithmetic but the reading: the 95% describes the procedure, not this particular interval.

Definition and how to read it

Formal · AoS (6.9)

A $1-\alpha$ confidence interval for $\theta$ is an interval $C_n = (a, b)$, where $a = a(X_1,\dots,X_n)$ and $b = b(X_1,\dots,X_n)$ are functions of the data, such that

$$P_\theta(\theta \in C_n) \ge 1 - \alpha \quad \text{for all } \theta.$$

$1-\alpha$ is the coverage. AoS's warning, in bold in the book: $C_n$ is random and $\theta$ is fixed.

Work it by hand · computing coverage exactly

Back to the nine samples of size 2 from $\{1, 3, 5\}$ ($\mu = 3$). Try the interval $\bar x \pm 1.5$. It contains 3 exactly when $\bar x \in \{2, 3, 4\}$, which has probability $\frac{2 + 3 + 2}{9} = \frac79$. So $\bar x \pm 1.5$ is a 78% confidence interval, by counting.

interval	covers $\mu = 3$ when	coverage
$\bar x \pm 0.5$	$\bar x = 3$	3/9 = 33%
$\bar x \pm 1.5$	$\bar x \in \{2, 3, 4\}$	7/9 = 78%
$\bar x \pm 2.5$	always	9/9 = 100%

Coverage is a property of the rule, computed over all samples before you see yours.

Wasserman calls the usual repeated-experiment reading "correct but useless," since nobody repeats the same experiment. His preferred reading: on day 1 you build a 95% interval for one parameter, on day 2 for an unrelated parameter, and so on for your whole career; 95% of all those intervals trap their true values. AoS Ex 6.13 applies it to polls: "83% ± 4, accurate 95% of the time" means that if you form such an interval every day for the rest of your life, 95% contain the truth, even though each day's question differs. CASI (footnote to §2.1) pushes back gently: clients care about their own problem, not a career average, so CASI frames coverage as hypothetical repetitions of the specific problem at hand.

Analogy · ring toss

The peg ($\theta$) is bolted to the floor. You throw rings (intervals). A "95% ring" is a ring size and throwing method that lands around the peg 95% of the time. After a throw, the ring either circles the peg or it doesn't; there's no 95% about that one ring. The 95% belongs to your arm.

Why it matters · real case · experts misread intervals too

Hoekstra, Morey, Rouder and Wagenmakers gave researchers and students a reported 95% interval, (0.1, 0.4), and six statements to mark true or false, including "there is a 95% probability that the true mean lies between 0.1 and 0.4." By the authors' standard all six were false, and most respondents, researchers included, endorsed at least one. Experience did not reliably protect against it. Psychonomic Bulletin & Review 2014

In data work: when a report or dashboard shows an interval, put the sentence next to it: "intervals built this way contain the true value in 95% of samples." Readers will otherwise supply the probability-about-$\theta$ reading, and plan budgets as if it were true.

Worked example · AoS Ex 6.14 (Berger & Wolpert)

$\theta$ is fixed; $X_1, X_2$ are independent ±1 with probability 1/2 each; you observe $Y_i = \theta + X_i$. Let $C = \{Y_1 - 1\}$ if $Y_1 = Y_2$, and $C = \{(Y_1+Y_2)/2\}$ if they differ. Whatever $\theta$ is, $P(\theta \in C) = 3/4$, so $C$ is a 75% confidence set. Now observe $Y_1 = 15$, $Y_2 = 17$: $C = \{16\}$, and you are certain $\theta = 16$. Nothing is wrong with calling $\{16\}$ a 75% interval; it just isn't a probability statement about $\theta$. This is the clearest demonstration that coverage and belief are different things.

AoS adds that Bayesian intervals (Ch 11) do make statements like "the probability $\theta$ is in $C_n$, given the data, is 95%," but those are degree-of-belief probabilities and will not, in general, trap the parameter 95% of the time.

Think about itThe 95% interval for the average car price is (6.92, 10.26). Is there a 95% probability that $\mu$ is in it?

Common mistake: Yes, that's what 95% confidence means.

The real answer: No. $\mu$ is a fixed number, and this interval either contains it or doesn't. The 95% describes the procedure: 95% of intervals built this way, from fresh samples, contain $\mu$. AoS Ex 6.14 is the knockout example: a 75% interval that you know with certainty contains $\theta$.

Think about itSame interval, (6.92, 10.26). Do about 95% of Charlottesville used cars cost between $6.9k and $10.3k?

Common mistake: Yes.

The real answer: No: only 14 of the 92 cars (15%) are priced in that range. The interval is about the average price; individual cars spread with SD $8.17k. A range meant to contain 95% of individual cars is a prediction interval and is far wider (Section 8).

Think about itI build 20 independent 95% confidence intervals. How many contain the truth?

Common mistake: 19.

The real answer: On average 19, but the count is Binomial(20, 0.95). The chance all 20 cover is $0.95^{20} = 0.36$; the chance exactly 19 do is 0.38; so about a quarter of the time two or more miss. And you never learn which ones.

Normal-based intervals

Formal · AoS Thm 6.16

Suppose $\hat\theta_n \approx N(\theta, \widehat{\text{se}}^2)$. With $z_{\alpha/2} = \Phi^{-1}(1 - \alpha/2)$, let $C_n = (\hat\theta_n - z_{\alpha/2}\widehat{\text{se}},\ \hat\theta_n + z_{\alpha/2}\widehat{\text{se}})$. Then $P_\theta(\theta \in C_n) \to 1 - \alpha$.

Proof. Let $Z_n = (\hat\theta_n - \theta)/\widehat{\text{se}} \rightsquigarrow Z$. Then

$$P(\hat\theta_n - z_{\alpha/2}\widehat{\text{se}} < \theta < \hat\theta_n + z_{\alpha/2}\widehat{\text{se}}) = P\Big(-z_{\alpha/2} < \frac{\hat\theta_n - \theta}{\widehat{\text{se}}} < z_{\alpha/2}\Big) \to P(-z_{\alpha/2} < Z < z_{\alpha/2}) = 1 - \alpha.$$

For 95%, $z_{.025} = 1.96 \approx 2$, giving $\hat\theta_n \pm 2\,\widehat{\text{se}}$.

Work it by hand · normal intervals on the toy data

$\bar x = 5$, $\widehat{\text{se}} = 1$. 90%: $5 \pm 1.645 = (3.36, 6.65)$. 95%: $5 \pm 1.96 = (3.04, 6.96)$. 99%: $5 \pm 2.576 = (2.42, 7.58)$. Higher confidence, wider interval. (At $n = 5$ the theorem's "$\approx$" is doing a lot of work; the honest 95% interval uses $t_4$: $(2.22, 7.78)$.)

Confidence	$\alpha$	Each tail	$z_{\alpha/2}$	Check
90%	0.10	0.05	1.645	$\Phi(1.645) - \Phi(-1.645) = 0.900$
95%	0.05	0.025	1.960	$\Phi(1.96) - \Phi(-1.96) = 0.950$
99%	0.01	0.005	2.576	$\Phi(2.576) - \Phi(-2.576) = 0.990$

Worked examples

AoS Ex 6.17 (Bernoulli): $V(\hat p_n) = p(1-p)/n$, so the interval is $\hat p_n \pm z_{\alpha/2}\sqrt{\hat p_n(1-\hat p_n)/n}$.

AoS Ex 6.15 (Hoeffding, exact but wider): $\hat p_n \pm \epsilon_n$ with $\epsilon_n^2 = \log(2/\alpha)/(2n)$ has coverage at least $1-\alpha$ for every $n$. At $n = 100$, $\alpha = 0.05$: $\epsilon_n = \sqrt{3.7/200} \approx \sqrt{0.0185} \approx 0.136$, versus the normal half-width $1.96(0.05) = 0.098$ at $\hat p = 0.5$. Exactness costs width. AoS: the normal interval "is shorter but it only has approximately (large sample) correct coverage."

Course, car prices: $8.59 \pm 1.645(0.852) = (7.19, 9.99)$ and $8.59 \pm 1.96(0.852) = (6.92, 10.26)$.

Pitfall · symmetric intervals for asymmetric problems

CASI Ch 11 opens with a Poisson observation $\hat\theta = 10$. The standard interval $10 \pm 1.96\sqrt{10} = (3.8, 16.2)$ is a "mediocre approximation" to the exact interval $(5.1, 17.8)$. The Poisson spreads more as $\theta$ grows, so the right answer reaches farther right; a symmetric $\pm$ can't.

Think about itGroup A's 95% interval and group B's 95% interval overlap. So the difference isn't significant, right?

Common mistake: Right, overlapping intervals mean no significant difference.

The real answer: Not necessarily. Two means 0 and 3, each with SE 1: intervals $(-1.96, 1.96)$ and $(1.04, 4.96)$ overlap. But the difference is 3 with SE $\sqrt{1^2 + 1^2} = 1.41$, so $z = 2.12$, $p = 0.034$: significant. The SE of a difference is smaller than the sum of the two SEs (it adds in squares), so "no overlap" is a much stricter test than $\alpha = 0.05$.

Check your understandingWhy is a 99% interval wider than a 95% one? Shouldn't more confidence come with more precision?

Confidence and precision trade off for a fixed amount of data. To be right more often, the interval has to cast a wider net: $2.576$ SEs instead of $1.96$. The only way to get both is more data, which shrinks the SE.

Bootstrap intervals: normal, pivotal, percentile

AoS §8.3 gives three intervals from the same $B$ replicates. They differ in which feature of the bootstrap distribution they use.

Formal · AoS (8.2)–(8.6), Thm 8.3

Normal: $\hat\theta_n \pm z_{\alpha/2}\,\widehat{\text{se}}_{\text{boot}}$. Uses only the spread; inaccurate unless the sampling distribution is close to normal.

Pivotal (basic): let the pivot be $R_n = \hat\theta_n - \theta$ with CDF $H$. If $H$ were known, $a = \hat\theta_n - H^{-1}(1-\alpha/2)$, $b = \hat\theta_n - H^{-1}(\alpha/2)$ would give exact coverage. Estimate $H$ from $R^*_b = \hat\theta^*_b - \hat\theta_n$; since its quantiles are $r^*_\beta = \theta^*_\beta - \hat\theta_n$,

$$C_n = \big(2\hat\theta_n - \theta^*_{1-\alpha/2},\ 2\hat\theta_n - \theta^*_{\alpha/2}\big).$$

Thm 8.3: under weak conditions on $T$, $P(T(F) \in C_n) \to 1 - \alpha$.

Percentile: $C_n = (\theta^*_{\alpha/2},\ \theta^*_{1-\alpha/2})$, the middle $1-\alpha$ of the replicates (justified in AoS §8.5.2).

Work it by hand · three intervals from ten replicates

Suppose $\hat\theta = 5$ and ten bootstrap replicates came out: 3.6, 4.0, 4.4, 4.6, 4.8, 5.0, 5.2, 5.6, 6.2, 7.0 (a long right tail). Take the 0.10 and 0.90 quantiles as 4.0 and 6.2, and the replicates' SD is 1.02. Build 80% intervals ($z_{.10} = 1.2816$).

method	calculation	80% interval
normal	$5 \pm 1.2816 \times 1.02$	(3.69, 6.31)
percentile	$(\theta^_{.10}, \theta^_{.90})$	(4.0, 6.2)
pivotal	$(2 \cdot 5 - 6.2,\ 2 \cdot 5 - 4.0)$	(3.8, 6.0)

The percentile interval leans right, with the tail. The pivotal interval leans left, flipping it. The normal interval ignores the skew.

Story proof · why the pivotal interval is "flipped"

The bootstrap world mimics the real one: in the real world $\hat\theta$ misses $\theta$ by $R = \hat\theta - \theta$; in the bootstrap world $\hat\theta^*$ misses $\hat\theta$ by $R^* = \hat\theta^* - \hat\theta$. If the replicates tend to overshoot $\hat\theta$ by a lot on the right, then $\hat\theta$ probably overshoots $\theta$ by a lot on the right too, so $\theta$ is likely to sit left of $\hat\theta$. The interval's long side therefore goes on the opposite side from the bootstrap distribution's long tail: $\hat\theta$ minus the upper miss, $\hat\theta$ minus the lower miss. That is where the "2θ̂ minus a quantile" comes from.

Story proof · why the percentile interval can work without knowing a transformation

Suppose some unknown monotone transformation $\phi = m(\theta)$ makes the estimator exactly normal with constant spread. On that scale the right interval is the obvious symmetric one, and it coincides with the percentiles of the transformed replicates. Monotone transformations carry percentiles along with them: the 2.5th percentile of $m(\hat\theta^*)$ is $m$ of the 2.5th percentile of $\hat\theta^*$. So transforming back, the percentile interval on the original scale is the right interval, even though you never learned $m$. The bootstrap histogram does the transformation for you CASI §11.2.

From the books · AoS Ex 8.4, 8.5

Method	Nerve-data skewness	Cholesterol median difference (est. 18.5, se 7.42)
Normal	(1.44, 2.09)	(3.7, 33.3)
Pivotal	(1.48, 2.11)	(5.0, 34.0)
Percentile	(1.42, 2.03)	(5.0, 33.3)

AoS: all three are approximate and have the same order of accuracy. Course car prices: normal $(6.92, 10.26)$ vs percentile $(7.02, 10.32)$; the percentile interval shifts right because the bootstrap distribution of the mean is still slightly right-skewed at $n = 92$.

Better bootstrap intervals: BC, BCa, and second-order accuracy

CASI Ch 11 builds a ladder of four bootstrap intervals, each fixing one more thing: standard (normal) → percentile → BC → BCa. AoS mentions that more accurate intervals exist but doesn't cover them.

Formal · CASI (11.31)–(11.39)

BC (bias-corrected). Let $p_0$ = proportion of replicates below $\hat\theta$ and $z_0 = \Phi^{-1}(p_0)$. With $\hat G$ the bootstrap CDF and $z^{(\alpha)} = \Phi^{-1}(\alpha)$,

$$\hat\theta_{BC}[\alpha] = \hat G^{-1}\big(\Phi(2z_0 + z^{(\alpha)})\big).$$

If $p_0 = 0.5$, then $z_0 = 0$ and BC reduces to the percentile interval.

BCa (bias-corrected and accelerated) adds an "acceleration" $a$ describing how the SE changes with $\theta$:

$$\hat\theta_{BCa}[\alpha] = \hat G^{-1}\Big(\Phi\Big(z_0 + \frac{z_0 + z^{(\alpha)}}{1 - a(z_0 + z^{(\alpha)})}\Big)\Big).$$

BCa corrects for non-normality (via percentiles), bias ($z_0$), and non-constant SE ($a$). With $a = 0$ it is BC; with $z_0 = 0$ as well, percentile; with a normal bootstrap histogram, standard.

Accuracy (CASI (11.36)–(11.37)). Standard intervals have coverage errors of order $1/\sqrt n$ ("first-order"): to halve the error you need four times the data. BCa is "second-order accurate": errors of order $1/n$. Percentile and BC are first-order.

Work it by hand · a bias correction with a normal table

Suppose 4 of the 10 replicates fall below $\hat\theta$: $p_0 = 0.4$, so $z_0 = \Phi^{-1}(0.4) = -0.253$. For an 80% BC interval, replace the 10th and 90th percentiles by:

lower: $\Phi(2z_0 + z^{(.10)}) = \Phi(-0.506 - 1.282) = \Phi(-1.788) = 0.037$, the 3.7th percentile;
upper: $\Phi(2z_0 + z^{(.90)}) = \Phi(-0.506 + 1.282) = \Phi(0.775) = 0.781$, the 78.1th percentile.

Both ends slide down, because the replicates sit high relative to $\hat\theta$ ($p_0 < 0.5$). With $p_0 = 0.5$, $z_0 = 0$ and you're back to (10th, 90th).

From the books · CASI §11.2–11.3 (student score correlation)

$n = 22$, $\hat\theta = 0.498$. The histogram of $B = 2000$ bootstrap correlations is long-tailed to the left, so asymptotic normality hasn't set in. Percentile 95% interval: $(0.118, 0.758)$. For the parametric version, $p_0 = 0.478$ gives $z_0 = -0.055$, shifting the BC limits slightly downward.

Analogy · the ladder of corrections

Think of fitting a picture frame. The standard interval assumes the wall is flat and level. The percentile method lets the wall be curved (skewed). BC also allows the nail to be off-center (bias). BCa also lets the wall's curvature change as you move along it (non-constant SE). Each step fits a less ideal wall.

Neyman's construction CASI §11.1 is the general recipe all of these approximate: for each candidate $\theta$, ask whether the observed $\hat\theta$ would be unusual if $\theta$ were true; the interval is the set of $\theta$ values for which it wouldn't be. That is the same as inverting a family of tests, which Section 6 makes precise.

Check your understandingWhen would the percentile and pivotal bootstrap intervals disagree, and which should you trust?

They disagree when the bootstrap distribution is skewed: the pivotal interval puts its long side opposite the long tail, the percentile interval puts it on the same side. They agree when the bootstrap histogram is symmetric. AoS treats them as equally accurate (first-order); CASI recommends BCa when it matters, which corrects for both bias and skew. If the two disagree badly, that's a signal $n$ is too small for any simple interval.

6 · Hypothesis testing

Go deeper: expanded page with more worked examples and interactive widgets →

A test asks a yes-or-no question about a parameter: is the data's evidence strong enough to abandon a default claim? The machinery is the same estimate-over-SE ratio as an interval, read in the other direction.

The framework

Formal · AoS (10.1)–(10.4), Def 10.1, Table 10.1

Partition the parameter space into $\Theta_0$ and $\Theta_1$ and test $H_0: \theta \in \Theta_0$ versus $H_1: \theta \in \Theta_1$. Choose a rejection region $R$, usually $R = \{x : T(x) > c\}$ for a test statistic $T$ and critical value $c$. Reject $H_0$ if $X \in R$; otherwise retain it.

	Retain $H_0$	Reject $H_0$
$H_0$ true	correct	type I error
$H_1$ true	type II error	correct

Power function: $\beta(\theta) = P_\theta(X \in R)$. Size: $\alpha = \sup_{\theta\in\Theta_0}\beta(\theta)$. A test has level $\alpha$ if its size is at most $\alpha$. $\theta = \theta_0$ is a simple hypothesis; $\theta > \theta_0$ is composite. $H_1: \theta \ne \theta_0$ is two-sided; $H_1: \theta > \theta_0$ is one-sided.

Work it by hand · a power function for five coin flips

Test $H_0: p = 1/2$ with 5 flips; reject if all 5 are heads or all 5 are tails. Power function $\beta(p) = p^5 + (1-p)^5$.

true $p$	0.5	0.6	0.8	0.95
$\beta(p)$	0.0625 (the size: 2/32)	0.088	0.328	0.774

Size 0.0625 means this "5-flip trial" convicts a fair coin 6% of the time. Even a coin that lands heads 80% of the time is caught only a third of the time: small samples have little power.

Analogy · a criminal trial (AoS's own)

The defendant is presumed innocent ($H_0$) unless the evidence strongly suggests guilt. Convicting an innocent person is a type I error; acquitting a guilty one is type II. The size $\alpha$ is how often the court convicts the innocent. Power is how often it convicts the guilty. "Not guilty" is not "innocent": retaining $H_0$ is not evidence for it.

AoS gives a warning: there is a tendency to use hypothesis testing even when it isn't appropriate. Often estimation and confidence intervals are better tools; test only when you have a well-defined hypothesis. AoS also adopts "retain the null" (credited to Chris Genovese) instead of "accept the null."

Think about itWe failed to reject $H_0$. So we've shown $H_0$ is true?

Common mistake: Yes, the data support the null.

The real answer: No. AoS deliberately says "retain," not "accept." A non-rejection can mean $H_0$ is true or the test lacked power. "Not guilty" isn't "innocent." Report the confidence interval: if it's narrow and hugs $\theta_0$, that's evidence of no large effect; if it's wide, you've learned little.

The Wald test

The Wald test is the estimate-over-SE test, and it covers almost every test in this course, including every row of a regression table.

Formal · AoS Def 10.3, Thms 10.4, 10.6

Test $H_0: \theta = \theta_0$ vs $H_1: \theta \ne \theta_0$, with $\hat\theta$ asymptotically normal. The size-$\alpha$ Wald test rejects when $|W| > z_{\alpha/2}$, where

$$W = \frac{\hat\theta - \theta_0}{\widehat{\text{se}}}.$$

Thm 10.4: asymptotically it has size $\alpha$. (Under $\theta_0$, $W \rightsquigarrow N(0,1)$, so $P(|W| > z_{\alpha/2}) \to P(|Z| > z_{\alpha/2}) = \alpha$.) Remark 10.5: using the SE computed at $\theta_0$ instead of at $\hat\theta$ is also valid.

Thm 10.6 (power): if the truth is $\theta_* \ne \theta_0$, the power is approximately

$$1 - \Phi\Big(\frac{\theta_0 - \theta_*}{\widehat{\text{se}}} + z_{\alpha/2}\Big) + \Phi\Big(\frac{\theta_0 - \theta_*}{\widehat{\text{se}}} - z_{\alpha/2}\Big).$$

Power is large when $\theta_*$ is far from $\theta_0$ and when the sample is large (since $\widehat{\text{se}} \to 0$).

Work it by hand · Wald test on the toy data

$H_0: \mu = 3$. $W = (5 - 3)/1 = 2 > 1.96$: reject at 0.05. If the truth really is $\mu = 5$ (2 SEs from the null), Thm 10.6 gives power $1 - \Phi(-2 + 1.96) + \Phi(-2 - 1.96) = 1 - \Phi(-0.04) + \Phi(-3.96) = 0.52$. A true 2-SE effect is caught about half the time.

Worked example · power is only 52% at "2 standard errors"

If the true effect is exactly 2 SEs from the null, the Wald test at $\alpha = 0.05$ has power $1 - \Phi(1.96 - 2) + \Phi(-1.96 - 2) = 0.516 + 0.000 = 0.52$ course. A real effect sitting right at the "significance line" is detected only about half the time. This is the one number that explains most failed replications.

Worked examples · AoS Ex 10.7–10.9, 10.15

Two prediction algorithms, separate test sets: $\hat\delta = \hat p_1 - \hat p_2$, $\widehat{\text{se}} = \sqrt{\hat p_1(1-\hat p_1)/m + \hat p_2(1-\hat p_2)/n}$.
Same test set (paired): the samples are no longer independent. Let $D_i = X_i - Y_i$ (1 if only algorithm 1 is right on case $i$, −1 if only algorithm 2, 0 otherwise); then $\hat\delta = \bar D$, $\widehat{\text{se}} = s_D/\sqrt n$. Using the unpaired formula here would be the dependence failure.
Two means: $W = (\bar X - \bar Y)/\sqrt{s_1^2/m + s_2^2/n}$.
Two medians: $\hat\delta = \hat\nu_1 - \hat\nu_2$, SE from the bootstrap, $W = \hat\delta/\widehat{\text{se}}$. A Wald test for a statistic with no formula SE.
Cholesterol (Ex 10.15): $W = (216.2 - 195.3)/\sqrt{5^2 + 2.4^2}$. The numerator is 20.9 and the denominator is $\sqrt{25 + 5.76} = 5.55$, so $W \approx 3.8$; p-value $2\Phi(-3.8) \approx 0.0002$ (AoS reports 3.78).

Why it matters · scenario · comparing two models on one test set

Illustrative numbers. Model A is right on 85 of 100 test cases, model B on 80. The unpaired formula gives $\widehat{\text{se}} = \sqrt{0.85(0.15)/100 + 0.80(0.20)/100} = \sqrt{0.002875} \approx 0.054$, so $z = 0.05/0.054 \approx 0.93$.

But both models saw the same 100 cases. Suppose A alone is right on 8 cases, B alone on 3, and they agree on the other 89. The per-case differences $D_i$ are eight 1s, three −1s and 89 zeros:

$\bar D = (8 - 3)/100 = 0.05$, the same 5-point gap.
$\sum(D_i - \bar D)^2 = \sum D_i^2 - n\bar D^2 = 11 - 0.25 = 10.75$, so $s_D^2 = 10.75/99 \approx 0.109$ and $s_D \approx 0.33$.
$\widehat{\text{se}} = 0.33/10 = 0.033$, so $z = 0.05/0.033 \approx 1.5$.

The paired SE is 40% smaller because the 89 cases both models get right or both get wrong cancel out. Neither $z$ clears 1.96 at $n = 100$. With 300 test cases at the same rates, both $z$'s grow by $\sqrt 3 = 1.73$: paired $z \approx 2.6$ (significant), unpaired $z \approx 1.6$ (not). The unpaired analysis would tell a team their better model isn't better. McNemar's test is the exact version of this paired comparison.

Think about itTwo research groups compare two algorithms on the same test set and use the two-sample SE formula. What's wrong?

Common mistake: Nothing, it's two samples.

The real answer: The two accuracy estimates are correlated: both algorithms see the same easy and hard cases. The two-sample formula assumes independence and usually overstates the SE of the difference. Analyze the per-case differences $D_i$ instead (AoS Ex 10.7, paired comparison).

Think about itA real effect sits exactly 1.96 SEs from the null, so the original study got $p = 0.05$. What's the chance an identical replication gets $p < 0.05$?

Common mistake: 95%, because the effect is real.

The real answer: About 50%. The replication's estimate is centered on the true effect (1.96 SEs out) and wobbles ±1 SE, so it lands past 1.96 about half the time (AoS Thm 10.6 gives 0.50). Barely significant results fail to replicate half the time even when they're true.

Why it matters · real case · drug programs built on findings that didn't replicate

Drug companies choose which biological targets to pursue partly from published results. When Bayer scientists reviewed 67 in-house projects that tried to confirm published drug targets, only about a fifth to a quarter matched the literature Prinz et al. 2011. Amgen scientists tried to reproduce 53 landmark cancer-biology papers and confirmed 6 Begley & Ellis 2012. Lab methods and selective reporting contribute, but the arithmetic above is part of it: a real effect that barely cleared $p = 0.05$ replicates only about half the time, and some published effects had no real effect behind them at all. Each failed confirmation costs months of work before a program is dropped.

In data work: the same arithmetic produces the "winner's curse" in A/B testing. The experiments that cleared the significance bar are, on average, the ones whose estimates came out high by luck, so the lift measured in the test usually overstates the lift you see after launch.

Tests and intervals are the same thing

Formal · AoS Thm 10.10

The size-$\alpha$ Wald test rejects $H_0: \theta = \theta_0$ if and only if $\theta_0 \notin C = (\hat\theta - z_{\alpha/2}\widehat{\text{se}},\ \hat\theta + z_{\alpha/2}\widehat{\text{se}})$.

Work it by hand · test three nulls with one interval

Toy-data 95% interval: $(3.04, 6.96)$.

$\mu_0$	inside?	$W = (5 - \mu_0)/1$	Wald decision
3	no	2.0	reject
4	yes	1.0	retain
7	no	−2.0	reject

The interval is exactly the set of $\mu_0$ values the test retains.

Story proof

Both statements say the same thing: "$\hat\theta$ and $\theta_0$ are more than $z_{\alpha/2}$ standard errors apart." The test measures the gap from $\theta_0$'s point of view; the interval measures it from $\hat\theta$'s. Distance is symmetric. So a confidence interval is the set of null values you would not reject, which is Neyman's construction in one sentence.

Pitfall · statistical vs. scientific significance (AoS Fig 10.2)

A result can be statistically significant and scientifically trivial. Two intervals can both exclude $\theta_0$, one hugging it and one far away; both "reject," but only the second matters. AoS's conclusion: significance doesn't imply importance, and confidence intervals are often more informative than tests.

Why it matters · real case · small changes, big money (and the reverse)

Kohavi and Thomke describe a Bing change to how ad headlines were displayed. It was shelved for months as low priority, and when finally tested it raised revenue about 12%, worth more than $100 million a year in the US. The effect was so large that it first triggered an alert for a likely bug. Harvard Business Review 2017

At that scale the reverse also happens: with millions of users per arm, an effect of a hundredth of a percent can be "highly significant" and not worth the engineering time to maintain.

By hand, illustrative numbers. With an SE of 0.02% on a lift, an observed lift of 0.1% has $z = 0.1/0.02 = 5$. Whether 0.1% matters depends on what it multiplies: on a $1 billion revenue line it is $1 million a year; on a $1 million product it is $1,000. The $z$ is the same in both cases; the decision isn't. Report the interval in dollars.

Check your understandingIf a test rejects at 0.05, does a 95% confidence interval necessarily exclude $\theta_0$?

Yes, when the test and interval use the same estimate, SE, and critical value (AoS Thm 10.10). They can disagree if they're built differently, for example a percentile bootstrap interval paired with a Wald test, or a test using the SE computed under $H_0$ (AoS Remark 10.5) paired with an interval using $\widehat{\text{se}}$ at $\hat\theta$. The course's Ames 2Story coefficient is a case where classical and bootstrap methods disagree.

p-values

Reporting only "reject" or "retain" throws information away. The p-value reports the boundary instead: the strictest level at which you would still reject.

Formal · AoS Def 10.11, Thms 10.12–10.14

If for every $\alpha$ there is a size-$\alpha$ test with region $R_\alpha$, then

$$\text{p-value} = \inf\{\alpha : T(X^n) \in R_\alpha\},$$

the smallest level at which we can reject $H_0$. For tests of the form "reject if $T \ge c_\alpha$," Thm 10.12 gives $\text{p-value} = \sup_{\theta\in\Theta_0} P_\theta\big(T(X^n) \ge T(x^n)\big)$: the probability, under $H_0$, of a test statistic at least as extreme as the one observed. For the Wald statistic (Thm 10.13):

$$\text{p-value} = P_{\theta_0}(|W| > |w|) \approx P(|Z| > |w|) = 2\Phi(-|w|).$$

Thm 10.14: if the test statistic is continuous, then under $H_0$ the p-value is Uniform(0,1). So rejecting when $p < \alpha$ has type I error probability exactly $\alpha$.

Work it by hand · p-values you can count

Flip a coin 5 times and get 5 heads. Under $H_0: p = 1/2$, all 32 sequences are equally likely.

One-sided ($H_1: p > 1/2$): outcomes at least as extreme = {HHHHH}: $p = 1/32 = 0.031$.
Two-sided: {HHHHH, TTTTT}: $p = 2/32 = 0.0625$. Retain at 0.05, though one-sided would reject: decide the sidedness first.

Wald version on the toy data: $W = 2$ gives $p = 2\Phi(-2) = 0.046$; $W = 1$ gives $2\Phi(-1) = 0.32$.

p-value	Evidence against $H_0$ (AoS's scale)
< 0.01	very strong
0.01–0.05	strong
0.05–0.10	weak
> 0.10	little or none

Pitfall · AoS's two warnings

A large p-value is not strong evidence for $H_0$. It can happen because $H_0$ is true, or because $H_0$ is false and the test has low power. The p-value is not $P(H_0 \mid \text{data})$. That quantity needs a prior (AoS Ch 11).

Analogy · CASI's "surprise" and the strict judge

CASI §1.2 describes a p-value as a measure of statistical surprise: something very unusual has happened if the null is true. Another way to say AoS's definition: imagine a line of judges, each with a stricter standard of proof (smaller $\alpha$). The p-value is the standard of the strictest judge who would still convict.

Story proof · why p-values are uniform under the null

The p-value is the null's upper-tail probability evaluated at your statistic: $p = 1 - F_0(T)$. Ask: how often is $p \le 0.05$? Exactly when $T$ lands in the top 5% of its null distribution, which by definition happens 5% of the time. Replace 0.05 by any $u$ and the same argument gives $P(p \le u) = u$. That's the Uniform(0,1) CDF. (This is the probability integral transform: feeding a continuous random variable into its own CDF produces a uniform.)

Worked examples · course

Same effect, two sample sizes: $\bar x = 52$, $s = 10$, $H_0: \mu = 50$. With $n = 100$: $\widehat{\text{se}} = 1$, $t = 2$, $p = 2(0.023) = 0.046$. With $n = 25$: $\widehat{\text{se}} = 2$, $t = 1$, $p = 2(0.159) = 0.317$. The p-value measures evidence, which grows with $n$; it does not measure the size of the effect.

Cars under $10k? $t = (8.59 - 10)/0.852 = -1.66$, $p = 2\Phi(-1.66) \approx 0.097$. Reject at 0.10, not at 0.05, matching $10 \notin (7.19, 9.99)$ and $10 \in (6.92, 10.26)$.

Think about it$p = 0.03$. What's the probability the null hypothesis is true?

Common mistake: 3%.

The real answer: The p-value can't tell you. It's $P(\text{data this extreme} \mid H_0)$, not $P(H_0 \mid \text{data})$. The second needs a prior: if almost every hypothesis tested in a field is null, most $p = 0.03$ results are still false positives (CASI Ch 15's large-scale testing makes this precise).

Why it matters · real case · the prosecutor's fallacy

In 1999 Sally Clark was convicted in England of murdering her two infant sons. A pediatrician testified that the chance of two sudden infant deaths in a family like hers was 1 in 73 million, obtained by squaring 1 in 8,543 ($8{,}543^2 \approx 73$ million). The number had two errors. Squaring assumes the two deaths are independent, which ignores shared genetic and environmental risk. And it invited the jury to hear a small $P(\text{evidence} \mid \text{innocent})$ as a small $P(\text{innocent} \mid \text{evidence})$. A double infant murder is also extremely rare, so the real question was which of two rare explanations was more likely. The Royal Statistical Society publicly criticized the statistic, and the conviction was quashed on appeal in 2003 (the successful appeal turned mainly on medical evidence that had been withheld).

A p-value is the same kind of number as that 1 in 73 million: $P(\text{data this extreme} \mid H_0)$, not $P(H_0 \mid \text{data})$.

Think about itStudy A finds $p = 0.04$; study B, on the same question, finds $p = 0.06$. Do they contradict each other?

Common mistake: Yes: one found an effect and one didn't.

The real answer: No. They correspond to $z = 2.05$ and $z = 1.88$, nearly the same evidence. The difference between "significant" and "not significant" is not itself significant. Compare the estimates and intervals, not which side of 0.05 they fell on.

Think about itIf $H_0$ is true, how often is $p < 0.05$? How often is $p < 0.5$?

Common mistake: Almost never, because the null is true.

The real answer: 5% and 50%. Under $H_0$ the p-value is Uniform(0,1) (AoS Thm 10.14), so $P(p < u) = u$ for every $u$. Small p-values are supposed to happen at exactly the rate $\alpha$ when nothing is going on.

Check your understandingShould I use a one-sided or two-sided test?

Decide before seeing the data. A one-sided p-value is half the two-sided one when the estimate falls in the predicted direction, so choosing after seeing which way the data went doubles your real false-positive rate. Use one-sided only when an effect in the other direction would genuinely be treated the same as no effect.

The t-test, and CASI's leukemia gene

The one-sample t-test is in Section 4. CASI §1.2 gives the two-sample version a memorable stage. For 72 leukemia patients (47 ALL, 25 AML), gene 136 has means 0.752 and 0.950, $t = 3.01$ on 70 df, two-sided $p = 0.0036$: classically, very strong evidence. But 7128 genes were measured, and 400 of them (5.6%) have $t > 3.01$. That doesn't mean gene 136 is "significant at the 0.056 level." It means the single-test logic doesn't apply when thousands of tests are run, which is Section 7.

The permutation test

If the group labels don't matter, shuffling them shouldn't change anything. Shuffle many times and see how often a shuffle looks as extreme as the real labels.

Formal · AoS §10.5, Thm 10.29

Test $H_0: F_X = F_Y$. With $N = m + n$ observations, compute the statistic $T$ for each of the $N!$ permutations; under $H_0$ each is equally likely. The permutation distribution puts mass $1/N!$ on each, and $\text{p-value} = P_0(T > t_{\text{obs}}) = \frac{1}{N!}\sum_j I(T_j > t_{\text{obs}})$. In practice, sample random permutations. The test is exact; no normality or large-sample approximation is needed.

Work it by hand · permutation test on four points

Treatment $\{6, 8\}$, control $\{1, 3\}$. $T = |\bar X - \bar Y| = |7 - 2| = 5$. There are $\binom42 = 6$ ways to choose which two are "treatment":

treatment	{6,8}	{1,3}	{1,6}	{3,8}	{1,8}	{3,6}
$T$	5	5	2	2	0	0

$p = P(T \ge 5) = 2/6 = 0.33$. With four points the smallest possible p-value is 1/3: no amount of separation can be "significant." Permutation tests make the cost of a tiny sample visible.

Worked example · AoS Ex 10.19, by hand

Data $(X_1, X_2, Y_1) = (1, 9, 3)$, $T = |\bar X - \bar Y|$, observed $T = |5 - 3| = 2$. The six permutations give $T = 2, 2, 7, 7, 5, 5$. AoS reports $P(T > 2) = 4/6$. Note: with the more common convention $P(T \ge t_{\text{obs}})$, the p-value is $6/6 = 1$. Conventions differ, so check which one a source uses.

Analogy · shuffled name tags

If a drug does nothing, the "treatment" and "placebo" name tags are decoration. Peel them off, shuffle, and stick them back on at random. If the real labeling produces a bigger difference than nearly every random relabeling, the labels weren't decoration.

Check your understandingWhat's the difference between a permutation test and a bootstrap?

They simulate different worlds. A permutation test simulates the null: if labels don't matter, shuffling them is harmless, so the shuffled statistics form the null distribution and give an exact p-value. A bootstrap simulates sampling variability around what you observed, which gives SEs and intervals. Permutation for testing "no difference"; bootstrap for "how precise is my estimate."

Neyman–Pearson: the best possible test

Formal · CASI (2.20)–(2.23); AoS Thm 10.30

Testing a simple null density $f_0$ against a simple alternative $f_1$: $\alpha = P_{f_0}(\text{reject})$, $\beta = P_{f_1}(\text{retain})$. Let $L(x) = f_1(x)/f_0(x)$ and reject when $\log L(x) \ge c$. Lemma: only rules of this likelihood-ratio form are optimal; for any other rule there is a likelihood-ratio rule with smaller $\alpha$ and smaller $\beta$.

Work it by hand · one observation, two normals

Single $X$; $f_0 = N(0,1)$, $f_1 = N(1,1)$. $\log L(x) = x - \frac12$, so the likelihood-ratio rule is "reject when $x$ is large." For $\alpha = 0.05$, reject when $x \ge 1.645$. Then $\beta = P_{f_1}(X < 1.645) = \Phi(0.645) = 0.74$. Even the best possible test misses a 1-SD shift 74% of the time with one observation.

Worked example · CASI Fig 2.2, by hand

Ten observations, $f_0 = N(0,1)$ vs $f_1 = N(0.5,1)$. Then $\log L = 0.5\sum x_i - 1.25$, so the rule is just "reject when $\sum x_i$ is large." At $c = 0.75$: reject iff $\sum x_i \ge 4$. Under $f_0$, $\sum x_i \sim N(0, 10)$, so $\alpha = P(Z \ge 4/\sqrt{10} = 1.26) \approx 0.10$. Under $f_1$, $\sum x_i \sim N(5, 10)$, so $\beta = P(Z < -1/\sqrt{10}) \approx 0.38$. Sliding $c$ traces the whole $(\alpha, \beta)$ trade-off curve.

Likelihood ratio tests and χ² tests AoS §10.3–10.4, §10.6. For nested models, $\lambda = 2\log\big(\mathcal L(\hat\theta)/\mathcal L(\hat\theta_0)\big) \rightsquigarrow \chi^2_{r-q}$, where $r - q$ is the number of parameters fixed by the null (Thm 10.22). Pearson's $\chi^2 = \sum (X_j - np_{0j})^2/(np_{0j})$ is approximately $\chi^2_{k-1}$ for multinomial data (Thm 10.17; Mendel's peas, Ex 10.18). These generalize the Wald test to several parameters at once; the overall F-statistic at the top of a regression table is a relative.

Peeking: how the stopping rule changes the p-value

From the books · CASI §3.3

Each month you observe $x_i \sim N(\mu, 1)$ and plan to test $H_0: \mu = 0$ after 30 months, using $Z_i = \sum_{j\le i}x_j/\sqrt i$. At month 30, $Z_{30} = 1.66 > 1.645$: significant at 0.05. But the team peeked at month 20 ($Z_{20} = 0.79$) and would have stopped if it had been significant. The rule actually used, "declare significance if $Z_{20}$ or $Z_{30}$ exceeds 1.645," has false-positive rate 0.074, not 0.05.

Analogy · checking the oven

If you open the oven every five minutes and take the cake out the first time it looks done, you will take out more underdone cakes than if you checked once at the recipe's time. Each look is another chance to be fooled.

Why it matters · real case · dashboards that invite peeking

Online testing tools show live p-values, and users watch them and stop a test when "significant" appears. Optimizely, a large A/B testing platform, reported that continuous peeking, combined with many goals and variations, could raise the chance of a false winner more than fivefold under classical statistics. In January 2015 it switched its results engine to sequential methods designed to stay valid under continuous monitoring, developed with Stanford statisticians (Johari, Pekelis and Walsh). A false winner shipped to production costs whatever the change costs, and delivers none of the promised lift.

By hand, a bracket for two looks. Each look alone has a 5% false-positive rate. If the two looks used independent data, the chance that at least one fires would be $1 - 0.95^2 = 9.75\%$. They share data (month 30 includes months 1–20), so they agree more often than independent tests would, and the true rate falls in between: 7.4% in CASI's example. Each additional look pushes it higher.

CASI uses this to show a Bayesian/frequentist difference: the Bayesian posterior depends only on the data observed and is unaffected by the peek, while the frequentist p-value depends on what could have happened. CASI's "meter reader" story (§3.3) makes the same point: an engineer's unbiased average stops being unbiased when she learns her voltmeter would have capped any reading above 100, even though none of her readings was capped.

Think about itThe team checked their results at month 20 and month 30 and only reported the final analysis. Does the early look matter if they didn't stop?

Common mistake: No, the reported test is the one at month 30.

The real answer: Yes. Their decision rule was "stop and declare success if either look is significant," and that rule has a false-positive rate of 0.074, not 0.05 (CASI §3.3). The p-value depends on what you would have done, not only on what you did.

7 · Many tests at once

Go deeper: expanded page with more worked examples and interactive widgets →

One test at $\alpha = 0.05$ has a 5% chance of a false alarm. Run many, and false alarms become nearly certain. CASI calls this "the most troubling fact about large-scale testing: how easy it is to be fooled."

Story proof · 20 tests, 64%

Twenty true nulls, each tested at 0.05, independently. The chance a single test stays quiet is 0.95. The chance all twenty stay quiet is $0.95^{20} = 0.36$. So the chance of at least one false "discovery" is $1 - 0.36 = 0.64$. CASI's version: 100 tests at 0.05 give about five "significant" results even if every case is null.

Analogy · the Texas sharpshooter

A man fires at the side of a barn, then paints a target around the tightest cluster of holes and claims to be a marksman. Testing many hypotheses and reporting the one that came out significant is painting the target after shooting.

Why it matters · real cases · the dead salmon, and genome-wide significance

Bennett and colleagues put a dead Atlantic salmon in an fMRI scanner, showed it photographs of people in social situations, and ran a standard voxel-by-voxel analysis without correcting for multiple comparisons. Several voxels "responded." With thousands of voxels each tested separately, some had to. After correction, nothing survived. Their 2009 poster became a standard teaching example and pushed the field toward routine correction.

Genetics learned the lesson at industrial scale. Genome-wide association studies test on the order of a million genetic variants, and the field's standard threshold is $p < 5 \times 10^{-8}$, which is Bonferroni by hand: $0.05/1{,}000{,}000 = 5 \times 10^{-8}$.

In data work: slicing an experiment into hundreds of segments until one is significant ("the effect is real for Android users in Ohio") is the salmon experiment with a business label.

Controlling the family-wise error rate (FWER)

Formal · AoS Thm 10.24; CASI (15.7)–(15.10)

FWER = $P(\text{reject any true } H_{0i})$. Bonferroni: with $m$ tests, reject $H_{0i}$ if $p_i < \alpha/m$. Then FWER $\le \alpha$.

Proof. Let $R_i$ be the event that true null $i$ is falsely rejected. By the union bound, $P(\bigcup R_i) \le \sum P(R_i) \le m\cdot\alpha/m = \alpha$. No independence is needed.

Holm (CASI (15.9)–(15.10)): order the p-values, find the first $i$ with $p_{(i)} > \alpha/(m - i + 1)$, and reject everything before it. It also controls FWER and is slightly less conservative.

Work it by hand · five p-values, three corrections

p-values (sorted): 0.001, 0.012, 0.020, 0.040, 0.300. $\alpha = 0.05$, $m = 5$.

$i$	$p_{(i)}$	uncorrected (0.05)	Bonferroni (0.01)	Holm $\alpha/(m - i + 1)$
1	0.001	✓	✓	0.010 ✓
2	0.012	✓	✗	0.0125 ✓
3	0.020	✓	✗	0.0167 ✗ stop
4	0.040	✓	✗	—
5	0.300	✗	✗	—

Uncorrected: 4 rejections. Bonferroni: 1. Holm: 2 (stop at the first failure).

Story proof · the union bound

The chance that at least one of several alarms goes off can't exceed the sum of their individual chances: adding them counts every way of getting at least one alarm, and counts overlaps more than once. Give each of $m$ alarms a budget of $\alpha/m$, and the total can't exceed $\alpha$.

Worked examples

AoS Ex 10.25: 2638 genes at $\alpha = 0.05$: reject only if $p < 0.05/2638 = 0.000019$. CASI §15.1: 6033 prostate genes, one-sided Bonferroni threshold $z > \Phi^{-1}(1 - 0.05/6033) = 4.31$ (vs 1.645 for one test); only 4 genes pass. That is "very conservative": it is trying to avoid even one false rejection.

Think about itA paper tests 20 outcomes at $\alpha = 0.05$ and reports the one that was significant. How impressed should you be?

Common mistake: Very: $p < 0.05$ is significant.

The real answer: Not at all, until you know about the other 19. With 20 true nulls you expect one significant result by chance, and the chance of at least one is 64%. Ask how many tests were run.

Check your understandingDoes Bonferroni still work if the tests are correlated?

Yes. The union bound $P(\bigcup R_i) \le \sum P(R_i)$ needs no independence, so FWER $\le \alpha$ still holds. It becomes more conservative under positive correlation, because correlated tests overlap and the bound double-counts them. BH needs independence (or positive dependence) for its basic version; AoS's $C_m$ factor covers arbitrary dependence.

Controlling the false discovery rate (FDR)

Formal · AoS (10.8), Thm 10.26; CASI (15.11)–(15.15)

Among $R$ rejections, let $V$ be the false ones. The false discovery proportion is FDP = $V/R$ (0 if $R = 0$), and FDR = $E(\text{FDP})$.

Benjamini–Hochberg: order the p-values $p_{(1)} \le \dots \le p_{(m)}$. Find the largest $i$ with $p_{(i)} \le i\alpha/m$ (AoS includes a factor $C_m = 1$ for independent p-values, $\sum_{j=1}^m 1/j$ otherwise). Reject the hypotheses with the $i$ smallest p-values.

Theorem: FDR $\le \frac{m_0}{m}\alpha \le \alpha$, where $m_0$ is the number of true nulls. CASI writes it FDR$(D_q) = \pi_0 q$ with $\pi_0 = N_0/N$.

Work it by hand · Benjamini–Hochberg on the same five

BH line $i\alpha/m$: 0.01, 0.02, 0.03, 0.04, 0.05. Compare: $0.001 \le 0.01$ ✓, $0.012 \le 0.02$ ✓, $0.020 \le 0.03$ ✓, $0.040 \le 0.04$ ✓, $0.300 > 0.05$ ✗. Largest passing $i$ is 4: reject 4.

Ranking on the same data: Bonferroni (1) < Holm (2) < BH (4). Each controls a different error rate; BH's promise is that on average at most 5% of the rejected hypotheses are false discoveries.

Worked example · AoS Ex 10.28, by hand

Ten p-values: 0.00017, 0.00448, 0.00671, 0.00907, 0.01220, 0.33626, 0.39341, 0.53882, 0.58125, 0.98617. Use $\alpha = 0.05$.

$i$	$p_{(i)}$	BH line $i\alpha/m$	below?
1	0.00017	0.005	yes
2	0.00448	0.010	yes
3	0.00671	0.015	yes
4	0.00907	0.020	yes
5	0.01220	0.025	yes
6	0.33626	0.030	no

Uncorrected ($p < 0.05$): 5 rejections. Bonferroni ($p < 0.005$): 2. BH: 5. On these numbers BH agrees with the uncorrected count while still guaranteeing FDR $\le 0.05$.

Analogy · the spam filter

FWER control is a filter so strict that the chance of even one real email landing in spam is under 5%; it ends up letting lots of spam through. FDR control promises instead that, on average, at most 5% of what lands in your "important" folder is junk. For screening thousands of genes, the second promise is the useful one.

CASI's leukemia coda: to keep a low chance of "crying wolf over an innocuous gene," the FDR threshold is $t > 6.16$, and only 47 of the 7128 genes pass. Gene 136's $t = 3.01$ does not.

Check your understandingWhen would you prefer FDR control to FWER control?

FWER when a single false positive is costly and the number of tests is small (a handful of pre-specified clinical endpoints). FDR when you're screening many candidates and will follow up on the hits (thousands of genes), so tolerating a known small fraction of false leads buys much more power. CASI §15.2: FWER "usually proved too conservative for scientists working with N in the thousands."

8 · Inference for regression coefficients

Go deeper: expanded page with more worked examples and interactive widgets →

A regression table is the Wald test and normal-based interval applied to each coefficient. Everything in Sections 1–6 shows up; the only new ingredient is the formula for each coefficient's SE.

Formal · AoS Def 13.1, Thm 13.4, (13.7), Thm 13.7

Model: $Y_i = \beta_0 + \beta_1 X_i + \epsilon_i$, with $E(\epsilon_i \mid X_i) = 0$ and $V(\epsilon_i \mid X_i) = \sigma^2$. The constant-variance assumption is part of the definition.

Least squares: $\hat\beta_1 = \sum(X_i - \bar X)(Y_i - \bar Y)/\sum(X_i - \bar X)^2$, $\hat\beta_0 = \bar Y - \hat\beta_1\bar X$. Error variance: $\hat\sigma^2 = \frac{1}{n-2}\sum\hat\epsilon_i^2$ (unbiased; the MLE divides by $n$).

Thm 13.7: under normal errors, least squares is maximum likelihood.

Work it by hand · fit a line to five points

$x = \{1, 2, 3, 4, 5\}$, $y = \{1, 3, 2, 5, 4\}$. Means $\bar x = \bar y = 3$.

$x$	$y$	$x - 3$	$y - 3$	product	$(x - 3)^2$	fitted $0.6 + 0.8x$	residual
1	1	−2	−2	4	4	1.4	−0.4
2	3	−1	0	0	1	2.2	0.8
3	2	0	−1	0	0	3.0	−1.0
4	5	1	2	2	1	3.8	1.2
5	4	2	1	2	4	4.6	−0.6
			sum	8	10		0

$\hat\beta_1 = 8/10 = 0.8$; $\hat\beta_0 = 3 - 0.8(3) = 0.6$. RSS $= 0.16 + 0.64 + 1.00 + 1.44 + 0.36 = 3.6$; $\hat\sigma^2 = 3.6/(5 - 2) = 1.2$, $\hat\sigma = 1.10$. (The MLE would divide by 5: 0.72.)

Standard errors and tests for the coefficients

Formal · AoS Thms 13.8–13.9

Conditional on the $X$'s, with $s_X^2 = \frac1n\sum(X_i - \bar X)^2$:

$$V(\hat\beta \mid X) = \frac{\sigma^2}{n s_X^2}\begin{pmatrix}\frac1n\sum X_i^2 & -\bar X\\ -\bar X & 1\end{pmatrix}, \qquad \widehat{\text{se}}(\hat\beta_1) = \frac{\hat\sigma}{s_X\sqrt n}, \qquad \widehat{\text{se}}(\hat\beta_0) = \frac{\hat\sigma}{s_X\sqrt n}\sqrt{\frac{\sum X_i^2}{n}}.$$

Under appropriate conditions the estimators are consistent and asymptotically normal, $\hat\beta_j \pm z_{\alpha/2}\widehat{\text{se}}(\hat\beta_j)$ are approximate $1-\alpha$ intervals, and the Wald test of $H_0: \beta_1 = 0$ rejects when $|\hat\beta_1/\widehat{\text{se}}(\hat\beta_1)| > z_{\alpha/2}$.

Work it by hand · standard errors for the five-point line

Note $s_X\sqrt n = \sqrt{\sum(x_i - \bar x)^2} = \sqrt{10}$.

$\widehat{\text{se}}(\hat\beta_1) = 1.095/\sqrt{10} = 0.346$. $t = 0.8/0.346 = 2.31$ on 3 df; $t_{3,.025} = 3.182$: retain (p = 0.10). With $z$ (2.31 > 1.96) you'd wrongly reject.
95% interval: $0.8 \pm 3.182(0.346) = (-0.30, 1.90)$.
$\widehat{\text{se}}(\hat\beta_0) = 0.346\sqrt{55/5} = 1.15$ ($\sum x_i^2 = 55$).
The bow-tie: SE of the fitted line is $\hat\sigma\sqrt{1/5 + (x - 3)^2/10}$: 0.49 at $x = 3$, 0.85 at $x = 5$, 1.15 at $x = 6$.

Story proof · why spread-out X's give a precise slope

$\widehat{\text{se}}(\hat\beta_1) = \hat\sigma/(s_X\sqrt n)$ has three ingredients, each with a reason. Noisier points ($\hat\sigma$) make the line wobble more. More points ($\sqrt n$) average the noise down. And X values spread far apart ($s_X$) act like a long lever: the same vertical noise tilts a line through far-apart points much less than one through bunched-up points. Try balancing a ruler on two fingers close together, then far apart.

Worked example · AoS Ex 13.10 (2000 election, log scale)

Buchanan votes regressed on Bush votes across Florida counties: $\hat\beta_1 = 0.7303$, $\widehat{\text{se}} = 0.0358$. 95% interval: $0.7303 \pm 2(0.0358) = (0.66, 0.80)$. Wald statistic: $|0.7303 - 0|/0.0358 = 20.4$, p-value ≈ 0.

From the books · CASI §1.1 (kidney function)

157 volunteers, kidney score "tot" on age: $\hat\beta_0 = 2.86$, $\hat\beta_1 = -0.079$ per year. The fitted value's SE is 0.21 at age 20, 0.15 at ages 30–40, and 0.42 at age 80, smallest near the middle of the ages and growing at the ends. CASI draws ±2 SE bars and notes that their roughly 95% coverage depends on the linear model being right. A lowess fit has no SE formula at all; its bootstrap SEs are about double (0.71 at age 20).

Story proof · the bow-tie

The fitted line pivots around $(\bar x, \bar y)$. Near the pivot, uncertainty about the slope barely moves the line; far from it, a small tilt moves the line a lot. So the fitted value's SE is smallest at $\bar x$ and widens in both directions, a bow-tie shape. The standard formula makes this explicit: $\widehat{\text{se}}(\hat y(x)) = \hat\sigma\sqrt{1/n + (x - \bar x)^2/\sum(x_i - \bar x)^2}$ standard.

Prediction intervals

Formal · AoS Thm 13.11

To predict a new $Y_*$ at $x_*$, the uncertainty has two parts: the fitted line's uncertainty and the new point's own noise. With $\hat\xi_n^2 = \widehat{\text{se}}^2(\hat Y_*) + \hat\sigma^2$, an approximate $1-\alpha$ prediction interval is $\hat Y_* \pm z_{\alpha/2}\hat\xi_n$. It does not shrink to zero as $n$ grows, because $\hat\sigma^2$ doesn't.

Work it by hand · predict a sixth point

At $x_* = 6$: $\hat y_* = 0.6 + 0.8(6) = 5.4$. Line uncertainty: $\widehat{\text{se}}(\hat y_*) = 1.15$. Add the new point's own noise: $\hat\xi = \sqrt{1.15^2 + 1.2} = \sqrt{2.52} = 1.59$. 95% prediction interval with $t_3$: $5.4 \pm 3.182(1.59) = (0.35, 10.45)$. Five points can barely predict a sixth.

Worked example · AoS Ex 13.12 (Palm Beach)

Predicted log Buchanan votes, rounded for hand work: $-2.33 + 0.73(11.94) \approx 6.39$ (AoS: 6.388), with $\hat\xi_n = 0.0938$. Interval $(6.200, 6.578)$, or 493 to 717 votes. Palm Beach's actual log vote was 8.151 (3,467 votes), nearly 20 standard errors above the prediction.

Analogy · the average height vs. the next person

With enough data you can pin down the average height of adults to within a millimeter. You still can't predict the next person through the door to within a millimeter. Confidence intervals are about the average; prediction intervals are about the next person.

Why it matters · real case · a point forecast treated as a guarantee

In spring 1997 the National Weather Service forecast that the Red River would crest at 49 feet at Grand Forks, North Dakota. The levees stood at about 51 feet, and the city prepared for 52. The river crested at about 54 feet, the levees failed, and much of the city flooded and was evacuated. The forecasts carried no numerical statement of uncertainty, and the 49-foot number was read as nearly exact, partly because a 49-foot forecast in 1979 had been off by only a few inches. National Research Council 2003

The statistical lesson. A 2-foot margin between forecast and levee is safe only if the forecast's prediction interval is narrower than 2 feet. A prediction interval for one future outcome includes that outcome's own noise ($\hat\sigma^2$ in Thm 13.11), so it is much wider than a confidence interval for the mean, which is the interval that is easy to compute and to quote.

Think about itWith 10 times as many houses, will the prediction interval for one new house shrink toward zero?

Common mistake: Yes, more data shrinks every interval.

The real answer: No. It has two parts: uncertainty about the line, which shrinks with $n$, and the new house's own deviation from the line, $\hat\sigma$, which doesn't. With infinite data the prediction interval tends to $\hat y \pm 1.96\sigma$: about ±$87k for Ames, using $\hat\sigma \approx$ $44k.

Multiple regression

Formal · AoS Thm 13.13, (13.23)

With an $n \times k$ design matrix $X$ (a column of 1s for the intercept) and $X^\top X$ invertible:

$$\hat\beta = (X^\top X)^{-1}X^\top Y, \qquad V(\hat\beta \mid X) = \sigma^2(X^\top X)^{-1}, \qquad \hat\beta \approx N\big(\beta, \sigma^2(X^\top X)^{-1}\big),$$ $$\hat\sigma^2 = \frac{1}{n-k}\sum_{i=1}^n\hat\epsilon_i^2, \qquad \hat\beta_j \pm z_{\alpha/2}\,\widehat{\text{se}}(\hat\beta_j), \quad \widehat{\text{se}}(\hat\beta_j) = \text{square root of the } j\text{th diagonal of } \hat\sigma^2(X^\top X)^{-1}.$$

Work it by hand · the matrix formula

Same five points, $X$ has a column of 1s and the column $x$.

$$X^\top X = \begin{pmatrix}5 & 15\\ 15 & 55\end{pmatrix}, \quad \det = 275 - 225 = 50, \quad (X^\top X)^{-1} = \frac{1}{50}\begin{pmatrix}55 & -15\\ -15 & 5\end{pmatrix}, \quad X^\top y = \begin{pmatrix}15\\ 53\end{pmatrix}.$$ $$\hat\beta = \frac{1}{50}\begin{pmatrix}55(15) - 15(53)\\ -15(15) + 5(53)\end{pmatrix} = \frac{1}{50}\begin{pmatrix}30\\ 40\end{pmatrix} = \begin{pmatrix}0.6\\ 0.8\end{pmatrix}.$$

$\hat\sigma^2(X^\top X)^{-1}$ has diagonal $1.2 \times 55/50 = 1.32$ and $1.2 \times 5/50 = 0.12$, so the SEs are $\sqrt{1.32} = 1.15$ and $\sqrt{0.12} = 0.346$, matching the simple-regression formulas. The $-15$ off the diagonal says the intercept and slope estimates are negatively correlated.

The $n - k$ is the degrees of freedom left after estimating $k$ coefficients: the "Df Residuals" line of a regression table, and the df of the t-distribution its p-values use.

Reading a regression table

Column	What it is	Where it came from
`coef`	$\hat\beta_j$	least squares (= MLE under normal errors)
`std err`	$\widehat{\text{se}}(\hat\beta_j)$, from $\hat\sigma^2(X^\top X)^{-1}$	AoS Thm 13.13: assumes constant error variance
`t`	$\hat\beta_j/\widehat{\text{se}}$: standard errors from 0	the Wald statistic, AoS Def 10.3
`P>\|t\|`	$2P(t_{n-k} \ge \|t\|)$	AoS Thm 10.13, with $t_{n-k}$ for $Z$
`[0.025 0.975]`	$\hat\beta_j \pm t_{n-k,.025}\,\widehat{\text{se}}$	AoS (13.23); same as the test by Thm 10.10

Worked example · course (Ames, Iowa: 2930 sales, 11 coefficients, df 2919)

area 113.32 2.009 56.40 0.000 109.38 117.26. Holding age, central air, and style fixed, one more square foot is associated with a $113 higher sale price on average. $t = 113.32/2.009 = 56.4$; interval $113.32 \pm 1.961(2.009)$. For central air: $\hat\beta = 13{,}380$, $\widehat{\text{se}} = 3{,}547$, $t = 3.77$, interval $(6{,}428,\ 20{,}332)$.

Think about itI measure house area in square meters instead of square feet. What happens to the coefficient, its SE, its t, and its p-value?

Common mistake: Everything changes, so the result might become insignificant.

The real answer: One square meter is 10.764 square feet, so the coefficient and its SE both multiply by 10.764 (about $1,220 per m²). Their ratio, the t statistic, and the p-value are unchanged. A test can't depend on the units you chose, and that's also why a big coefficient doesn't mean an important variable.

Think about itArea has $t = 56$ and the smallest p-value in the table. Is it the most important predictor of price?

Common mistake: Yes, it's the most significant.

The real answer: Not necessarily. $t$ measures how precisely the effect is estimated, which depends on how much area varies and how many houses there are, not how much area matters. Importance needs a question in real units: how much does price change over a typical range of area, compared with having central air?

When the formula is wrong: unequal error variance

The std err column rests on $V(\epsilon_i \mid X_i) = \sigma^2$ for every observation. When errors are bigger for some observations than others (heteroskedasticity), the formula is wrong, sometimes too small and sometimes too large. Three routes don't use the assumption: the robust "sandwich" formula, the pairs bootstrap, and the jackknife.

Formal · standard (not in AoS or CASI)

The heteroskedasticity-robust covariance replaces $\hat\sigma^2(X^\top X)^{-1}$ by

$$(X^\top X)^{-1}\Big(\sum_i \hat\epsilon_i^2\,x_ix_i^\top\Big)(X^\top X)^{-1}$$

(HC0; HC1 multiplies by $n/(n-k)$). Each observation contributes its own squared residual instead of a shared $\hat\sigma^2$.

Work it by hand · robust SE for the five-point slope

For a simple slope the sandwich reduces to $\widehat{\text{se}}^2_{HC0} = \sum(x_i - \bar x)^2\hat\epsilon_i^2 / \big(\sum(x_i - \bar x)^2\big)^2$.

Numerator: $4(0.16) + 1(0.64) + 0(1.00) + 1(1.44) + 4(0.36) = 4.16$. So $\widehat{\text{se}}_{HC0} = \sqrt{4.16}/10 = 0.204$ (HC1: $\times\sqrt{5/3}$ gives 0.263), vs. classical 0.346.

Robust is smaller here because the biggest residuals (−1.0 at $x = 3$, 1.2 at $x = 4$) sit near the middle, where they barely affect the slope. On Ames it went the other way (big residuals on big, high-leverage houses). With five points, any SE is itself very noisy.

Ames coefficient	classical	robust (HC1)	pairs bootstrap	jackknife	p: classical → robust
area	2.0	6.5	6.3	6.6	<0.001 → <0.001
2Story	2,159	3,625	3,584	3,679	<0.001 → <0.001
1.5Unf	10,301	4,919	5,009	5,156	0.074 → <0.001
SFoyer	5,029	3,018	3,038	3,058	0.23 → 0.045

course Three assumption-free routes agree; the classical formula is the odd one out, too small for area by a factor of three and too large for two house styles. Residual SD by house-size quartile: $21k, $27k, $38k, $71k. Big houses are harder to price.

Analogy · a postal scale

A postal scale accurate to a gram for letters and to a kilogram for furniture has no single "accuracy." The classical formula reports one average accuracy for every package; the robust formula reports each package's own.

The general rule: when two legitimate routes to an SE disagree, one of them rests on an assumption the data violate. Find it. Here it is $V(\epsilon \mid X) = \sigma^2$, which AoS writes into the definition of the model.

Think about itThe regression's $R^2$ is 0.69, quite high. Does that mean the standard errors are trustworthy?

Common mistake: Yes, the model fits well.

The real answer: No. $R^2$ measures how much variation the line explains; it says nothing about whether the error variance is constant or the observations are independent, which is what the SE formula assumes. Ames has a respectable $R^2$ and a classical SE on area that is three times too small.

Check your understandingWhy does the pairs bootstrap agree with the robust SE and not the classical one?

Resampling whole rows $(x_i, y_i)$ keeps each house's residual attached to its own $x_i$, so the big errors stay with the big houses, and the resampled slopes vary the way they really would. The classical formula pools all residuals into one $\hat\sigma^2$ and spreads that evenly over every $x$. The robust sandwich formula, like the pairs bootstrap, lets each observation contribute its own squared residual. A residual bootstrap (shuffle residuals across rows) would re-impose constant variance and agree with the classical answer instead.

9 · Failure catalog

Go deeper: expanded page with more worked examples and interactive widgets →

Every interval is $\hat\theta \pm c\cdot\widehat{\text{se}}$, and every test is $\hat\theta$ over $\widehat{\text{se}}$. Each failure breaks one of three things: the SE, the critical value $c$, or the reading.

Failure	What breaks	Symptom	Fix	Source
SD reported as SE	reading	"8.59 ± 8.17" vs "8.59 ± 0.85"	say which one	course
Dependent data	SE (independence step of σ²/n)	correlated series (ρ = 0.7, n = 100): s/√n = 0.14, truth 0.33; "95%" covers 59%	model the dependence; block bootstrap	course; CASI §10.3
Paired data analyzed as independent	SE	wrong SE for algorithm comparison on one test set	analyze differences $D_i$	AoS Ex 10.7
Unequal error variance in regression	SE ($V(\epsilon\mid X) = \sigma^2$)	Ames area SE 2.0 vs 6.5	robust SE, pairs bootstrap	course
Rare categories	SE (too few points)	8-house style: SE 16k–33k by method	pool categories; don't trust it	course
Jackknife on a non-smooth statistic	SE	median SE 1.43 vs bootstrap 0.75; lowess at age 25	bootstrap	AoS §8.5.1; CASI Fig 10.1
No finite variance	SE does not exist	Cauchy: s/√n never settles	robust statistics (median)	AoS §2.4
Bootstrap of an extreme statistic	SE	max, min: resamples can't exceed the data	parametric methods	standard
Non-representative sample	everything	resampling reproduces the bias	better data	course
Small n, normal data	c	z covers 88% at n = 5	t with n − 1 df	AoS §10.10.2; course
Small n, skewed data	c	even t covers 86% at n = 5, missing low	more data, transform, BCa/bootstrap-t	CASI Ch 11; course
Symmetric interval for asymmetric problem	c	Poisson 10: (3.8, 16.2) vs exact (5.1, 17.8)	percentile, BCa	CASI Ch 11
"95% chance θ is in the interval"	reading	Berger–Wolpert {16}: 75% interval, certain answer	coverage language	AoS §6.3.2
p as P(H₀ \| data)	reading	"3% chance the null is true"	Bayesian analysis if that's the question	AoS §10.2
Large p as proof of H₀	reading	low power looks like "no effect"	report the interval	AoS §10.2
Significant ≠ important	reading	area t = 56	report the interval	AoS Fig 10.2
Multiple testing	c	20 tests: 64% chance of a false discovery	Bonferroni, Holm, BH	AoS §10.7; CASI Ch 15
Peeking / optional stopping	c	false-positive rate 0.074, not 0.05	pre-register; sequential methods	CASI §3.3
Winner's curse	estimate	top gene 5.29, empirical Bayes 4.11	shrinkage	CASI §3.3, §4.1
Model-dependent coverage	SE	±2SE bars valid only if the line is right	check the model; bootstrap a flexible fit	CASI §1.1

Analogy · dependence

Ask 100 people who all read the same newspaper for their opinion and you have far fewer than 100 independent opinions. The formula $s/\sqrt n$ counts all 100 as fresh information, so it is too confident.

Why it matters · scenario · 1,000 rows, 50 people

Illustrative numbers. An experiment logs 1,000 sessions from 50 users, 20 each, and the analysis treats the sessions as independent. Sessions from the same user resemble each other. With $m$ observations per cluster and within-cluster correlation $\rho$, the variance of the mean is multiplied by the design effect $1 + (m - 1)\rho$ standard.

$m = 20$, $\rho = 0.5$: design effect $1 + 19(0.5) = 10.5$.
The true SE is $\sqrt{10.5} \approx 3.2$ times the naive $s/\sqrt{1000}$.
The data carry about as much information as $1000/10.5 \approx 95$ independent sessions: closer to the 50 users than to the 1,000 rows.

Fixes: analyze per-user averages, use cluster-robust SEs, or bootstrap whole users rather than rows.

10 · Formula sheet

Object	Formula	Reference
Standard error	$\text{se}(\hat\theta) = \sqrt{V(\hat\theta)}$	AoS (6.5)
MSE	$\text{bias}^2 + \text{variance}$	AoS Thm 6.9
SE of a mean	$s/\sqrt n$	CASI (1.2)
SE of a proportion	$\sqrt{\hat p(1-\hat p)/n}$	AoS Ex 6.8
CLT	$\sqrt n(\bar X_n - \mu)/S_n \rightsquigarrow N(0,1)$	AoS Thms 5.8, 5.10
Delta method	$\widehat{\text{se}}(g(\hat\theta)) \approx \|g'(\hat\theta)\|\,\widehat{\text{se}}(\hat\theta)$	AoS Thm 9.24
DKW band	$\hat F_n \pm \sqrt{\log(2/\alpha)/(2n)}$	AoS Thm 7.5
Fisher SE	$1/\sqrt{n I(\hat\theta)}$	AoS Thm 9.18; CASI (10.21)
Bootstrap SE	SD of $\hat\theta^_1,\dots,\hat\theta^_B$	AoS (8.1); CASI (10.16)
Jackknife SE	$\sqrt{\frac{n-1}{n}\sum(\hat\theta_{(i)} - \hat\theta_{(\cdot)})^2}$	CASI (10.6)
Noise in ŝe (normal data)	$\approx 1/\sqrt{2(n-1)}$ relative	standard
Normal interval	$\hat\theta \pm z_{\alpha/2}\widehat{\text{se}}$; $z = 1.645, 1.96, 2.576$	AoS Thm 6.16
Pivotal bootstrap interval	$(2\hat\theta - \theta^_{1-\alpha/2},\ 2\hat\theta - \theta^_{\alpha/2})$	AoS (8.6)
Percentile interval	$(\theta^_{\alpha/2},\ \theta^_{1-\alpha/2})$	AoS §8.3; CASI §11.2
Wald statistic	$W = (\hat\theta - \theta_0)/\widehat{\text{se}}$	AoS (10.5)
p-value (Wald)	$2\Phi(-\|w\|)$	AoS Thm 10.13
One-sample t	$\sqrt n(\bar X - \mu_0)/S_n \sim t_{n-1}$	AoS §10.10.2
Two-sample t	$(\bar x_2 - \bar x_1)/(\hat\sigma\sqrt{1/n_1 + 1/n_2}) \sim t_{n_1+n_2-2}$	CASI (2.17)
Bonferroni	reject if $p_i < \alpha/m$	AoS Thm 10.24
Benjamini–Hochberg	largest $i$ with $p_{(i)} \le i\alpha/m$	AoS Thm 10.26; CASI (15.14)
OLS	$\hat\beta = (X^\top X)^{-1}X^\top Y$, $V = \sigma^2(X^\top X)^{-1}$, $\hat\sigma^2 = \text{RSS}/(n-k)$	AoS Thm 13.13
Slope SE	$\hat\sigma/(s_X\sqrt n)$	AoS Thm 13.8
Prediction interval	$\hat Y_* \pm z_{\alpha/2}\sqrt{\widehat{\text{se}}^2(\hat Y_*) + \hat\sigma^2}$	AoS Thm 13.11

11 · Analogy and story-proof bank

Every analogy and story proof in this guide, in one place, for review.

Analogies

Idea	Analogy	Section
Sampling distribution	a thousand parallel research teams	1
Bias and variance	archery: cluster center vs. spread	1
SD vs. SE	people's weights vs. the average's wobble	1
Convergence in distribution	matching histograms, not darts near the bullseye	2
CLT	stadium crowd noise	2
Delta method	funhouse mirror: slope is the stretch	2
Plug-in	the architect's scale model	3
Fisher information	sharpness of a mountain peak	3
Bootstrap	ladling soup from one pot	3
Jackknife failure	a table with a hinge	3
Pivotal statistic	distance in body lengths	3
Uncertain SE	a rubber ruler	4
Confidence interval	ring toss: the peg doesn't move	5
BC / BCa	fitting a frame to a curved wall	5
Hypothesis test	a criminal trial (AoS's)	6
p-value	surprise meter; the strictest judge who'd convict	6
Permutation test	shuffled name tags	6
Peeking	opening the oven every five minutes	6
Multiple testing	the Texas sharpshooter	7
FDR vs. FWER	the spam folder	7
Prediction interval	average height vs. the next person	8
Heteroskedasticity	a postal scale for letters and furniture	8
Dependence	100 readers of the same newspaper	9

Story proofs

Result	The story	Section
MSE = bias² + variance	Pythagoras: scatter is perpendicular to bias on average	1
$V(\bar X) = \sigma^2/n$	random walk of errors, divided by $n$	2
CLT	repeated smearing keeps only center and spread	2
MLE is normal with SE $1/\sqrt{nI}$	CLT on the score, LLN on the curvature	3
Resamples omit ~37%	$(1 - 1/n)^n \to e^{-1}$	3
Number of bootstrap samples	stars and bars: $\binom{2n-1}{n}$	3
Jackknife = $s/\sqrt n$	leave-one-out means are $n-1$ times too close	3
Noisy SE fattens tails	small denominators blow up; big ones only shrink a little	4
t distribution	a normal over a random scale is a mixture of bells	4
Pivotal bootstrap interval	the long tail flips to the other side	5
Percentile interval	monotone maps carry percentiles along	5
Test ⇔ interval	distance is symmetric	6
p uniform under $H_0$	top 5% of the null happens 5% of the time	6
20 tests, 64%	$1 - 0.95^{20}$	7
Bonferroni	the union bound: alarms' chances add at most	7
Slope SE	a long lever tilts less	8
Bow-tie of fitted SEs	the line pivots at $\bar x$	8

Why it matters: real cases and scenarios

Idea	Case	Kind	Section
SE depends on $n$	Gates Foundation small schools	real	1
Bias doesn't shrink with $n$	Literary Digest 1936 poll	real	1
Biased can beat unbiased	Efron–Morris batting averages; star ratings	real + scenario	1
Independence in σ²/n	mortgage pools before 2007–08	real	2
CLT and heavy tails	one big spender in an A/B test	scenario	2
Delta method	click-through-rate SEs (Microsoft)	real	2
Bootstrap a percentile	p95 latency SLA	scenario	3
Small-sample t	Gosset at Guinness	real	4
Reading an interval	Hoekstra et al. survey	real	5
Power and replication	Amgen and Bayer replication audits	real	6
Paired comparisons	two models, one test set	scenario	6
Significant vs. important	Bing ad-headline test	real	6
p is not P(H₀ \| data)	Sally Clark	real	6
Peeking	Optimizely's 2015 switch to sequential tests	real	6
Multiple testing	dead salmon fMRI; GWAS threshold	real	7
Prediction intervals	1997 Red River flood forecast	real	8
Clustered data	sessions vs. users	scenario	9

The guide also contains 31 "Think about it" and 17 "Check your understanding" questions, placed at the end of the topic each one tests.

Symbol	Meaning
\(F\)	the unknown distribution that generated the data (the population)
\(\hat F_n\)	the empirical distribution: probability \(1/n\) on each data point
\(\theta = T(F)\)	the parameter: a fixed number, some feature of \(F\)
\(\hat\theta_n\)	an estimate (a number) or estimator (a random variable); CASI writes \(\hat\theta\) vs. \(\hat\Theta\)
\(\text{se}\), \(\widehat{\text{se}}\)	the true standard error, and an estimate of it
\(\hat\theta^*_b\)	the statistic computed on bootstrap sample \(b\)
\(\hat\theta_{(i)}\)	the statistic with observation \(i\) left out (jackknife)
\(z_{\alpha/2}\)	the standard normal value with \(\alpha/2\) above it: \(z_{.025} = 1.96\)
\(\Phi\)	the standard normal CDF

\(\bar x\)	1	2	3	4	5	total
probability	\(\frac19\)	\(\frac29\)	\(\frac39\)	\(\frac29\)	\(\frac19\)	1
deviation \(\bar x - 3\)	\(-2\)	\(-1\)	0	1	2
squared deviation	4	1	0	1	4
probability × squared deviation	\(\frac49\)	\(\frac29\)	0	\(\frac29\)	\(\frac49\)	\(\frac{12}{9} = \frac43\)

WHAT YOU CANNOT SAY	WHAT YOU CAN SAY INSTEAD
"\(\theta\) is distributed around 4 with SD 1.15."	"Estimates from this procedure scatter around \(\theta\) with SD 1.15, and 4 is one of them."
"There is a \(\frac79\) probability that \(\theta\) is within 1 of 4."	"The rule \(\bar X\) lands within 1 of \(\theta\) with probability \(\frac79\), whatever \(\theta\) is, and my 4 came from that rule."
"The interval \(4 \pm 1\) has a \(\frac79\) chance of containing \(\theta\)."	"\(4 \pm 1\) came from a rule that covers \(\theta\) in \(\frac79\) of samples. This interval either contains \(\theta\) or it doesn't." (Here it does: \([3, 5]\) contains 3.)
"The SE measures how uncertain \(\theta\) is."	"The SE measures how much my estimate would vary from sample to sample."
"\(P(\theta = 3) = \frac39\)."	"\(P(\bar X = 3) = \frac39\)." Probabilities attach to \(\bar X\), never to \(\theta\).

\(\|x_1 - x_2\|\)	0	2	4	average
samples (of 9)	3	4	2
\(s^2\)	0	2	8	\(\frac{0 + 8 + 16}{9} = \frac83\) ✓ unbiased
\(\widehat{\text{se}}\)	0	1	2	\(\frac{0 + 4 + 4}{9} = \frac89 = 0.89\), not 1.15

\(\|x_1 - x_2\|\)	how many samples	\(\hat\sigma^2\)	\(s^2\)
0	3	0	0
2	4	1	2
4	2	4	8

Name	Data	Why it's convenient
Toy data	\(\{2, 4, 5, 6, 8\}\)	mean 5, deviations \(-3, -1, 0, 1, 3\), \(s^2 = 5\), SE exactly 1
Tiny population	\(\{1, 3, 5\}\), samples of 2	only 9 samples, so the whole sampling distribution fits in a row; \(\sigma^2 = 8/3\)
Bootstrap toy	\(\{1, 3, 5\}\), resamples of 3	only 10 distinct resamples, so the ideal bootstrap can be listed
Two groups	\(\{2, 4, 6\}\) vs. \(\{5, 7, 9\}\)	means 4 and 7, pooled variance 4
Five-point line	\(x = \{1, \dots, 5\}\), \(y = \{1, 3, 2, 5, 4\}\)	slope 0.8, intercept 0.6, \(\hat\sigma^2 = 1.2\)
Coins	5 or 100 flips	p-values and power by counting

	mean	bias	variance	MSE = bias² + var
\(\hat\sigma^2\) (divide by \(n\))	\(\frac{0+4+8}{9} = \frac43\)	\(-\frac43\)	\(\frac{20}{9}\)	\(\frac{16}{9} + \frac{20}{9} = 4\)
\(s^2\) (divide by \(n-1\))	\(\frac83\)	0	\(\frac{80}{9}\)	\(\frac{80}{9} \approx 8.9\)

\(n\)	4	16	100	400
se	1	0.5	0.2	0.1
bound on \(P(\text{miss by} > 1)\)	1 (useless)	0.25	0.04	0.01

Result in this guide	Kind of convergence
MSE → 0 (bias → 0 and se → 0)	quadratic mean, which gives consistency
Weak LLN, \(\bar X_n \to \mu\)	probability
Consistency of an estimator; the MLE	probability
CLT, \(\sqrt n(\bar X_n - \mu)/\sigma\)	distribution
CLT with \(S_n\) (Thm 5.10)	distribution, via Slutsky
Delta method	distribution, via a Taylor step plus Slutsky
MLE asymptotic normality	distribution (CLT on the score, LLN on the curvature, Slutsky to combine)

Missing ingredient	Example	What goes wrong	Fix
Independence	time series; repeated measures; many rows per person	covariances don't vanish, so \(\sigma/\sqrt n\) is too small. Lag-1 correlation 0.7, \(n = 100\): true SE ≈ 2.4 × \(s/\sqrt n\); "95%" covers about 59%	model the dependence; bootstrap blocks or clusters
Finite variance	Cauchy; Pareto-type tails (incomes, losses, views)	the mean of \(n\) Cauchy draws is Cauchy for every \(n\); with very heavy tails one value can move the mean arbitrarily	robust statistics (median, trimmed mean); model the tail
Enough data for the skew	car prices at small \(n\); rare events	Berry–Esseen error grows with skew. Cars, \(n = 5\), estimated SE: 21% of "95%" intervals miss low, 1% high	more data; transform (log); bootstrap-t or BCa
A sum underneath	sample max or min; Uniform(0, θ) via the max; parameters at a boundary	maxima of normals tend to a Gumbel, not a normal; the max's error shrinks like \(1/n\) with an exponential limit	extreme-value theory; exact or likelihood methods
A correct SE	small \(n\); a failing bootstrap; classical regression SE under unequal variance	standardizing by a wrong or noisy SE: t-like tails (Section 4); Ames area SE a third of the truth (Section 8)	t or bootstrap-t; robust SE; block bootstrap
Random sampling	convenience samples; selecting the largest of many estimates	the CLT controls variance, not bias: a precise normal interval around the wrong number (CASI's winner's curse: 5.29 vs ≈ 4.1)	better design; shrinkage

\(g(\bar x)\)	\(g'(x)\)	\(\|g'(5)\|\)	\(\widehat{\text{se}}(g(\bar x))\)
\(\bar x^2 = 25\)	\(2x\)	10	10
\(\sqrt{\bar x} = 2.24\)	\(\frac{1}{2\sqrt x}\)	0.224	0.22
\(\log \bar x = 1.61\)	\(1/x\)	0.2	0.20
\(1/\bar x = 0.2\)	\(-1/x^2\)	0.04	0.04

\(g(\bar x)\)	values at \(\bar x = 1, \dots, 5\)	slope at \(\mu = 3\)	delta-method SE	exact SE
\(\bar x^2\)	1, 4, 9, 16, 25	6	\(6 \times 1.15 = 6.93\)	7.09
\(\log \bar x\)	0, 0.69, 1.10, 1.39, 1.61	\(1/3\)	\(1.15/3 = 0.38\)	0.46
\(1/\bar x\)	1, 0.50, 0.33, 0.25, 0.20	\(-1/9\)	\(1.15/9 = 0.13\)	0.23

Device	What it computes	Where the unknown \(F\) is replaced	Book
1 · Plug-in	a formula like \(\sqrt{\text{var}_F(X)/n}\)	at the end: \(\text{var}_F\) → sample variance	CASI (2.8)–(2.9); AoS Ch 7
2 · Taylor / delta	\(\|g'(\theta)\|\,\sigma/\sqrt n\)	at the end: \(\theta \to \hat\theta\)	CASI (2.10); AoS 5.13, 9.24
3 · MLE theory	\(1/\sqrt{n\,I(\theta)}\)	at the end: \(\theta \to \hat\theta\)	CASI (4.27); AoS 9.18
4 · Bootstrap	SD of \(t(X^)\), \(X^ \sim \hat F\)	first: simulate from \(\hat F\)	CASI §10.2; AoS Ch 8
5 · Pivotal statistic	an exact distribution, e.g. \(t_{n-1}\)	nowhere: the unknown cancels	CASI (2.17)–(2.19)

Parameter	Plug-in calculation	Value
mean	\(25/5\)	5
\(E[X^2]\) (linear functional, \(r(x) = x^2\))	\((4 + 16 + 25 + 36 + 64)/5\)	29
variance	\(29 - 5^2\)	4 (divisor \(n\))
\(P(X > 5)\)	\(r(x) = I(x > 5)\): two of five	0.4
median	middle value	5
skewness	cubed deviations \(-27, -1, 0, 1, 27\) sum to 0	0

Parameter	\(T(F)\)	Plug-in \(T(\hat F_n)\)	AoS
Mean	\(\int x\,dF\)	\(\bar X_n\)	Ex 7.10
Variance	\(\int x^2 dF - (\int x\,dF)^2\)	\(\frac1n\sum(X_i - \bar X_n)^2\) (divisor \(n\))	Ex 7.11
Skewness	\(E(X-\mu)^3/\sigma^3\)	\(\frac1n\sum(X_i-\bar X)^3/\hat\sigma^3\)	Ex 7.12
Correlation	\(\text{Cov}(X,Y)/(\sigma_X\sigma_Y)\)	sample correlation	Ex 7.13
Quantile	\(F^{-1}(p)\)	\(\hat F_n^{-1}(p) = \inf\{x : \hat F_n(x) \ge p\}\)	Ex 7.14

	Sampling distribution	Bootstrap distribution
Samples come from	the population \(F\) (unknown)	your sample, treated as the population (\(\hat F_n\))
How many exist	one, fixed by \(F\) and \(n\)	one per dataset: every sample gives a different one
Centered at	the true \(\theta\) (if unbiased)	your estimate \(\hat\theta\)
Spread	the true SE	an estimate of the SE
Values it can take	anything the population can produce	only combinations of your data points
Error	none: it is the truth	statistical (\(\hat F_n \ne F\), shrinks with \(n\)) plus Monte Carlo (finite \(B\), shrinks with \(B\))

student's sample	bootstrap center \(\bar x\)	plug-in \(\hat\sigma^2\)	bootstrap SE	vs. truth (3, 0.94)
\(\{1, 3, 5\}\)	3	\((4 + 0 + 4)/3 = 8/3\)	\(\sqrt{8/9} = 0.94\)	exact match
\(\{3, 5, 5\}\)	13/3 = 4.33	\((16/9 + 4/9 + 4/9)/3 = 8/9\)	\(\sqrt{8/27} = 0.54\)	center off by 1.33; SE 40% too small
\(\{3, 3, 3\}\)	3	0	0	every resample is \(\{3, 3, 3\}\): a single spike

interval	covers \(\mu = 3\) when	coverage
\(\bar x \pm 0.5\)	\(\bar x = 3\)	3/9 = 33%
\(\bar x \pm 1.5\)	\(\bar x \in \{2, 3, 4\}\)	7/9 = 78%
\(\bar x \pm 2.5\)	always	9/9 = 100%

method	calculation	80% interval
normal	\(5 \pm 1.2816 \times 1.02\)	(3.69, 6.31)
percentile	\((\theta^_{.10}, \theta^_{.90})\)	(4.0, 6.2)
pivotal	\((2 \cdot 5 - 6.2,\ 2 \cdot 5 - 4.0)\)	(3.8, 6.0)

	Retain \(H_0\)	Reject \(H_0\)
\(H_0\) true	correct	type I error
\(H_1\) true	type II error	correct

Column	What it is	Where it came from
`coef`	\(\hat\beta_j\)	least squares (= MLE under normal errors)
`std err`	\(\widehat{\text{se}}(\hat\beta_j)\), from \(\hat\sigma^2(X^\top X)^{-1}\)	AoS Thm 13.13: assumes constant error variance
`t`	\(\hat\beta_j/\widehat{\text{se}}\): standard errors from 0	the Wald statistic, AoS Def 10.3
`P>\|t\|`	\(2P(t_{n-k} \ge \|t\|)\)	AoS Thm 10.13, with \(t_{n-k}\) for \(Z\)
`[0.025 0.975]`	\(\hat\beta_j \pm t_{n-k,.025}\,\widehat{\text{se}}\)	AoS (13.23); same as the test by Thm 10.10

Object	Formula	Reference
Standard error	\(\text{se}(\hat\theta) = \sqrt{V(\hat\theta)}\)	AoS (6.5)
MSE	\(\text{bias}^2 + \text{variance}\)	AoS Thm 6.9
SE of a mean	\(s/\sqrt n\)	CASI (1.2)
SE of a proportion	\(\sqrt{\hat p(1-\hat p)/n}\)	AoS Ex 6.8
CLT	\(\sqrt n(\bar X_n - \mu)/S_n \rightsquigarrow N(0,1)\)	AoS Thms 5.8, 5.10
Delta method	\(\widehat{\text{se}}(g(\hat\theta)) \approx \|g'(\hat\theta)\|\,\widehat{\text{se}}(\hat\theta)\)	AoS Thm 9.24
DKW band	\(\hat F_n \pm \sqrt{\log(2/\alpha)/(2n)}\)	AoS Thm 7.5
Fisher SE	\(1/\sqrt{n I(\hat\theta)}\)	AoS Thm 9.18; CASI (10.21)
Bootstrap SE	SD of \(\hat\theta^_1,\dots,\hat\theta^_B\)	AoS (8.1); CASI (10.16)
Jackknife SE	\(\sqrt{\frac{n-1}{n}\sum(\hat\theta_{(i)} - \hat\theta_{(\cdot)})^2}\)	CASI (10.6)
Noise in ŝe (normal data)	\(\approx 1/\sqrt{2(n-1)}\) relative	standard
Normal interval	\(\hat\theta \pm z_{\alpha/2}\widehat{\text{se}}\); \(z = 1.645, 1.96, 2.576\)	AoS Thm 6.16
Pivotal bootstrap interval	\((2\hat\theta - \theta^_{1-\alpha/2},\ 2\hat\theta - \theta^_{\alpha/2})\)	AoS (8.6)
Percentile interval	\((\theta^_{\alpha/2},\ \theta^_{1-\alpha/2})\)	AoS §8.3; CASI §11.2
Wald statistic	\(W = (\hat\theta - \theta_0)/\widehat{\text{se}}\)	AoS (10.5)
p-value (Wald)	\(2\Phi(-\|w\|)\)	AoS Thm 10.13
One-sample t	\(\sqrt n(\bar X - \mu_0)/S_n \sim t_{n-1}\)	AoS §10.10.2
Two-sample t	\((\bar x_2 - \bar x_1)/(\hat\sigma\sqrt{1/n_1 + 1/n_2}) \sim t_{n_1+n_2-2}\)	CASI (2.17)
Bonferroni	reject if \(p_i < \alpha/m\)	AoS Thm 10.24
Benjamini–Hochberg	largest \(i\) with \(p_{(i)} \le i\alpha/m\)	AoS Thm 10.26; CASI (15.14)
OLS	\(\hat\beta = (X^\top X)^{-1}X^\top Y\), \(V = \sigma^2(X^\top X)^{-1}\), \(\hat\sigma^2 = \text{RSS}/(n-k)\)	AoS Thm 13.13
Slope SE	\(\hat\sigma/(s_X\sqrt n)\)	AoS Thm 13.8
Prediction interval	\(\hat Y_* \pm z_{\alpha/2}\sqrt{\widehat{\text{se}}^2(\hat Y_*) + \hat\sigma^2}\)	AoS Thm 13.11

Result	The story	Section
MSE = bias² + variance	Pythagoras: scatter is perpendicular to bias on average	1
\(V(\bar X) = \sigma^2/n\)	random walk of errors, divided by \(n\)	2
CLT	repeated smearing keeps only center and spread	2
MLE is normal with SE \(1/\sqrt{nI}\)	CLT on the score, LLN on the curvature	3
Resamples omit ~37%	\((1 - 1/n)^n \to e^{-1}\)	3
Number of bootstrap samples	stars and bars: \(\binom{2n-1}{n}\)	3
Jackknife = \(s/\sqrt n\)	leave-one-out means are \(n-1\) times too close	3
Noisy SE fattens tails	small denominators blow up; big ones only shrink a little	4
t distribution	a normal over a random scale is a mixture of bells	4
Pivotal bootstrap interval	the long tail flips to the other side	5
Percentile interval	monotone maps carry percentiles along	5
Test ⇔ interval	distance is symmetric	6
p uniform under \(H_0\)	top 5% of the null happens 5% of the time	6
20 tests, 64%	\(1 - 0.95^{20}\)	7
Bonferroni	the union bound: alarms' chances add at most	7
Slope SE	a long lever tilts less	8
Bow-tie of fitted SEs	the line pivots at \(\bar x\)	8

Idea	Case	Kind	Section
SE depends on \(n\)	Gates Foundation small schools	real	1
Bias doesn't shrink with \(n\)	Literary Digest 1936 poll	real	1
Biased can beat unbiased	Efron–Morris batting averages; star ratings	real + scenario	1
Independence in σ²/n	mortgage pools before 2007–08	real	2
CLT and heavy tails	one big spender in an A/B test	scenario	2
Delta method	click-through-rate SEs (Microsoft)	real	2
Bootstrap a percentile	p95 latency SLA	scenario	3
Small-sample t	Gosset at Guinness	real	4
Reading an interval	Hoekstra et al. survey	real	5
Power and replication	Amgen and Bayer replication audits	real	6
Paired comparisons	two models, one test set	scenario	6
Significant vs. important	Bing ad-headline test	real	6
p is not P(H₀ \| data)	Sally Clark	real	6
Peeking	Optimizely's 2015 switch to sequential tests	real	6
Multiple testing	dead salmon fMRI; GWAS threshold	real	7
Prediction intervals	1997 Red River flood forecast	real	8
Clustered data	sessions vs. users	scenario	9