DS 5030 · Understanding Uncertainty · Go deeper · §8
Where every column of a regression table comes from, on a line you can drag.
A regression table is a Wald test and a normal (or t) interval for each coefficient. On five points every number can be produced by hand. Data: \(x = 1, 2, 3, 4, 5\), \(y = 1, 3, 2, 5, 4\).
\(\bar x = \bar y = 3\). \(\sum(x - \bar x)(y - \bar y) = 4 + 0 + 0 + 2 + 2 = 8\); \(\sum(x - \bar x)^2 = 10\). Slope \(\hat\beta_1 = 0.8\); intercept \(\hat\beta_0 = 3 - 0.8 \times 3 = 0.6\).
Fitted \(1.4, 2.2, 3.0, 3.8, 4.6\); residuals \(-0.4, 0.8, -1.0, 1.2, -0.6\) (sum 0). RSS \(= 3.6\). \(\hat\sigma^2 = 3.6/(n - 2) = 1.2\), \(\hat\sigma = 1.095\). Two degrees of freedom were spent estimating two coefficients.
| coef | std err | t | P>|t| (t₃) | [0.025 | 0.975] | |
|---|---|---|---|---|---|---|
| x | 0.8 | \(1.095/\sqrt{10} = 0.346\) | 2.31 | 0.10 | \(0.8 - 3.182 \times 0.346 = -0.30\) | 1.90 |
| const | 0.6 | \(0.346\sqrt{55/5} = 1.15\) | 0.52 | 0.64 | −3.06 | 4.26 |
With \(z\) instead of \(t_3\), the slope's \(t = 2.31 > 1.96\) would "reject." With three residual degrees of freedom, the honest critical value is 3.18.
SE of the fitted line at \(x\): \(\hat\sigma\sqrt{1/5 + (x - 3)^2/10}\) = 0.49 at \(x = 3\), 0.85 at \(x = 5\), 1.15 at \(x = 6\). Predicting a new \(y\) at \(x = 6\): \(\hat y = 5.4\), SE \(= \sqrt{1.15^2 + 1.2} = 1.59\), 95% prediction interval \(5.4 \pm 3.18 \times 1.59 = (0.35, 10.45)\).
\(\hat\sigma^2(X^\top X)^{-1} = 1.2 \times \frac{1}{50}\begin{pmatrix}55 & -15\\-15 & 5\end{pmatrix} = \begin{pmatrix}1.32 & -0.36\\-0.36 & 0.12\end{pmatrix}\). SEs \(\sqrt{1.32} = 1.15\) and \(\sqrt{0.12} = 0.346\). The correlation between \(\hat\beta_0\) and \(\hat\beta_1\) is \(-0.36/\sqrt{1.32 \times 0.12} = -0.905\): if the slope came out too steep, the intercept probably came out too low. That's the pivot of the bow-tie.
\(\widehat{\text{se}}^2_{HC0} = \sum(x_i - \bar x)^2\hat\epsilon_i^2/\big(\sum(x_i - \bar x)^2\big)^2 = \frac{4(0.16) + 1(0.64) + 0 + 1(1.44) + 4(0.36)}{100} = 0.0416\), so 0.204; HC1 multiplies by \(\sqrt{n/(n-2)} = \sqrt{5/3}\): 0.263. Smaller than the classical 0.346 here, because the biggest residuals sit at low-leverage points near \(\bar x\).
2,930 house sales, 11 coefficients, df 2,919. Row: area 113.32 2.009 56.40 0.000 109.38 117.26.
Leverage. The slope is \(\sum(x_i - \bar x)(y_i - \bar y)/\sum(x_i - \bar x)^2\), so a point's pull is proportional to \(x_i - \bar x\), which is 0 at \(x = 3\). Moving the middle point raises the intercept and inflates \(\hat\sigma\) (so every SE grows), but leaves the slope where it was.
Both are about right on average (set strength to 0), but the classical one is slightly less noisy and its coverage is closer to 95% at small \(n\): when its assumption holds, it uses it. The robust SE costs a little efficiency in exchange for not depending on the assumption.
The confidence band covers the average \(y\) at each \(x\), whose uncertainty shrinks like \(1/\sqrt n\). The prediction band covers a single new \(y\), which adds that point's own noise \(\hat\sigma^2\). With infinite data the confidence band collapses onto the line; the prediction band stays at roughly \(\pm 1.96\sigma\).