Statistical Inference

The arc: an estimator is a random variable → Fisher information prices what the data can tell you → Cramér–Rao sets the speed limit → the MLE reaches it asymptotically → sufficiency says how far data compresses → testing is the same machinery run as a decision. Two unifications carry the whole subject: the likelihood function is the interface between data and parameter (MLE, sufficiency, Bayes, LRT are all operations on it), and the MLE asymptotics theorem $\hat{\theta} \approx N(\theta_0, \frac{1}{nI(\theta_0)})$, where Fisher information, Cramér–Rao, and the CLT meet. Prerequisites cashed in from track 1: CLT, delta method, tower/variance decomposition, exp family, the t-pivot.

1. The estimation frame

• The data $X_1, \ldots, X_n$ are random variables whose joint distribution depends on an unknown parameter $\theta$. An estimator $\hat{\theta} = T(X_1, \ldots, X_n)$ is a function of the data alone — therefore itself a random variable, with a distribution called its sampling distribution. All of inference is the study of that distribution.
• Keep three thetas straight: $\theta$ the parameter (ranges over the parameter space), $\theta_0$ the one true value, $\hat{\theta}$ your random estimate.
• Standard error = the standard deviation of the sampling distribution; for a mean, $\sigma_{\hat{\theta}} = \frac{\sigma}{\sqrt{n}}$. You rarely know it (it depends on the true parameter), so you plug in: $s_{\hat{\theta}} = \frac{s}{\sqrt{n}}$.
• Unbiased: $E(\hat{\theta}) = \theta$ — right on average. Consistent: $\hat{\theta}_n \xrightarrow{p} \theta$ — right eventually. Neither implies the other; consistency is the one you can’t do without.
• Frequentist: $\theta$ is a fixed unknown number, the data are random. Bayesian (§7): $\theta$ is itself a random variable with a prior. Same likelihood, different object treated as random — most “philosophy” disputes in statistics are this one modeling choice.

Core competency set

• State why an estimator is an RV and what its sampling distribution is.
• Distinguish the three thetas; define SE vs estimated SE; define unbiased vs consistent.

2. Fisher information — pricing the data

Target idea first. Everything in this section funnels to one number, $I(\theta)$, the Fisher information: how sharply the data can distinguish the true $\theta$ from values nearby. It will turn out to be inverse variance — the best any estimator can do is $\operatorname{Var} \approx \frac{1}{nI}$ (Cramér–Rao, §3), and the MLE achieves it (§4). So $I(\theta)$ is the exchange rate between data and precision, and it is built entirely from the slope of the log-likelihood.

Setup and notation. For a single observation $x$ with density $f(x \mid \theta)$, write the one-observation log-likelihood and its derivative in the parameter:

$\lambda(x \mid \theta) = \log f(x \mid \theta), \qquad \lambda'(x \mid \theta) = \frac{\partial}{\partial \theta}\log f(x \mid \theta)$

$\lambda'$ is the score. The prime is always $\partial/\partial\theta$ (the parameter), never $\partial/\partial x$ — the data point is held fixed and we ask how its likelihood responds as $\theta$ moves.

The score has mean zero — the hinge the section hangs on. Start from the one identity every density obeys, that it integrates to 1, and differentiate it in $\theta$.

Differentiate the normalization, moving $\frac{d}{d\theta}$ inside the integral (Leibniz rule; legal exactly when the support doesn’t depend on $\theta$):

$0 = \frac{d}{d\theta}\int f(x\mid\theta)\,dx = \int \frac{\partial f}{\partial \theta}\,dx$

Multiply and divide the integrand by $f$ — the trick that manufactures a score:

$\int \frac{\partial f}{\partial \theta}\,dx = \int \frac{\partial f/\partial\theta}{f}\,f\,dx$

Recognize the ratio as the score, since $\dfrac{\partial f/\partial\theta}{f} = \dfrac{\partial}{\partial\theta}\log f = \lambda'$:

$\int \lambda'(x\mid\theta)\,f(x\mid\theta)\,dx = E_\theta[\lambda'(X\mid\theta)]$

Chaining the three displays, $E_\theta[\lambda'] = 0$. (The differentiate-the-identity move that gave $E[T] = A'$ in the exp family — track 1 §2; named move, first of several appearances this section.)

From one observation to $n$ — the bridge. For an iid sample the log-likelihood adds, so the sample score is a sum of one-observation scores:

$\ell(\theta) = \sum_{i=1}^n \lambda(X_i \mid \theta), \qquad \ell'(\theta) = \sum_{i=1}^n \lambda'(X_i\mid\theta)$

Each term has mean zero and they’re independent, so $E_\theta[\ell'] = 0$ as well, and variance adds. Hold this bridge — $\ell' = \sum\lambda'$, sample information is $n$ times per-observation information — because Cramér–Rao (§3) and the MLE (§4) both run on it.

Fisher information is the variance of the score. Define it that way for one observation, then read off two more forms:

$I(\theta) = \operatorname{Var}_\theta[\lambda'(X\mid\theta)] = E_\theta\big[\lambda'(X\mid\theta)^2\big] = -E_\theta\big[\lambda''(X\mid\theta)\big]$

Form 1 = form 2 because variance is $E[(\lambda')^2] - (E[\lambda'])^2$ and the second term vanishes (score mean zero):

$\operatorname{Var}[\lambda'] = E[(\lambda')^2] - (E[\lambda'])^2 = E[(\lambda')^2] - 0$

Form 2 = form 3 by differentiating the score-mean-zero identity $\int \lambda' f\,dx = 0$ a second time. Product rule on $\lambda' f$:

$0 = \frac{d}{d\theta}\int \lambda' f\,dx = \int \lambda'' f\,dx + \int \lambda'\,\frac{\partial f}{\partial\theta}\,dx$

Re-insert $\partial f/\partial\theta = \lambda' f$ in the second integral (same multiply-by-$f$ recognition as before):

$\int \lambda'\,\frac{\partial f}{\partial\theta}\,dx = \int (\lambda')^2 f\,dx = E[(\lambda')^2]$

So $E[\lambda''] + E[(\lambda')^2] = 0$, i.e. $E[(\lambda')^2] = -E[\lambda'']$. For the sample, each form scales by $n$: $I_n(\theta) = nI(\theta)$.

Reading the three forms. $I(\theta)$ is the expected squared slope of the log-likelihood (form 2) and equally its expected curvature at the truth (form 3). Steep slope ⟹ a small change in $\theta$ swings the likelihood a lot ⟹ the data discriminates between nearby parameter values ⟹ informative. The curvature form says the same of the peak: a sharply curved (peaked) log-likelihood pins $\theta$ down; a flat one leaves many values about equally plausible.

The picture to hold: two log-likelihood curves through the same maximum — one sharply peaked (high $I$: the data pins $\theta$ down), one nearly flat (low $I$: many $\theta$‘s explain the data about equally well). The estimator’s uncertainty is the width of that peak.

Numbers — Bernoulli($p$), $\theta = p$. Log-likelihood of one observation, then differentiate twice in $p$:

$\lambda = x\log p + (1-x)\log(1-p)$ $\lambda' = \frac{x}{p} - \frac{1-x}{1-p}$ $\lambda'' = -\frac{x}{p^2} - \frac{1-x}{(1-p)^2}$

Take $-E[\lambda'']$ with $E[x] = p$, so $E[1-x] = 1-p$:

$I(p) = -E[\lambda''] = \frac{E[x]}{p^2} + \frac{E[1-x]}{(1-p)^2} = \frac{p}{p^2} + \frac{1-p}{(1-p)^2} = \frac{1}{p} + \frac{1}{1-p} = \frac{1}{p(1-p)}$

Information explodes near $p = 0, 1$ (rare events are very informative about rarity) and bottoms out at $p = \frac{1}{2}$; the implied MLE variance $\frac{p(1-p)}{n}$ is the familiar binomial-proportion variance, recovered instantly by §4’s capstone. (In the natural parameter $\eta$ = log-odds, $I(\eta) = A''(\eta) = p(1-p)$ — same object, different coordinates; information transforms by $[\frac{d\eta}{dp}]^2$.)

Implications

• $I(\theta)$ converts “how much data do I need?” into arithmetic: equate $nI(\theta)$ across two sampling plans to compare them.
• In exp families, $\lambda(x \mid \eta) = \log h(x) + \eta T(x) - A(\eta)$, so $\lambda'' = -A''(\eta)$ and $I(\eta) = A''(\eta) = \operatorname{Var}[T]$ — the log-partition function prices the information too.
• The variance of good estimators will turn out to be $\approx \frac{1}{nI}$ — information is literally inverse variance, the claim §3 makes precise.

Core competency set

• Prove the score has mean zero (differentiate the normalization, multiply-and-divide by $f$, recognize $\lambda'$).
• State the bridge $\ell' = \sum\lambda'$, $I_n = nI$, and derive the three equivalent forms of $I(\theta)$.
• Narrate slope and curvature intuitions; run the Bernoulli computation $\lambda\to\lambda'\to\lambda''\to I(p)$.

3. Cramér–Rao — the speed limit

Target idea. There is a floor on how well any unbiased estimator can do, and Fisher information sets it. The reason is one mechanism: the data’s distribution responds to $\theta$ through exactly one channel, the score, so an estimator can track $\theta$ only insofar as it correlates with the score — and correlation is bounded by variance. That sentence is the theorem; the algebra cashes it.

Restate the §2 bridge, leaned on twice below: for an iid sample the sample score is $\ell'(\theta) = \sum_i \lambda'(X_i\mid\theta)$, it has mean zero, and its variance is the sample information

$\operatorname{Var}_\theta[\ell'(\theta)] = nI(\theta) = I_n(\theta)$

(variance adds over the iid terms). Let $f_n(x\mid\theta) = \prod_i f(x_i\mid\theta)$ be the joint density, so $\ell' = \partial_\theta\log f_n = (\partial_\theta f_n)/f_n$ — the same ratio as in §2, now for the whole sample.

Cramér–Rao inequality. For $T = T(X_1,\dots,X_n)$ an unbiased estimator of $\theta$,

$\operatorname{Var}(T) \geq \frac{1}{nI(\theta)}, \qquad\text{and more generally, if } E_\theta(T) = m(\theta),\quad \operatorname{Var}(T) \geq \frac{[m'(\theta)]^2}{nI(\theta)}$

The proof is two moves. Move 1 — differentiate the identity $E_\theta(T) = m(\theta)$ to write the slope $m'$ as a covariance with the score. The estimator $T(x)$ doesn’t depend on $\theta$; only the density does. Write the expectation as an integral and differentiate, Leibniz inside:

$m'(\theta) = \frac{d}{d\theta}\int T(x)\,f_n(x\mid\theta)\,dx = \int T(x)\,\frac{\partial f_n}{\partial\theta}\,dx$

Multiply and divide by $f_n$ and recognize the sample score $\ell' = (\partial_\theta f_n)/f_n$:

$\int T\,\frac{\partial f_n}{\partial\theta}\,dx = \int T\,\ell'\,f_n\,dx = E[T\,\ell']$

Turn that expectation into a covariance, using $E[\ell'] = 0$:

$E[T\ell'] = E[T\ell'] - E[T]\,\underbrace{E[\ell']}_{0} = \operatorname{Cov}(T,\ \ell')$

So $m'(\theta) = \operatorname{Cov}(T, \ell')$ — and for the plain unbiased case $m(\theta) = \theta$ this is just $m' = 1$.

Move 2 — Cauchy–Schwarz on that covariance. The covariance inequality $\operatorname{Cov}(A,B)^2 \le \operatorname{Var}(A)\operatorname{Var}(B)$ with $A = T$, $B = \ell'$:

$[m'(\theta)]^2 = \operatorname{Cov}(T,\ell')^2 \leq \operatorname{Var}(T)\,\operatorname{Var}(\ell')$

Substitute $\operatorname{Var}(\ell') = nI(\theta)$ (the bridge) and divide through by it:

$[m'(\theta)]^2 \leq \operatorname{Var}(T)\,nI(\theta) \quad\Rightarrow\quad \operatorname{Var}(T) \geq \frac{[m'(\theta)]^2}{nI(\theta)}$

The plain form is the case $m' = 1$.

Read the mechanism off the proof. The score is the only channel through which the data’s distribution responds to $\theta$; an estimator tracks $\theta$ exactly to the extent it correlates with the score (move 1), and that correlation is capped by the two variances (move 2). Equality — an estimator that achieves the bound, called efficient — holds iff Cauchy–Schwarz is tight, i.e. iff $T$ is a linear function of the score.

Numbers — Bernoulli($p$), $\bar X$ as an estimator of $p$. From §2, $I(p) = \frac{1}{p(1-p)}$, so the bound is $\frac{1}{nI(p)} = \frac{p(1-p)}{n}$. And $\operatorname{Var}(\bar X) = \frac{p(1-p)}{n}$ exactly — $\bar X$ sits on the floor, so the sample mean is an efficient estimator of a Bernoulli rate (consistent with the equality condition: the Bernoulli score is linear in $\bar X$).

Efficiency: $\operatorname{eff}(\hat{\theta}, \bar{\theta}) = \frac{\operatorname{Var}(\bar{\theta})}{\operatorname{Var}(\hat{\theta})}$; with variances $\propto \frac{c}{n}$, this is the ratio of sample sizes needed for equal precision. An unbiased estimator achieving the bound is efficient.

Implications

• The bound is the benchmark that makes “asymptotically efficient” meaningful — and the MLE (§4) hits it as $n \to \infty$. That’s the reason MLE is the default estimator.
• The bound governs only unbiased estimators. Counterexample on duty — shrink the sample mean, $\tilde\theta = c\bar X$ with $0 < c < 1$. Its MSE is $c^2\operatorname{Var}(\bar X) + (1-c)^2\theta^2$, and lowering $c$ below 1 cuts the variance term faster than it adds bias, so $\tilde\theta$ drops below the Cramér–Rao floor $\frac{1}{nI}$ in MSE. This is the loophole Bayesian/shrinkage estimators exploit — seed of the bias–variance tradeoff in track 6 and of ridge in track 4 §8.

Core competency set

• State both forms of Cramér–Rao; restate the bridge $\operatorname{Var}(\ell') = nI$ the proof uses.
• Run the two moves (differentiate-the-identity to a covariance; Cauchy–Schwarz) and the equality condition ($T$ linear in the score); define efficiency as a sample-size ratio.
• Show $\bar X$ is efficient for a Bernoulli rate; explain the unbiasedness loophole with the shrink-the-mean counterexample.

4. Building estimators: Method of Moments and MLE

Method of Moments — match theoretical to observed moments:

1. Express low-order moments in terms of the parameters: $\mu_k = g_k(\theta_1, \ldots, \theta_p)$ (need as many moments as parameters).
2. Invert: parameters in terms of moments.
3. Plug in sample moments $\hat{\mu}_k = \frac{1}{n}\sum X_i^k$.

• Fast, consistent, sometimes crude. Not range-respecting (can return illegal values like negative variances). MoM variance is the $\frac{1}{n}$ version: $\hat{\sigma}^2 = \hat{\mu}_2 - \hat{\mu}_1^2$.

MLE — make the observed data most probable:

$\operatorname{lik}(\theta) = \prod_{i=1}^n f(X_i \mid \theta), \qquad \ell(\theta) = \sum_{i=1}^n \log f(X_i \mid \theta), \qquad \hat{\theta} = \arg\max_\theta \ell(\theta)$

1. Write the log-likelihood; differentiate in each parameter; set to zero; solve.
2. Check it’s a max (second derivative / boundary).
3. Read the result: in exp families this is exactly moment matching, $A'(\hat{\eta}) = \bar{T}$ (track 1 §2).

• Invariance: $g(\hat{\theta})$ is the MLE of $g(\theta)$ — transformations ride for free.
• Range-respecting: never returns an illegal parameter value (unlike MoM).
• Caution: the MLE is the parameter that makes the data most probable, not the parameter that is most probable given the data — that inversion needs a prior (§7).

The capstone theorem — MLE asymptotics. Under smoothness conditions (the support doesn’t depend on $\theta$),

$\sqrt{nI(\theta_0)}\,(\hat{\theta} - \theta_0) \xrightarrow{d} N(0, 1) \qquad \text{i.e.} \qquad \hat{\theta} \approx N\Big(\theta_0,\ \frac{1}{nI(\theta_0)}\Big)$

This is where Fisher information, Cramér–Rao, and the CLT meet: the limit variance is exactly the Cramér–Rao floor, so the MLE is asymptotically efficient. The derivation is one Taylor expansion of the score, after which the §2 bridge and track 1’s CLT/LLN do all the work.

Start from the defining property — the MLE zeroes the sample score, $\ell'(\hat\theta) = 0$. Taylor-expand $\ell'$ about the truth $\theta_0$, keeping the linear term:

$0 = \ell'(\hat\theta) \approx \ell'(\theta_0) + \ell''(\theta_0)(\hat\theta - \theta_0)$

Solve for the estimation error:

$\hat\theta - \theta_0 \approx \frac{\ell'(\theta_0)}{-\ell''(\theta_0)}$

Now read numerator and denominator separately, each a sum of iid terms. Numerator — the score at the truth is a sum of iid mean-zero terms (the §2 bridge: $\ell'(\theta_0) = \sum_i\lambda'(X_i\mid\theta_0)$, each mean zero, each variance $I(\theta_0)$), so by the CLT:

$\ell'(\theta_0) = \sum_i \lambda'(X_i\mid\theta_0) \;\approx\; N\big(0,\ nI(\theta_0)\big)$

Denominator — minus the second derivative is a sum of iid terms with mean $I(\theta_0)$ (form 3), so by the LLN:

$-\ell''(\theta_0) = -\sum_i \lambda''(X_i\mid\theta_0) \;\approx\; nI(\theta_0)$

Divide: a $N(0, nI)$ numerator over the constant $nI$ scales the variance by $(nI)^{-2}$:

$\hat\theta - \theta_0 \;\approx\; \frac{N\big(0,\ nI(\theta_0)\big)}{nI(\theta_0)} = N\Big(0,\ \frac{nI(\theta_0)}{[nI(\theta_0)]^2}\Big) = N\Big(0,\ \frac{1}{nI(\theta_0)}\Big)$

Every piece is load-bearing: the mean-zero score makes the CLT apply, the form-2/form-3 identity makes the same $I$ appear top and bottom, and the limit variance is exactly the Cramér–Rao bound.

• Caveat: fails when the support depends on $\theta$ (e.g. $U[0, \theta]$) — the differentiate-under-the-integral moves break, the same pathology that excluded Uniform from the exp family.

Implications — what falls out of the capstone:

• The asymptotic CI machine of §5: $\hat{\theta} \pm z_{\alpha/2}/\sqrt{nI(\hat{\theta})}$, for any smooth model.
• Composes with the delta method (track 1 §6): $g(\hat{\theta}) \approx N\big(g(\theta_0),\ \frac{[g'(\theta_0)]^2}{nI(\theta_0)}\big)$ — standard errors for any transformed parameter, free.
• The observed curvature $-\ell''(\hat{\theta})$ estimates $nI$ — which is why an optimizer’s Hessian at the maximum doubles as the standard-error matrix in practice.
• Wilks’ theorem (§9) is this theorem squared — the same Taylor expansion drives the χ² limit of likelihood ratios.

Core competency set

• Run both procedures; state MLE’s invariance and range-respecting properties.
• Reproduce the MLE asymptotics derivation and name where CLT, LLN, mean-zero score, and the information identity each enter.
• State the support-depends-on-θ caveat.

5. Confidence intervals — three routes

• Exact (pivots): when a pivot exists, invert its known quantiles — $\frac{\bar{X} - \mu}{s/\sqrt{n}} \sim t_{n-1}$ gives the t-interval (track 1 §8); $\frac{(n-1)s^2}{\sigma^2} \sim \chi^2_{n-1}$ gives the (asymmetric!) variance interval.
• Asymptotic: plug the MLE theorem into the usual recipe —

$\hat{\theta} \pm z_{\alpha/2}\,\frac{1}{\sqrt{nI(\hat{\theta})}}$

estimate ± z × SE, with SE priced by Fisher information.

Numbers: Poisson counts, $\bar{x} = 6.4$ over $n = 25$, with $\hat{\lambda} = \bar{x}$ (moment matching). Get the information the §2 way — the one-count log-likelihood is $\log f = -\lambda + k\log\lambda - \log k!$, so the score and its derivative in $\lambda$ are

$\frac{\partial}{\partial\lambda}\log f = -1 + \frac{k}{\lambda}, \qquad \frac{\partial^2}{\partial\lambda^2}\log f = -\frac{k}{\lambda^2}$

and, using $E[k] = \lambda$,

$I(\lambda) = -E\Big[\frac{\partial^2}{\partial\lambda^2}\log f\Big] = \frac{E[k]}{\lambda^2} = \frac{\lambda}{\lambda^2} = \frac{1}{\lambda}$

So SE $= 1/\sqrt{nI(\hat\lambda)} = \sqrt{\hat\lambda/n} = \sqrt{6.4/25} \approx 0.51$ (since $nI = n/\hat\lambda$) and the 95% CI is $6.4 \pm 0.99$.

• Bootstrap — when no formula exists, simulate the sampling distribution: generate $B$ samples from the fitted model (or resample the data), compute $\theta^*_j$ each time, and use the distribution of $\theta^* - \hat{\theta}$ as a stand-in for the unknowable $\hat{\theta} - \theta_0$. The whole trick is one approximation: the estimated world relates to your estimate the way the real world relates to the truth. CI: $(\hat{\theta} - \bar{\delta},\ \hat{\theta} - \underline{\delta})$ from the quantiles of $\theta^* - \hat{\theta}$.

Core competency set

• Produce all three CI routes and say when each applies.
• State the bootstrap’s one approximation in a sentence.

6. Sufficiency — how far data compresses

• $T(X_1, \ldots, X_n)$ is sufficient for $\theta$ if the conditional distribution of the data given $T = t$ does not depend on $\theta$ — once you know $T$, the rest of the data is noise as far as $\theta$ is concerned.
• Factorization theorem (the usable test): $T$ is sufficient ⟺ the joint density factors as

$f(x_1, \ldots, x_n \mid \theta) = g\big(T(x_1, \ldots, x_n),\ \theta\big)\,h(x_1, \ldots, x_n)$

— $\theta$ touches the data only through $T$. Read it as: the likelihood, as a function of $\theta$, depends on the data only through $T$. So anything computed from the likelihood (MLE, posterior) is automatically a function of $T$.

• Exp families are sufficiency’s home turf: the $n$-point likelihood is $\exp(\eta \sum T(x_i) - nA(\eta))\prod h(x_i)$, so $\sum_i T(x_i)$ is sufficient — fixed dimension at any $n$ (track 1 §2, now made precise).

Rao–Blackwell — sufficiency as an improvement engine. Let $\hat{\theta}$ be any estimator, $T$ sufficient, and $\bar{\theta} = E(\hat{\theta} \mid T)$. Then

$E(\bar{\theta} - \theta)^2 \leq E(\hat{\theta} - \theta)^2$

The proof reduces the MSE comparison to a variance comparison, then applies the variance decomposition from track 1 §5. First, the bias is unchanged. By the tower property the conditioned estimator has the same mean:

$E[\bar\theta] = E\big[E(\hat\theta \mid T)\big] = E[\hat\theta]$

so $\bar\theta$ and $\hat\theta$ carry identical bias. Since $\text{MSE} = \text{bias}^2 + \text{variance}$, comparing their MSEs is comparing their variances. Now decompose $\operatorname{Var}(\hat\theta)$ by conditioning on $T$:

$\operatorname{Var}(\hat\theta) = E[\operatorname{Var}(\hat\theta \mid T)] + \operatorname{Var}[E(\hat\theta \mid T)]$

The second term is exactly $\operatorname{Var}(\bar\theta)$ (since $\bar\theta = E(\hat\theta\mid T)$); the first is non-negative. Drop it:

$\operatorname{Var}(\hat\theta) = \underbrace{E[\operatorname{Var}(\hat\theta \mid T)]}_{\geq 0} + \operatorname{Var}(\bar\theta) \;\geq\; \operatorname{Var}(\bar\theta)$

Same bias plus smaller variance gives $E(\bar\theta - \theta)^2 \leq E(\hat\theta - \theta)^2$. Conditioning on $T$ throws away exactly the noise that carries no information about $\theta$. (Sufficiency is what makes $E(\hat{\theta} \mid T)$ a statistic — computable without knowing $\theta$.)

Implications

• Estimation should start from sufficient statistics — any estimator that doesn’t can be improved for free.
• Sufficiency is the license behind summary statistics: in exp families, keeping only $(n, \sum T(x_i))$ loses nothing about $\theta$ — the theory behind streaming and compressed data analysis.
• Foreshadow: Rao–Blackwell + a complete sufficient statistic yields the unique minimum-variance unbiased estimator (Lehmann–Scheffé — advanced pass).

Core competency set

• Define sufficiency, state and apply the factorization theorem; identify $\sum T(x_i)$ in an exp family on sight.
• Prove Rao–Blackwell from the variance decomposition and say why sufficiency is needed for the construction to be legal.

7. Bayesian estimation

• Treat $\theta$ as random with prior $f_\Theta(\theta)$. Then everything is track 1 conditioning:

$f_{\Theta \mid X}(\theta \mid x) = \frac{f_{X \mid \Theta}(x \mid \theta)\,f_\Theta(\theta)}{\int f_{X \mid \Theta}(x \mid \theta)\,f_\Theta(\theta)\,d\theta} \;\propto\; \text{likelihood} \times \text{prior}$

The denominator doesn’t involve $\theta$ — recognize the posterior’s shape as a known distribution and skip the integral (the conjugate machinery of track 1 §2 exists exactly for this).

• Sequential updating works: today’s posterior is tomorrow’s prior; the answer matches processing all data at once.
• The conjugate formula track 1’s table left implicit — Normal–Normal (known $\sigma^2$; prior $\theta \sim N(\mu_0, \tau_0^2)$). Work in precisions $\omega = 1/\tau^2$:

$\theta \mid x_1, \ldots, x_n \;\sim\; N\left(\frac{\omega_0\,\mu_0 + n\omega\,\bar{x}}{\omega_0 + n\omega},\ \frac{1}{\omega_0 + n\omega}\right), \qquad \omega = \frac{1}{\sigma^2}$

Precisions add; the posterior mean is the precision-weighted average of prior mean and data mean. Same pseudo-count grammar as Beta–Binomial: the prior is worth $\omega_0/\omega$ observations.

• Choosing the point estimate is a loss-function decision, and track 1’s best-predictor results decide it:
  • Posterior mean minimizes posterior mean squared error.
  • Posterior median minimizes posterior mean absolute error.
  • Posterior mode (MAP) is the Bayesian shadow of the MLE.
• Interval: take posterior quantiles (credible interval), or HPD (lower a horizontal line on the density until $1-\alpha$ is enclosed — differs from quantiles only for asymmetric posteriors).
• Interpretation contrast worth keeping sharp: a credible interval is a probability statement about $\theta$ given the data; a confidence interval is a probability statement about the procedure over repeated data, with $\theta$ fixed.
• Large-sample reconciliation: the posterior is approximately $N\big(\hat{\theta}_{MLE},\ [-\ell''(\hat{\theta})]^{-1}\big)$ — the prior washes out and Bayes converges to the MLE with the same Fisher-information variance. The two frameworks disagree mostly at small $n$, which is exactly when the prior is doing work.

Core competency set

• Run a conjugate posterior update and a posterior-mean/median estimate, naming the loss each minimizes.
• Articulate credible vs confidence in one sentence each; state the large-sample posterior ≈ MLE result.

8. Hypothesis testing — the Neyman–Pearson frame

The decision frame:

	$H_0$ true	$H_0$ false
reject	Type I (prob $\alpha$, the level)	correct — prob = power $= 1 - \beta$
accept	correct	Type II (prob $\beta$)

• A test statistic $T$, its null distribution, a critical value at level $\alpha$. The p-value is the probability under $H_0$ of a result at least as extreme as observed — equivalently the smallest $\alpha$ at which you’d reject. Under $H_0$ it’s Uniform(0,1) for a continuous test statistic (discrete ones are conservative) — universality of the CDF (track 1 §3) wearing its testing hat; that’s the calibration of the whole machine.

The picture to hold: two overlapping sampling densities of $T$ — null and alternative — with a vertical critical-value line. $\alpha$ = the null’s tail beyond the line, $\beta$ = the alternative’s mass before it. Sliding the line trades one error for the other; only separating the curves (more data, bigger effect, less noise) improves both.

• Simple hypothesis = fully specifies the distribution; otherwise composite.
• Neyman–Pearson lemma: for simple vs simple, the likelihood ratio test — reject when $\frac{f(x \mid H_0)}{f(x \mid H_1)} < c$ — is the most powerful test at its level. Optimality belongs to the likelihood ratio; everything else is implementation.
• The working pattern (used constantly): write the LR, observe it’s monotone in some statistic (e.g. $\bar{X}$), so thresholding the LR ⟺ thresholding the statistic — then use the statistic’s known null distribution to set the cutoff.
• UMP tests (most powerful against every alternative simultaneously) exist for one-sided alternatives via that monotonicity; no UMP test exists against two-sided alternatives — there’s no single best direction to spend your $\alpha$ in.

The duality — tests and confidence intervals are the same object:

$\mu_0 \in \text{the } (1-\alpha)\text{ CI} \iff \text{the level-}\alpha\text{ test of } H_0: \mu = \mu_0 \text{ accepts}$

A CI is the set of nulls the data can’t reject; a test checks whether the null is in the CI. One computation, two readings.

Core competency set

• Define α, β, power, p-value; explain why p-values are uniform under the null and why that matters.
• State Neyman–Pearson and run the monotone-LR pattern; say why two-sided UMP fails.
• State the CI–test duality and use it in both directions.

9. Generalized LRT and the chi-square family

When hypotheses are composite, maximize over each:

$\Lambda = \frac{\max_{\theta \in \omega_0} \operatorname{lik}(\theta)}{\max_{\theta \in \Omega} \operatorname{lik}(\theta)} \leq 1$

• Wilks’ theorem: under $H_0$, as $n \to \infty$,

$-2\log\Lambda \xrightarrow{d} \chi^2_d, \qquad d = \dim\Omega - \dim\omega_0$

• Sketch for $d = 1$ — the capstone cashed in. Under $H_0$ the numerator of $\Lambda$ maximizes at $\theta_0$, the denominator at $\hat\theta$, so $-2\log\Lambda = 2[\ell(\hat\theta) - \ell(\theta_0)]$. Taylor-expand $\ell(\theta_0)$ about the MLE $\hat\theta$; the linear term drops because $\ell'(\hat\theta) = 0$:

$\ell(\theta_0) \approx \ell(\hat\theta) + \underbrace{\ell'(\hat\theta)}_{0}(\theta_0 - \hat\theta) + \tfrac12\ell''(\hat\theta)(\theta_0 - \hat\theta)^2$

Substitute into $-2\log\Lambda$ (the $\ell(\hat\theta)$ terms cancel, the sign flips the second-derivative term):

$-2\log\Lambda = 2\big[\ell(\hat\theta) - \ell(\theta_0)\big] \approx -\ell''(\hat\theta)\,(\hat\theta - \theta_0)^2$

Replace $-\ell''(\hat\theta)$ by $nI(\theta_0)$ (LLN, as in §4) and group the square:

$-\ell''(\hat\theta)\,(\hat\theta - \theta_0)^2 \approx \Big[\sqrt{nI(\theta_0)}\,(\hat\theta - \theta_0)\Big]^2 \xrightarrow{d} Z^2 = \chi^2_1$

the bracket being the standardized MLE $\to N(0,1)$ from the capstone. The likelihood-ratio statistic is the squared standardized MLE; each additional free parameter contributes one more squared normal — that’s the χ² recipe (track 1 §8) assembling itself.

• The degrees of freedom are a count of free parameters: $N(2,1)$ has 0, $N(\mu, \sigma^2)$ has 2. Intuition for the rule: every extra free parameter lets the alternative chase the data a little harder, inflating the ratio even under $H_0$ — the $\chi^2_d$ reference is exactly the chase-allowance, one squared-normal unit per parameter ($\chi^2_d = \sum_1^d Z_i^2$, track 1 §8).
• Pearson’s chi-square is the GLRT in disguise: a Taylor expansion of $-2\log\Lambda$ for multinomial data gives

$X^2 = \sum_{\text{cells}} \frac{(O_i - E_i)^2}{E_i} \approx -2\log\Lambda \sim \chi^2_{df}$

with $df$ = (number of cells − 1) − (parameters estimated).

Numbers — testing a 3:1 Mendelian ratio, $n = 100$: observed (84, 16), expected (75, 25):

$X^2 = \frac{(84-75)^2}{75} + \frac{(16-25)^2}{25} = 1.08 + 3.24 = 4.32 > 3.84 = \chi^2_{1,\,0.05}$

Reject at 5% — and note the small-expected-count cell carries three-quarters of the statistic.

• One statistic, three sampling designs — keep them distinct by what’s fixed:
  • Goodness of fit: one sample vs a theoretical distribution.
  • Homogeneity: column totals fixed (J independent multinomials) — are the column distributions equal?
  • Independence: only the grand total fixed (one multinomial on the I×J grid) — does $\pi_{ij} = \pi_{i\cdot}\pi_{\cdot j}$? Same $X^2$ formula, same $(I-1)(J-1)$ df, different hypothesis and design. Fisher’s exact test handles the small-sample 2×2 case (hypergeometric null, all margins fixed).
• Specific beats generic: testing Poisson rate-constancy against the specific alternative of varying rates (dispersion test, $-2\log\Lambda \approx \frac{1}{\bar{x}}\sum(x_i - \bar{x})^2$) has more power than an omnibus goodness-of-fit test. If you know what you’re looking for, test for it.

Core competency set

• Write the GLRT, state Wilks with the df-counting rule, and narrate the chase-allowance intuition.
• Connect Pearson’s $X^2$ to the GLRT; keep homogeneity vs independence straight by what’s fixed.

10. Two samples, pairs, and ANOVA

Two independent samples (the workhorse): $\bar{X} - \bar{Y} \sim N\big(\mu_X - \mu_Y,\ \sigma^2(\frac{1}{n} + \frac{1}{m})\big)$, pool the variance estimate

$s_p^2 = \frac{(n-1)s_X^2 + (m-1)s_Y^2}{n + m - 2}, \qquad t = \frac{(\bar{X} - \bar{Y}) - (\mu_X - \mu_Y)}{s_p\sqrt{\frac{1}{n} + \frac{1}{m}}} \sim t_{n+m-2}$

— the same pivot architecture as track 1 §8: normal numerator over independent scaled-χ² denominator. (Unequal variances: replace the pooled SE with $\sqrt{\frac{s_X^2}{n} + \frac{s_Y^2}{m}}$ and use Welch’s adjusted df — pooling assumes a shared $\sigma^2$.) Power rises with effect size $\Delta$, with $\alpha$, with $n$, and falls with $\sigma$.

Paired samples: work with $D_i = X_i - Y_i$,

$\operatorname{Var}(D_i) = \sigma_X^2 + \sigma_Y^2 - 2\rho\sigma_X\sigma_Y$

Pairing pays exactly when $\rho > 0$ — matching subjects subtracts the shared noise. Then it’s a one-sample t-test on the $D_i$.

ANOVA (one-way) — many groups at once. Model $Y_{ij} = \mu + \alpha_i + \varepsilon_{ij}$, errors iid $N(0, \sigma^2)$, $\sum\alpha_i = 0$. The add-and-subtract move (track 1’s named move, third appearance) splits the variation. Add and subtract the group mean $\bar Y_{i.}$ inside each total deviation:

$Y_{ij} - \bar Y_{..} = (Y_{ij} - \bar Y_{i.}) + (\bar Y_{i.} - \bar Y_{..})$

Square and sum over all $i,j$. The cross-term vanishes because within each group the deviations from the group mean sum to zero ($\sum_j(Y_{ij} - \bar Y_{i.}) = 0$):

$2\sum_i (\bar Y_{i.} - \bar Y_{..})\sum_j (Y_{ij} - \bar Y_{i.}) = 0$

leaving (the between term carries a factor $J$ because $\bar Y_{i.} - \bar Y_{..}$ is constant across the $J$ members of group $i$):

$\sum_{i,j}(Y_{ij} - \bar{Y}_{..})^2 = \underbrace{\sum_{i,j}(Y_{ij} - \bar{Y}_{i.})^2}_{SS_W \text{ (within)}} + \underbrace{J\sum_i(\bar{Y}_{i.} - \bar{Y}_{..})^2}_{SS_B \text{ (between)}}$

— the sample version of $\operatorname{Var}(Y) = E[\operatorname{Var}(Y|X)] + \operatorname{Var}[E(Y|X)]$.

• Expectations tell you what to compare: $E\big[\frac{SS_W}{I(J-1)}\big] = \sigma^2$ always; $E\big[\frac{SS_B}{I-1}\big] = \sigma^2 + \frac{J}{I-1}\sum\alpha_i^2$ — equal iff all $\alpha_i = 0$. Hence

$F = \frac{SS_B/(I-1)}{SS_W/[I(J-1)]} \sim F_{I-1,\ I(J-1)} \text{ under } H_0$

≈ 1 under the null, inflated otherwise — the F recipe from track 1 §8 (ratio of independent χ²’s, independence by the same Cochran-style splitting).

• Multiple comparisons: a significant F says “something differs,” not what. Tukey (studentized range, simultaneous CIs for all pairs) or Bonferroni (test each of $k$ comparisons at $\alpha/k$ — crude, general). The problem this solves — $\alpha$ inflation over many tests — only grows downstream.
• Two-way layout adds $\beta_j$ and interaction $\delta_{ij}$; the same decomposition produces $SS_A + SS_B + SS_{AB} + SS_E$, each scaled by $\sigma^2$ a χ² under its null, each tested by an F against $SS_E$.

Core competency set

• Build the two-sample t (pooled SE) and the paired t; say exactly when pairing helps and why.
• Derive the ANOVA decomposition via add-and-subtract; explain the F statistic through the expected mean squares.
• Name the multiple-comparisons problem and the two standard fixes.

11. Nonparametric tests — rank everything

One idea: replace values by ranks; under the null every rank assignment is equally likely, so the null distribution is known combinatorially, with no distributional assumption — and outliers lose their leverage.

• Mann–Whitney / rank-sum (two independent samples, $H_0: F = G$): pool, rank, sum the ranks of one sample. Under $H_0$: $E(T_Y) = \frac{m(m+n+1)}{2}$, $\operatorname{Var}(T_Y) = \frac{mn(m+n+1)}{12}$, normal approximation for $m, n \gtrsim 10$.
• Sign test (paired): count positive differences; under $H_0$, $M \sim \text{Bin}(n, \frac{1}{2})$. Crude but assumption-free.
• Wilcoxon signed-rank (paired, sharper): rank $|D_i|$, restore signs, sum the positive ranks; $E(W_+) = \frac{n(n+1)}{4}$, $\operatorname{Var}(W_+) = \frac{n(n+1)(2n+1)}{24}$.
• Kruskal–Wallis = Mann–Whitney generalized to $I$ groups (the rank version of ANOVA): $K \sim \chi^2_{I-1}$ approximately.
• The surprise worth remembering: even when normality holds, rank tests lose very little power vs the t-test — and they’re far more robust when it doesn’t. For small samples they’re often simply preferable.

Core competency set

• State the one idea (equally likely rank assignments) and match each test to its design.
• Explain why rank tests are robust and what the power cost is.

12. Memorize cold

The instant-recall layer — these should cost nothing to produce. Everything else in the guide is rederivable from them plus the named moves; the fluency flashcards drill exactly this list.

• Score mean zero: $E_\theta[\lambda'] = 0$. Fisher info three forms: $I(\theta) = E[(\lambda')^2] = \operatorname{Var}[\lambda'] = -E[\lambda'']$; $I_n = nI$.
• Cramér–Rao: $\operatorname{Var}(T) \geq \frac{1}{nI(\theta)}$ (unbiased); equality ⟺ $T$ linear in the score.
• Worked information: $I(p) = \frac{1}{p(1-p)}$ (Bernoulli), $I(\lambda) = \frac{1}{\lambda}$ (Poisson).
• MLE asymptotics: $\hat{\theta} \approx N\big(\theta_0, \frac{1}{nI(\theta_0)}\big)$; asymptotic CI $\hat{\theta} \pm z_{\alpha/2}/\sqrt{nI(\hat{\theta})}$.
• MLE invariance: $g(\hat{\theta})$ is the MLE of $g(\theta)$.
• Factorization: $f(x \mid \theta) = g(T(x), \theta)\,h(x)$ ⟺ $T$ sufficient.
• Rao–Blackwell: $\bar{\theta} = E(\hat{\theta} \mid T)$ never worse (via variance decomposition).
• Posterior ∝ likelihood × prior; posterior mean ↔ squared loss, median ↔ absolute loss; large-sample posterior $\approx N(\hat{\theta}_{MLE}, [-\ell'']^{-1})$.
• Normal–Normal: precisions add; posterior mean = precision-weighted average of $\mu_0$ and $\bar{x}$.
• α / β / power / p-value definitions; p-value uniform under $H_0$; CI–test duality.
• Neyman–Pearson: LRT most powerful for simple vs simple.
• Wilks: $-2\log\Lambda \to \chi^2_d$, $d$ = free params($\Omega$) − free params($\omega_0$); Pearson $X^2 = \sum\frac{(O-E)^2}{E}$.
• Two-sample t with $s_p$; paired $\operatorname{Var}(D) = \sigma_X^2 + \sigma_Y^2 - 2\rho\sigma_X\sigma_Y$.
• ANOVA: $SS_{TOT} = SS_W + SS_B$; $F = \frac{SS_B/(I-1)}{SS_W/[I(J-1)]}$; Bonferroni $\alpha/k$.
• Rank-test null moments: $E(T_Y) = \frac{m(m+n+1)}{2}$; sign test $M \sim \text{Bin}(n, \frac{1}{2})$.
• Named moves: differentiate-the-identity (score mean zero, $E[T] = A'$); Taylor-the-score (MLE asymptotics); condition-on-sufficient (Rao–Blackwell); monotone-LR-then-threshold; add-and-subtract (ANOVA); invert-the-test (CI duality); estimated-world-as-real-world (bootstrap).