Probability Foundations

The arc of this guide: events → random variables → expectation → tail control → limit theorems → the normal sampling family. Each layer compresses the previous one: a distribution compresses an experiment, the exponential family compresses the zoo to one schema, moments compress a distribution, and the limit theorems say that after enough averaging only two moments survive. The endpoint is the t-statistic — the first object where every piece of the chapter is load-bearing at once.

1. Probability spaces and conditioning

Three axioms generate everything: $P(\Omega) = 1$ ; $P(A) \geq 0$ ; additivity over disjoint events. Inclusion–exclusion repairs additivity for overlapping events: $P(A \cup B) = P(A) + P(B) - P(A \cap B)$ — the intersection got counted twice.

Conditioning is restriction + renormalization. $P(A \mid B) = \frac{P(A \cap B)}{P(B)}$ : shrink the sample space to $B$ , rescale so it has mass 1. Everything downstream is this one move applied repeatedly:

Multiplication law: $P(A \cap B) = P(A \mid B)P(B)$ — build joint events sequentially.
Law of total probability: for a partition $\{B_i\}$ , $P(A) = \sum_i P(A \mid B_i)P(B_i)$ — answer within each scenario, then average over scenarios, weighted by how likely each was.
Bayes is the multiplication law read in both directions. Write the joint event both ways, then solve:

$P(A \cap B_j) = P(A \mid B_j)P(B_j) = P(B_j \mid A)P(A)$

$\Rightarrow\; P(B_j \mid A) = \frac{P(A \mid B_j)P(B_j)}{P(A)} = \frac{P(A \mid B_j)P(B_j)}{\sum_i P(A \mid B_i)P(B_i)}$

with total probability expanding the denominator.

Implications — what falls out:

Bayes in slogan form: posterior ∝ likelihood × prior — the denominator is just normalization. This one proportionality is the entire Bayesian track (STATS371) in miniature.
Total probability is the universal de-conditioning move: whenever a quantity is easy given something, condition on it, then average it back out. The expectation version (tower, §5) and the variance version (decomposition, §5) are the same move at higher moments.

Independence is factorization: $P(A \cap B) = P(A)P(B)$ — conditioning on $B$ teaches you nothing about $A$ . Mutual independence (every subset factors) implies pairwise but not conversely; the standing counterexample is XOR — two fair bits and their parity are pairwise independent but not mutually.

Counting is the combinatorial backbone of the discrete zoo. Every “choose $k$ from $n$ ” sorts along two axes — does order matter? and is replacement allowed? — giving one $2\times2$ table to hold:

	Without replacement	With replacement
Ordered (sequences)	$\dfrac{n!}{(n-k)!}$ — permutations	$n^k$
Unordered (sets)	$\dbinom{n}{k}$ — combinations	$\dbinom{n+k-1}{k}$ — stars & bars

The cells are linked by de-ordering: $\binom{n}{k} = \frac{1}{k!}\cdot\frac{n!}{(n-k)!}$ — count the ordered arrangements, then divide by the $k!$ orderings of each set you don’t want to distinguish. (Stars & bars is the same de-ordering applied to $n^k$ , but with repeats the division isn’t clean — you count multisets directly: $k$ stars into $n$ bins separated by $n-1$ bars.)
Multinomial coefficient $\binom{n}{n_1\cdots n_r} = \frac{n!}{n_1!\cdots n_r!}$ generalizes combinations ( $r=2$ ): ways to deal $n$ labeled objects into $r$ labeled groups of fixed sizes $n_i$ . The binomial theorem $(a+b)^n = \sum_k\binom{n}{k}a^kb^{n-k}$ is the $r=2$ bookkeeping — $\binom{n}{k}$ counts which $k$ of the $n$ factors contribute an $a$ .
Anchor — choose 2 from $\{A,B,C\}$ : ordered/no-replace $= 3{\cdot}2 = 6$ (AB, BA, AC, CA, BC, CB); unordered/no-replace $= \binom{3}{2} = 3$ ( $\{AB\},\{AC\},\{BC\}$ ); ordered/with $= 3^2 = 9$ ; unordered/with $= \binom{4}{2} = 6$ (the 3 sets plus AA, BB, CC).
These factors are the discrete distributions (§2): the $\binom{n}{k}$ in Binomial counts which trials succeed (unordered, no replacement of positions); the Hypergeometric is unordered-without-replacement of items; sampling models each pick a cell of this table.

Core competency set

Conditioning = restrict and renormalize; derive Bayes from the joint-written-both-ways in two lines.
Partition an awkward event and apply total probability without prompting; recognize tower/variance-decomposition as the same move.
State why pairwise independence ≠ mutual independence.
Reproduce the counting $2\times2$ (ordered/unordered × with/without replacement) and the de-ordering link $\binom{n}{k} = \frac{1}{k!}\frac{n!}{(n-k)!}$ .

2. Random variables and the distribution zoo

What a distribution actually is. Build the object before the catalog:

A random variable is a map $X: \Omega \to R$ — a numerical question asked of a random outcome. The randomness lives in $\Omega$ ; $X$ just reads a number off it.
The distribution of $X$ is the probability measure $X$ induces on the real line: for any set $B$ , it answers $P(X \in B) = P(\{\omega : X(\omega) \in B\})$ . The original experiment is forgotten; only the landing probabilities survive. That compression is the whole point — statistics never sees $\Omega$ , only the induced measure.
The CDF $F(x) = P(X \leq x)$ is the universal bookkeeping device: intervals generate every set you care about, so $F$ pins down the entire measure. It always exists (discrete, continuous, or mixed), is nondecreasing and right-continuous, and runs 0 → 1. Convergence in distribution (§7) is defined through it for exactly this reason.
pmf/pdf are the local views of $F$ : the pmf is the jumps of $F$ , the pdf its slope ( $f = F'$ ). A density is not a probability — $f(x)$ can exceed 1; only integrals of it are probabilities: $f(x)\,dx \approx$ mass in a tiny interval around $x$ . (This is also why densities transform with Jacobians in §3 while probabilities never need correction — mass is conserved, height isn’t.)
Two RVs with the same distribution are statistically indistinguishable yet can be different functions of $\omega$ (for symmetric laws, $X$ and $-X$ ). Every “iid” hypothesis in the theorems to come asks only for matching distributions — function-level identity never matters.

With the object in hand, the catalog. Hold the zoo as generative stories (“what’s the law of ___?”), not formulas — and keep the two halves apart, because the type tracks the story: discrete laws count (a pmf, mass on $\{0,1,2,\dots\}$ ), continuous laws measure (a pdf spread over an interval).

Discrete — built from coin flips and arrivals:

Distribution	Story	pmf	$E$	$\operatorname{Var}$
Bernoulli( $p$ )	one coin flip	$p^x(1-p)^{1-x}$	$p$	$p(1-p)$
Binomial( $n,p$ )	# successes in $n$ flips $= \sum_n \text{Bern}$	$\binom{n}{k}p^k(1-p)^{n-k}$	$np$	$np(1-p)$
Geometric( $p$ )	# flips until the 1st success	$p(1-p)^{k-1}$	$1/p$	$\frac{1-p}{p^2}$
NegBin( $r,p$ )	# flips until the $r$ th success $= \sum_r \text{Geo}$	$\binom{k-1}{r-1}p^r(1-p)^{k-r}$	$r/p$	$\frac{r(1-p)}{p^2}$
Hypergeom( $n,r,m$ )	draw $m$ from a population of $n$ ( $r$ successes), count successes — without replacement	$\frac{\binom{r}{k}\binom{n-r}{m-k}}{\binom{n}{m}}$	$m\frac{r}{n}$	$m\frac{r}{n}\frac{n-r}{n}\cdot\frac{n-m}{n-1}$
Poisson( $\lambda$ )	count of rare events; $\text{Bin}(n, \lambda/n)$ as $n \to \infty$	$\frac{\lambda^k}{k!}e^{-\lambda}$	$\lambda$	$\lambda$

Continuous — proportions, waits, and the averaging limit:

Distribution	Support	Story	pdf	$E$	$\operatorname{Var}$
Uniform( $a,b$ )	$[a,b]$	no information beyond the range	$\frac{1}{b-a}$	$\frac{a+b}{2}$	$\frac{(b-a)^2}{12}$
Exponential( $\lambda$ )	$[0,\infty)$	continuous waiting time, memoryless	$\lambda e^{-\lambda x}$	$1/\lambda$	$1/\lambda^2$
Gamma( $\alpha,\lambda$ )	$[0,\infty)$	wait for the $\alpha$ th arrival $= \sum_\alpha \text{Exp}$	$\frac{\lambda^\alpha}{\Gamma(\alpha)}t^{\alpha-1}e^{-\lambda t}$	$\alpha/\lambda$	$\alpha/\lambda^2$
Beta( $a,b$ )	$[0,1]$	a random proportion; posterior of $p$ after $a{-}1$ successes, $b{-}1$ failures	$\frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)}u^{a-1}(1-u)^{b-1}$	$\frac{a}{a+b}$	$\frac{ab}{(a+b)^2(a+b+1)}$
Normal( $\mu,\sigma^2$ )	$R$	CLT fixed point — a sum of many small effects	$\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu)^2}{2\sigma^2}}$	$\mu$	$\sigma^2$
Cauchy	$R$	ratio of two std normals $= t_1$	$\frac{1}{\pi(1+z^2)}$	none	none

The relationship map (draw it from a blank page — the zoo is really one diagram, not twelve facts):

Coin-flip chain (count successes): $\text{Bern}(p) \xrightarrow{\ \text{sum }n\ } \text{Bin}(n,p) \xrightarrow{\ n\to\infty,\ np\to\lambda\ } \text{Pois}(\lambda)$ — the last arrow is the rare-events limit.
Waiting chain (count trials until success — the mirror image): $\text{Geo}(p) \xrightarrow{\ \text{sum }r\ } \text{NegBin}(r,p)$ in discrete time; $\text{Exp}(\lambda) \xrightarrow{\ \text{sum }\alpha\ } \text{Gamma}(\alpha,\lambda)$ in continuous time.
The Poisson-process hub — where the discrete and continuous halves are the same process. Events at rate $\lambda$ : the count in a window $t$ is $\text{Pois}(\lambda t)$ ; the wait to the next event is $\text{Exp}(\lambda)$ ; the wait to the $\alpha$ th is $\text{Gamma}(\alpha,\lambda)$ . One randomness, sliced by counting vs. timing (discrete shadow: Binomial counts ↔ Geometric/NegBin waits).
Conjugate partners (next subsection): Beta sits behind Binomial’s $p$ , Gamma behind Poisson/Exponential’s rate — the continuous laws that are the posteriors of the discrete ones.
The CLT sink: average anything with finite variance and you flow to the Normal (§7). Cauchy is the one law that escapes — no mean ⟹ no LLN, no CLT; the average of Cauchys is again Cauchy. It is the standing counterexample throughout.

Structural facts the map encodes:

Hypergeometric = Binomial without replacement. Same mean $mp$ (each draw has marginal success rate $p = r/n$ ), but the variance is binomial’s $mp(1-p)$ times the finite-population correction $\frac{n-m}{n-1} \le 1$ . Drawing without replacement makes successive draws negatively correlated — pull a success and the rest are slightly likelier to fail — so the count is less variable than the with-replacement Binomial. The FPC is $1$ for a single draw ( $m=1$ , just a Bernoulli) and $0$ once you’ve drawn the whole population ( $m=n$ , no randomness left); as $n\to\infty$ with $m$ fixed it $\to 1$ and Hypergeometric $\to$ Binomial (replacement stops mattering when the pool is huge).
Memoryless pair: Geometric (discrete) and Exponential (continuous) are the only memoryless laws — $P(X > s+t \mid X > s) = P(X > t)$ , the wait resets no matter how long you’ve already waited.
Sums stay in the family: $\sum_n\text{Bern} = \text{Bin}$ , $\sum_r\text{Geo} = \text{NegBin}$ , $\sum_\alpha\text{Exp} = \text{Gamma}$ , $\sum\text{Pois} = \text{Pois}$ , $\sum\text{Normal} = \text{Normal}$ — each a two-line computation once we have the moment generating function $E[e^{tX}]$ (§6), which turns “sum of independent” into “product of MGFs.”
Numbers to anchor the hub: at rate $\lambda = 2$ /hour, $P(\text{no events in an hour}) = e^{-2} \approx 0.135$ — the $\text{Pois}(2)$ pmf at $0$ , and $P(\text{Exp}(2)\text{ wait} > 1)$ . Same number, two slices of one process.

The exponential family — one schema behind the zoo

Most of the table is one distribution wearing different clothes. A family $\{f(x \mid \theta)\}$ is an exponential family if it can be written

$f(x \mid \theta) = h(x)\,\exp\big(\eta(\theta)\,T(x) - A(\eta)\big)$

$T(x)$ — the sufficient statistic: the only feature of the data the parameter ever interacts with.
$\eta$ — the natural parameter: the scale on which the family is exponentially tilted.
$A(\eta)$ — the log-partition function: whatever it takes to normalize. Looks like bookkeeping; turns out to run the show.
$h(x)$ — base measure, carrying the parts that don’t involve $\theta$ .

The recipe — how to cast a density into this form (the examples below are just this, run twice):

Write $f = \exp(\log f)$ — exponentiate the log-density (the named move: exponentiate-the-log).
Expand $\log f$ and sort the terms. Anything of the form (function of $\theta$ ) $\times$ (function of $x$ ) is the $\eta(\theta)\,T(x)$ piece; pure- $x$ terms collect into $\log h(x)$ ; pure- $\theta$ terms are $-A(\eta)$ .
Read off $T$ (the $x$ -factor multiplying $\theta$ ), $\eta$ (its $\theta$ -coefficient), $h$ ; then $A(\eta)$ is whatever normalizes — usually already sitting there as the leftover pure- $\theta$ term.
Re-express in $\eta$ and you’re done. Why bother: once you have $(T,\eta,A)$ , every downstream quantity — moments, the MLE, the sufficient statistic, the conjugate prior — is mechanical, read off $A$ and its derivatives rather than re-derived per distribution.

See it once and you’ll believe it everywhere — Bernoulli, rewritten by exponentiating its own log:

$p^x(1-p)^{1-x} = \exp\Big(x\log\tfrac{p}{1-p} + \log(1-p)\Big) \qquad \eta = \log\tfrac{p}{1-p},\quad T(x) = x,\quad A(\eta) = \log(1 + e^\eta)$

The natural parameter of a coin flip is the log-odds — logistic regression’s logit link isn’t a modeling choice, it’s the family’s native scale. Poisson, same move:

$\frac{\lambda^ke^{-\lambda}}{k!} = \frac{1}{k!}\exp\big(k\log\lambda - \lambda\big) \qquad \eta = \log\lambda,\quad T(k) = k,\quad A(\eta) = e^\eta$

— Poisson regression’s log link. Normal (known $\sigma^2$ ), Exponential, Gamma, Beta, Binomial, Geometric all fit the same schema. Two familiar holdouts do not: the Uniform (its support depends on the parameter, which the form can’t express) and the Cauchy (no fixed-dimension sufficient statistic exists for it — the same pathology behind its missing MGF). The usual suspects, excluded again.

Why the schema earns its place — implications:

$A$ generates the moments. Differentiate the normalization identity $\int h(x)\,e^{\eta T(x) - A(\eta)}\,dx = 1$ with respect to $\eta$ :

$\int h(x)\,e^{\eta T(x) - A(\eta)}\big(T(x) - A'(\eta)\big)\,dx = 0 \;\Rightarrow\; E[T(X)] = A'(\eta)$

Differentiate again, same move: $\operatorname{Var}[T(X)] = A''(\eta)$ . One function, differentiated, regenerates every E/Var entry in the table — check it on Poisson: $A = e^\eta$ , so $A' = A'' = e^\eta = \lambda$ , both mean and variance. And $A'' = \operatorname{Var} \geq 0$ means $A$ is convex, so log-likelihoods in $\eta$ are concave: one MLE, no local optima. (This is no coincidence: $A$ is the cumulant generating function of the family — $A'$ the first cumulant, $A''$ the second — i.e. the MGF machinery of §6 wearing exp-family clothes.)

Sufficiency. For $n$ iid observations the likelihood is $\big(\prod h(x_i)\big)\exp\big(\eta\sum_iT(x_i) - nA(\eta)\big)$ — the data enters only through $\sum_i T(x_i)$ . Any dataset, any size, compresses to a fixed-dimension summary with zero information loss about $\theta$ . (Track 3’s factorization theorem makes this precise; exp families are where it bites hardest.)
MLE = moment matching. Set the derivative of the log-likelihood to zero:

$\frac{d}{d\eta}\Big(\eta\sum_iT(x_i) - nA(\eta)\Big) = 0 \;\Rightarrow\; A'(\hat\eta) = \frac{1}{n}\sum_iT(x_i)$

Choose the parameter whose theoretical mean of $T$ equals the observed mean. Every textbook MLE you half-remember ( $\hat{p}$ = sample proportion, $\hat\lambda$ = sample mean) is this one equation instantiated.

GLMs and the role of the link. The mean is $\mu = E[T] = A'(\eta)$ , so the natural parameter and the mean are tied by $\eta = (A')^{-1}(\mu)$ . Call that map the canonical link $g$ : $g(\mu) = \eta$ . A GLM models the natural parameter as linear in covariates, $\eta = x^T\beta$ , i.e. $g(\mu) = x^T\beta$ — the link is the coordinate in which the parameter is naturally linear, handed to you by $A$ , not chosen. Read it off each family: Bernoulli $\eta = \log\frac{p}{1-p}$ → logit link (logistic regression); Poisson $\eta = \log\lambda$ → log link (Poisson regression — the log isn’t a fix for positivity, it’s the count’s native scale); Normal $\eta = \mu$ → identity link (ordinary regression). So linear/logistic/Poisson regression are one algorithm ranging over family members, each at its canonical link.
Memorization payoff: don’t memorize a dozen unrelated densities; memorize the schema plus each member’s $(T, \eta)$ , and rederive moments from $A$ .

Conjugacy — families closed under Bayesian updating

The mechanism is just posterior ∝ likelihood × prior (§1). If the prior has the same functional shape in $\theta$ as the likelihood, the product stays in the prior’s family — only the exponents update. Watch Beta–Binomial do it:

$p(\theta \mid x) \;\propto\; \underbrace{\theta^x(1-\theta)^{n-x}}_{\text{likelihood}}\;\underbrace{\theta^{a-1}(1-\theta)^{b-1}}_{\text{Beta}(a,b)\text{ prior}} \;=\; \theta^{(a+x)-1}(1-\theta)^{(b+n-x)-1}$

$\Rightarrow\; \theta \mid x \sim \text{Beta}(a + x,\; b + n - x)$

Updating is counting: add the data’s successes and failures straight onto the prior’s two exponents — infinite-dimensional posterior calculus collapses to arithmetic. The prior’s “size” has one weight worth pinning down: its concentration $a+b$ , which is the number of pseudo-observations it carries in the posterior mean. (The table’s ” $a-1$ successes, $b-1$ failures” is the same prior read as data added to a flat $\text{Beta}(1,1)$ base — and $a+b = (a-1) + (b-1) + 2$ , the base itself counting for 2; that reading is what makes the posterior mode a success frequency.)
Numbers: prior $\text{Beta}(2,3)$ — mean $0.4$ , concentration $a+b = 5$ . Observe 7 heads in 10 flips → posterior $\text{Beta}(9,6)$ , mean $\frac{9}{15} = 0.6 = \frac{5(0.4) + 10(0.7)}{15}$ : the prior’s 5 pseudo-flips are outvoted by the data’s 10, proportionally.
The standard pairs — all the same mechanism, worth recognizing on sight:

Likelihood	Conjugate prior	Posterior
Bernoulli / Binomial	Beta( $a, b$ )	Beta( $a + \text{successes},\ b + \text{failures}$ )
Poisson	Gamma( $\alpha, \beta$ )	Gamma( $\alpha + \sum k_i,\ \beta + n$ )
Exponential	Gamma( $\alpha, \beta$ )	Gamma( $\alpha + n,\ \beta + \sum x_i$ )
Normal mean (known $\sigma^2$ )	Normal	Normal — precision-weighted average of prior mean and $\bar{x}$
Multinomial	Dirichlet( $\alpha_1, \ldots, \alpha_K$ )	Dirichlet( $\alpha_k + n_k$ )

Why conjugates exist at all — the exponential family paying off. The $n$ -point likelihood is $\exp\big(\eta\sum T(x_i) - nA(\eta)\big)$ , so take a prior of the matching shape in $\eta$ :

$\pi(\eta) \propto \exp\big(\eta\,\tau - \nu A(\eta)\big) \;\Rightarrow\; \pi(\eta \mid x) \propto \exp\big(\eta(\tau + \textstyle\sum T(x_i)) - (\nu + n)A(\eta)\big)$

Same shape, updated bookkeeping: $\tau$ absorbs the sufficient statistics, $\nu$ counts the observations. Every exponential family has a conjugate prior, built exactly this way. The table above isn’t five coincidences to memorize — it’s one construction instantiated five times, and $(\tau, \nu)$ are the pseudo-counts.

This is the working core of the Bayesian track (7): conjugate updating as pseudo-count arithmetic, with exp-family structure as the reason the bookkeeping stays finite-dimensional forever.

Core competency set

State what a distribution is (the induced measure), why the CDF determines it, and why a density is not a probability.
Recite each distribution’s generative story; recover E/Var for the core ones directly — or rederive them from $A'$ , $A''$ .
Draw the zoo relationship map from a blank page (coin-flip chain, waiting chain, Poisson hub, CLT sink); tell the Poisson-process story in its three slices (count / first wait / $\alpha$ th wait) and its discrete shadow.
Cast a density into exponential form via the 4-step recipe; read off the canonical link $g(\mu) = \eta$ per family (logit / log / identity).
Cast Bernoulli and Poisson into exponential-family form from memory; name $(T, \eta, A)$ and read off the logit/log links.
Reproduce the $E[T] = A'$ derivation (differentiate the normalization identity) and the MLE-as-moment-matching equation.
Run the Beta–Binomial update and narrate pseudo-counts; construct the conjugate prior for a generic exp family.
Explain why the Cauchy breaks the limit theorems — and why it’s also outside the exponential family.

3. Transformations of random variables

For $Y = g(X)$ , two routes — the first always works, the second is the first packaged into a formula.

CDF method:

Write $F_Y(y) = P(g(X) \leq y)$ , manipulate the event into a statement about $X$ , then differentiate.
Example, $Z = X/2$ :

$F_Z(z) = P(X/2 \leq z) = P(X \leq 2z) = F_X(2z) \;\Rightarrow\; f_Z(z) = 2f_X(2z)$

The factor 2 is the Jacobian showing up on its own.

Change of variables (monotone $g$ ) — derive it once from the CDF method. For increasing $g$ :

$F_Y(y) = P(g(X) \leq y) = P(X \leq g^{-1}(y)) = F_X(g^{-1}(y))$

$f_Y(y) = \frac{d}{dy}F_X(g^{-1}(y)) = f_X(g^{-1}(y))\cdot\frac{d}{dy}g^{-1}(y)$

Decreasing $g$ flips the inequality, producing a minus sign — hence the absolute value in the general formula:

$f_Y(y) = f_X(g^{-1}(y))\left|\frac{d}{dy}g^{-1}(y)\right|$

The Jacobian factor is conservation of probability mass: $f_Y(y)\,|dy| = f_X(x)\,|dx|$ . Where $g$ stretches an interval, the density must thin proportionally.

Multivariate version — reach for this when you transform a vector of RVs and want the joint law of the new coordinates: the distribution of a sum and difference together, the polar change in Box–Muller (uniforms → two independent normals), or building $\chi^2$ / $t$ / $F$ out of normals (§8). Same idea, $|J|$ is now a volume-scaling factor. For $(U,V) = g(X,Y)$ :

Invert the transformation: $x = h_1(u,v)$ , $y = h_2(u,v)$ .
Compute the Jacobian of the inverse map:

$J = \det\begin{bmatrix} \partial x/\partial u & \partial x/\partial v \\ \partial y/\partial u & \partial y/\partial v \end{bmatrix}$

Then $f_{UV}(u,v) = f_{XY}(h_1(u,v), h_2(u,v))\,|J|$ .

Universality of the uniform — one fact, two hats. For continuous $F$ :

$P(F(X) \leq u) = P(X \leq F^{-1}(u)) = F(F^{-1}(u)) = u$

so $F(X) \sim U(0,1)$ , and running it backward, $F^{-1}(U) \sim F$ .

Implications:

Backward hat: sampling any distribution reduces to sampling uniforms — inverse-CDF sampling, the basis of simulation.
Forward hat: a p-value is $F$ -of-the-data under the null, so p-values are uniform under the null — the fact that makes rejection thresholds calibrated. Same theorem, statistics costume.

Sums — why we care, and the convolution. Sums of independent RVs are everywhere: a total, the sample sum behind every average (§7), errors accumulating. The density of $Z = X + Y$ ( $X \perp Y$ ) is the convolution of $f_X, f_Y$ — and it’s just the multivariate transform above in disguise: set $(Z, W) = (X + Y,\ X)$ , whose inverse $x = w,\ y = z - w$ has Jacobian $1$ , then marginalize out $W$ :

$f_Z(z) = \int f_X(x)\,f_Y(z - x)\,dx$

— sum over every way to split $z = x + (z - x)$ . Convolution integrals are painful, which is precisely why MGFs exist (§6): independence gives $M_{X+Y} = M_XM_Y$ , so adding independent RVs becomes multiplying MGFs — the two-line route behind every “sum within a family” fact (§2). Affine normal closure: $X \sim N(\mu, \sigma^2) \Rightarrow aX + b \sim N(a\mu + b, a^2\sigma^2)$ .

Core competency set

Execute the CDF method and re-derive the change-of-variables formula from it; explain the Jacobian as mass conservation.
Run the 3-step multivariate Jacobian procedure.
Prove universality of the uniform from memory; produce both hats (simulation, p-values).

4. Joint distributions

Joint CDF $F(x,y)$ ; marginal = integrate the other variable out fully: $f_X(x) = \int f(x,y)\,dy$ . Even when the support couples $x$ and $y$ , the marginal’s domain is free of $y$ — the dependence lives in the integration limits, which get consumed.
Conditional density: $f_{Y|X}(y \mid x) = \frac{f_{XY}(x,y)}{f_X(x)}$ — restriction + renormalization again, now slicing the joint surface at $x$ and rescaling the slice to integrate to 1.
Independence ⟺ the joint factors: $F(x_1, \ldots, x_n) = \prod F_{X_i}(x_i)$ . Equivalently densities factor and the support is a product set — a triangular support kills independence regardless of the formula.
A copula is a joint CDF with uniform marginals — by universality (§3), any joint distribution separates into marginals + copula: “what each variable does” + “how they co-move.”

Core competency set

Marginalize and condition fluently, including with coupled supports.
Spot non-independence from a non-product support without computing anything.

5. Expectation

$E(X) = \sum x_ip(x_i)$ or $\int xf(x)\,dx$ (requires $E|X| < \infty$ ). LOTUS: $E[g(X)] = \int g(x)f(x)\,dx$ — no need to derive the distribution of $g(X)$ first.

The algebra that does all the work:

Linearity, no independence required: $E(\sum a_iX_i) = \sum a_iE(X_i)$ . The single most-used fact in the subject — it’s how $E[\text{Bin}] = np$ falls out with zero computation (sum of $n$ Bernoulli means), and the engine of every indicator-variable argument.
Independence ⟹ $E(XY) = E(X)E(Y)$ . Converse false: uncorrelated ≠ independent — except jointly normal, where it’s true (§8 leans on exactly this).
The variance shortcut, derived once:

$\operatorname{Var}(X) = E(X - \mu)^2 = E(X^2) - 2\mu E(X) + \mu^2 = E(X^2) - \mu^2$

Covariance is bilinear: $\operatorname{Cov}(aX, bY) = ab\operatorname{Cov}(X,Y)$ , and $\operatorname{Var}(X) = \operatorname{Cov}(X,X)$ . Variance of a sum is the quadratic expansion of that bilinearity:

$\operatorname{Var}(X + Y) = \operatorname{Cov}(X+Y,\, X+Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X,Y)$

generalizing to $\operatorname{Var}(\sum X_i) = \sum\operatorname{Var}(X_i) + 2\sum_{i<j}\operatorname{Cov}(X_i,X_j)$ — cross-terms vanish under independence.

Correlation = covariance of the standardized variables: $\rho = \frac{\sigma_{XY}}{\sigma_X\sigma_Y} \in [-1,1]$ (Cauchy–Schwarz).

Conditional expectation is a random variable. Build it in three steps:

$E(Y \mid X = x)$ : the mean of $Y$ inside the slice $X = x$ — a number that depends on $x$ , i.e. a function of $x$ .
$E(Y \mid X)$ : plug the random $X$ into that function — now a random variable, the best guess of $Y$ given what $X$ tells you.
Tower / total expectation: $E(Y) = E[E(Y \mid X)]$ — compute the answer within each scenario, then average over scenarios. The proof is just unpacking the double integral:

$E[E(Y \mid X)] = \int E(Y \mid X = x)\,f_X(x)\,dx = \int\!\!\int y\,f_{Y|X}(y \mid x)\,f_X(x)\,dy\,dx = \int\!\!\int y\,f_{XY}(x,y)\,dy\,dx = E(Y)$

(the inner product $f_{Y|X} \cdot f_X$ reassembles the joint — conditioning run in reverse).

Variance decomposition: total spread = average spread inside each slice + spread of the slice means:

$\operatorname{Var}(Y) = \underbrace{E[\operatorname{Var}(Y \mid X)]}_{\text{within-group}} + \underbrace{\operatorname{Var}[E(Y \mid X)]}_{\text{between-group}}$

Derivation — apply tower to both pieces of the variance shortcut. First piece, using $E(Y^2 \mid X) = \operatorname{Var}(Y \mid X) + E(Y \mid X)^2$ :

$E(Y^2) = E[E(Y^2 \mid X)] = E[\operatorname{Var}(Y \mid X)] + E[E(Y \mid X)^2]$

Second piece, by tower: $(EY)^2 = (E[E(Y \mid X)])^2$ . Subtract:

$\operatorname{Var}(Y) = E[\operatorname{Var}(Y \mid X)] + \underbrace{E[E(Y \mid X)^2] - (E[E(Y \mid X)])^2}_{=\ \operatorname{Var}[E(Y \mid X)]}$

Implications — why $E(Y \mid X)$ is the central object of statistics:

$E(Y \mid X)$ is the best predictor of $Y$ under squared loss. For any competitor $g(X)$ , add and subtract $E(Y \mid X)$ :

$E[(Y - g(X))^2] = E[(Y - E(Y \mid X))^2] + E[(E(Y \mid X) - g(X))^2]$

The cross-term dies by conditioning on $X$ (inside the slice, $Y - E(Y \mid X)$ has mean zero while the other factor is a constant). Both remaining terms are $\geq 0$ and only the second depends on $g$ — minimized by $g = E(Y \mid X)$ . Regression is the project of estimating this function.

The variance decomposition is the seed of ANOVA and $R^2$ : “fraction of variance explained” = between-group share $\frac{\operatorname{Var}[E(Y|X)]}{\operatorname{Var}(Y)}$ .
Tower is the universal trick for expectations of layered randomness (random sums, mixtures, hierarchical models): condition on the layer that makes it easy, then average it out.

Core competency set

Use linearity + indicators as the default attack on any “expected count” problem.
Reproduce the tower proof (joint reassembly) and the variance decomposition derivation.
Prove that $E(Y|X)$ minimizes squared loss and say why the cross-term vanishes.
Expand the variance of a sum with covariances without error.

6. Tail bounds, MGFs, delta method

Markov’s inequality — built from one pointwise comparison. For $X \geq 0$ , $t > 0$ , it’s true case-by-case that $t\,\mathbf{1}\{X \geq t\} \leq X$ (if $X \geq t$ , the left side is $t \leq X$ ; if not, it’s $0 \leq X$ ). Expectations preserve ≤:

$t\,P(X \geq t) = E[t\,\mathbf{1}\{X \geq t\}] \leq E(X) \;\Rightarrow\; P(X \geq t) \leq \frac{E(X)}{t}$

A mean constrains how much mass can live far out.

Chebyshev — Markov applied to a transformed variable. $(X - \mu)^2$ is a nonnegative RV with mean $\sigma^2$ , so:

$P(|X - \mu| > t) = P\big((X-\mu)^2 > t^2\big) \leq \frac{E(X-\mu)^2}{t^2} = \frac{\sigma^2}{t^2}$

Distribution-free tail control from two moments.

Implications — the transform, then Markov pattern scales:

Apply Markov to $e^{tX}$ instead and optimize over $t$ :

$P(X \geq a) = P(e^{tX} \geq e^{ta}) \leq e^{-ta}M(t) \;\Rightarrow\; P(X \geq a) \leq \min_t\, e^{-ta}M(t)$

— the Chernoff bound, with exponentially decaying tails where Chebyshev gives only polynomial. Each higher transform spends more moment information to buy a sharper tail. This is the engine of concentration inequalities (Hoeffding, Bernstein) used everywhere in ML theory.

Chebyshev is already enough to prove the WLLN (§7) — the cheapest possible route from moments to convergence.

MGF $M(t) = E(e^{tX})$ — one function doing three jobs the chapter keeps needing: it encodes every moment, it fingerprints the distribution (uniqueness), and it turns sums into products. The third is the payoff: convolution (§3) is painful, but independence makes MGFs multiply, so “add independent RVs” becomes “multiply MGFs and recognize the result” — that is how every sum-within-a-family fact (§2) is a two-line proof. Defined when it exists near 0:

$M^{(r)}(0) = E(X^r)$ — $e^{tX}$ is a moment-encoding power series; differentiation reads the moments off.
Uniqueness: same MGF ⟹ same distribution.
Independence turns convolution into multiplication — the tool for sums:

$M_{X+Y}(t) = E[e^{t(X+Y)}] = E[e^{tX}e^{tY}] = E[e^{tX}]\,E[e^{tY}] = M_X(t)M_Y(t)$

Affine: $M_{a+bX}(t) = e^{at}M_X(bt)$ .
Cumulants, and the exponential-family bridge. The cumulant generating function $K(t) = \log M(t)$ has $K'(0) = E(X)$ and $K''(0) = \operatorname{Var}(X)$ — mean and variance off the first two derivatives at 0. This is exactly the exponential family’s log-partition $A$ (§2): $A(\eta)$ is the CGF of the natural family, which is the real reason $A'(\eta) = E[T]$ and $A''(\eta) = \operatorname{Var}[T]$ there. The normalizer you collect when casting a density into exponential form and the MGF you build here are one object in two costumes — that is what “the exp family gets into moment generation” was pointing at.
Caveat: MGFs can fail to exist (Cauchy); existence near 0 is what makes the uniqueness/continuity machinery run.

Delta method — pushing moments through a smooth function:

$X$ concentrated near $\mu$ ; zoomed in there, $g$ is approximately a line:

$Y = g(X) \approx g(\mu) + g'(\mu)(X - \mu)$

Variance through a line is slope-squared (constants don’t vary; coefficients square):

$\operatorname{Var}(Y) \approx \operatorname{Var}\big(g(\mu) + g'(\mu)(X - \mu)\big) = [g'(\mu)]^2\,\sigma^2$

Keep the second-order term for the mean and a bias appears:

$E(Y) \approx g(\mu) + \frac{1}{2}g''(\mu)\,\sigma^2$

— Jensen’s inequality made quantitative: convex $g$ ( $g'' > 0$ ) drags the mean up.

Implications: the delta method composes with the CLT — if $\bar{X}_n$ is asymptotically normal, so is $g(\bar{X}_n)$ , with variance $[g'(\mu)]^2\sigma^2/n$ . This is how standard errors for ratios, logs, and odds get built in practice.

Core competency set

Prove Markov from the pointwise inequality; derive Chebyshev and Chernoff from it as instances of transform-then-Markov.
Derive $M_{X+Y} = M_XM_Y$ and use it to identify the distribution of a sum in two lines.
Run the delta method including the second-order bias term, and say where the $[g']^2$ comes from.

7. Convergence and limit theorems

Three modes, strictly ordered (a.s. ⟹ in probability ⟹ in distribution):

Mode	Statement	What it controls
Almost sure	$P(X_n \to X) = 1$	each sample path eventually settles for good
In probability	$P(\lvert X_n - X\rvert > \epsilon) \to 0$ for all $\epsilon$	at any large $n$ a deviation is unlikely — but a given path may keep misbehaving rarely forever
In distribution	$F_n(x) \to F(x)$ at continuity points of $F$	only the CDFs converge; says nothing about the RVs as functions

WLLN. $X_i$ iid with mean $\mu$ , variance $\sigma^2$ : then $\bar{X}_n \xrightarrow{p} \mu$ . Proof — independence kills the cross-terms in the variance, then Chebyshev finishes:

$\operatorname{Var}(\bar{X}_n) = \frac{1}{n^2}\sum_i \operatorname{Var}(X_i) = \frac{\sigma^2}{n}$

$P(|\bar{X}_n - \mu| > \epsilon) \leq \frac{\operatorname{Var}(\bar{X}_n)}{\epsilon^2} = \frac{\sigma^2}{n\epsilon^2} \to 0$

(SLLN upgrades to almost-sure convergence under just $E|X| < \infty$ ; no two-line proof exists.)

Continuity theorem. If $M_n(t) \to M(t)$ for all $t$ near 0, then $F_n \to F$ at continuity points. Converts distributional convergence into calculus on MGFs — the engine behind the CLT.

CLT. $X_i$ iid, mean $\mu$ , variance $\sigma^2 < \infty$ :

$\frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} N(0,1)$

The one proof in this guide worth rehearsing fully:

WLOG $\mu = 0$ (center first). Let $S_n = \sum X_i$ ; target the MGF of $\frac{S_n}{\sigma\sqrt{n}}$ .
Independence factorizes it:

$M_{S_n/\sigma\sqrt{n}}(t) = \left[M\!\left(\frac{t}{\sigma\sqrt{n}}\right)\right]^n$

Taylor-expand $M$ at 0 — the crux: $M(0) = 1$ , $M'(0) = E(X) = 0$ (centering!), $M''(0) = E(X^2) = \sigma^2$ . The first surviving term is second order:

$M\!\left(\frac{t}{\sigma\sqrt{n}}\right) = 1 + 0 + \frac{\sigma^2}{2}\cdot\frac{t^2}{\sigma^2 n} + o\!\left(\frac{1}{n}\right) = 1 + \frac{t^2}{2n} + o\!\left(\frac{1}{n}\right)$

Power up using $(1 + \frac{a}{n} + o(1/n))^n \to e^a$ :

$\left[1 + \frac{t^2}{2n} + o(1/n)\right]^n \to e^{t^2/2}$

— the $N(0,1)$ MGF. 5. Finish by the continuity theorem.

Read the mechanism off the proof:

Centering deletes the first-order term; that’s what makes step 3 work.
$\sqrt{n}$ is the unique scale where the second-order term lands at $\Theta(1/n)$ , so the $n$ th power converges to something finite and non-degenerate. Scale by $n$ and everything dies; scale by 1 and it blows up.
Every moment beyond the second enters at $o(1/n)$ and is annihilated in the limit — averaging erases all of a distribution’s personality except $(\mu, \sigma^2)$ . That is why the normal is universal: it’s the fixed point of averaging.

Implications — what falls out of the CLT:

The universal error rate of statistics: estimates based on $n$ samples have noise $\sim \frac{\sigma}{\sqrt{n}}$ — quadruple the data to halve the error.
Numbers: a fair coin flipped 100 times — $\sigma = 0.5$ , so the proportion of heads has se $= \frac{0.5}{\sqrt{100}} = 0.05$ . Observing 60% heads is $z = 2$ , probability $\approx 2.3\%$ . Every A/B-test readout is this arithmetic.
Large-sample confidence intervals with no normality assumption on the data: $\bar{X} \pm z_{\alpha/2}\frac{\sigma}{\sqrt{n}}$ is approximately calibrated for any finite-variance population.
Normal approximations to Binomial ( $np$ large) and Poisson ( $\lambda$ large) — both are sums, so the CLT applies to them directly.
Most classical estimators are averages (or smooth functions of averages — delta method, §6), which is why “asymptotically normal” is the default ending of estimation theory (track 3).

Core competency set

State the three convergence modes and their ordering; give the flavor of why in-probability ≠ almost-sure.
Prove the WLLN in two displayed lines.
Reproduce the CLT MGF proof and explain why $\sqrt{n}$ and why only two moments survive.
Produce the implications: $\sqrt{n}$ -rate, large-sample CIs, normal approximations.

8. The normal sampling family

A constructed family — each member is a recipe over independent ingredients:

Chi-square: $\chi^2_n = \sum_{i=1}^n Z_i^2$ for iid $Z_i \sim N(0,1)$ . The law of “squared normal noise.” Equals $\text{Gamma}(\frac{n}{2}, \frac{1}{2})$ ; $E = n$ , $\operatorname{Var} = 2n$ .
t: $t_n = \frac{Z}{\sqrt{U/n}}$ with $Z \sim N(0,1) \perp U \sim \chi^2_n$ . A normal whose scale is itself noisy ⟹ heavier tails. $t_1$ = Cauchy (maximal scale noise); $t_n \to N(0,1)$ as the scale noise vanishes.
F: $F_{m,n} = \frac{U/m}{V/n}$ , independent chi-squares — a ratio of two variance estimates. $t_n^2 = F_{1,n}$ .

The sampling theorem. For $X_1, \ldots, X_n$ iid $N(\mu, \sigma^2)$ , with $\bar{X} = \frac{1}{n}\sum X_i$ and $s^2 = \frac{1}{n-1}\sum(X_i - \bar{X})^2$ :

$\bar{X} \sim N(\mu, \frac{\sigma^2}{n})$
$\bar{X} \perp s^2$
$\frac{(n-1)s^2}{\sigma^2} \sim \chi^2_{n-1}$
therefore $\frac{\bar{X} - \mu}{s/\sqrt{n}} \sim t_{n-1}$

Proof of (2) — the magic ingredient, special to the normal:

Compute the covariance of the mean with each deviation:

$\operatorname{Cov}(\bar{X},\, X_i - \bar{X}) = \operatorname{Cov}(\bar{X}, X_i) - \operatorname{Var}(\bar{X}) = \frac{\sigma^2}{n} - \frac{\sigma^2}{n} = 0$

$(\bar{X},\, X_1 - \bar{X}, \ldots, X_n - \bar{X})$ is jointly normal (a linear image of normals), and for jointly normal vectors uncorrelated ⟹ independent.
$s^2$ is a function of the deviations only, so $\bar{X} \perp s^2$ . For any non-normal distribution they are dependent — this independence characterizes the normal.

Proof skeleton for (3) — degrees of freedom as spent information. Add and subtract $\bar{X}$ , then expand:

$\sum_i (X_i - \mu)^2 = \sum_i \big((X_i - \bar{X}) + (\bar{X} - \mu)\big)^2 = \sum_i (X_i - \bar{X})^2 + n(\bar{X} - \mu)^2$

(the cross-term is $2(\bar{X} - \mu)\sum_i(X_i - \bar{X}) = 0$ since deviations sum to zero). Divide through by $\sigma^2$ :

$\underbrace{\sum_i \frac{(X_i - \mu)^2}{\sigma^2}}_{\chi^2_n} = \underbrace{\sum_i \frac{(X_i - \bar{X})^2}{\sigma^2}}_{=\ (n-1)s^2/\sigma^2} + \underbrace{\frac{(\bar{X} - \mu)^2}{\sigma^2/n}}_{\chi^2_1}$

The two right-hand pieces are independent (by (2)), so MGFs multiply; matching against the left side forces the first piece to be $\chi^2_{n-1}$ . One degree of freedom was spent estimating the mean — the deviations obey one linear constraint, which is also why $s^2$ divides by $n - 1$ .

Assembling (4) — the destination of the whole track. Divide and conquer the unknown $\sigma$ :

$\frac{\bar{X} - \mu}{s/\sqrt{n}} = \frac{(\bar{X} - \mu)\big/(\sigma/\sqrt{n})}{\sqrt{s^2/\sigma^2}} = \frac{Z}{\sqrt{\chi^2_{n-1}/(n-1)}} \sim t_{n-1}$

— exactly the t recipe, with the required independence supplied by (2), and $\sigma$ cancelling between numerator and denominator.

Implications — why this theorem is the payoff:

The t-statistic is a pivot: a statistic whose distribution is fully known and free of unknown parameters. Pivots are what make exact inference possible — invert the pivot’s quantiles and you get the small-sample confidence interval

$\bar{X} \pm t_{n-1,\,\alpha/2}\cdot\frac{s}{\sqrt{n}}$

valid at any $n$ , not just asymptotically.

The price of estimating $\sigma$ is visible: $t_{n-1}$ has heavier tails than $Z$ , so intervals widen at small $n$ ; as $n \to \infty$ , $s \to \sigma$ and t collapses back to the normal — the CLT regime takes over.
The same architecture (ratio of independent normal-ish numerator to chi-square-ish denominator) recurs everywhere downstream: regression t-tests for coefficients, F-tests comparing variances/models (track 4) — learn the pattern once here.

Core competency set

Build χ², t, F from their recipes; place $t_1$ = Cauchy and $t_\infty = Z$ on the spectrum.
Prove $\bar{X} \perp s^2$ via covariance + joint normality; run the add-and-subtract χ² split and narrate the lost degree of freedom.
Assemble the t-statistic, articulate the pivot property, and write the CI it produces.

9. Memorize cold

The instant-recall layer — these should cost nothing to produce. Everything else in the guide is rederivable from them plus the named moves; the fluency flashcards drill exactly this list.

Bayes, slogan and formula: posterior ∝ likelihood × prior; $P(B_j \mid A) = \frac{P(A \mid B_j)P(B_j)}{\sum_iP(A \mid B_i)P(B_i)}$ .
The zoo’s core rows with E/Var: Bern, Bin, Geo, Pois, Unif, Exp, Gamma, Normal — including the normal pdf $\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu)^2}{2\sigma^2}}$ outright.
Variance shortcut $\operatorname{Var}(X) = E(X^2) - (EX)^2$ ; sum rule $\operatorname{Var}(X+Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X,Y)$ .
Tower $E(Y) = E[E(Y \mid X)]$ ; decomposition $\operatorname{Var}(Y) = E[\operatorname{Var}(Y \mid X)] + \operatorname{Var}[E(Y \mid X)]$ .
Change of variables: $f_Y(y) = f_X(g^{-1}(y))\left|\frac{d}{dy}g^{-1}(y)\right|$ .
Markov $P(X \geq t) \leq \frac{E(X)}{t}$ ; Chebyshev $P(|X - \mu| > t) \leq \frac{\sigma^2}{t^2}$ .
Counting $2\times2$ : ordered/unordered × with/without replacement = $\frac{n!}{(n-k)!}$ , $n^k$ , $\binom{n}{k}$ , $\binom{n+k-1}{k}$ .
Exponential family form $h(x)\exp(\eta T(x) - A(\eta))$ ; $E[T] = A'(\eta)$ , $\operatorname{Var}[T] = A''(\eta)$ ( $A$ = the CGF); Bernoulli’s $\eta$ = log-odds, Poisson’s $\eta = \log\lambda$ ; canonical link $g(\mu) = \eta = (A')^{-1}(\mu)$ .
The conjugate table as pseudo-count updates: Beta–Binomial, Gamma–Poisson, Gamma–Exponential, Normal–Normal, Dirichlet–Multinomial.
CLT statement: $\frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} N(0,1)$ ; the $\frac{\sigma}{\sqrt{n}}$ error rate.
The recipes: $\chi^2_n = \sum Z_i^2$ , $t_n = \frac{Z}{\sqrt{U/n}}$ , $F_{m,n} = \frac{U/m}{V/n}$ ; the t-statistic $\frac{\bar{X} - \mu}{s/\sqrt{n}} \sim t_{n-1}$ and its CI $\bar{X} \pm t_{n-1,\alpha/2}\frac{s}{\sqrt{n}}$ .
Named moves: restrict-and-renormalize (conditioning); condition-then-average (total probability / tower); transform-then-Markov; add-and-subtract; exponentiate-the-log (exp-family rewrites).