Sequence Models, Attention & Transformers

Language forces three demands no fixed-size feedforward net meets: inputs are variable-length, order matters, and a word’s meaning is context-dependent. The subject is the arc that answers them, each stage fixing the previous one’s bottleneck. Static word vectors (word2vec, GloVe) give dense meaning but one vector per word-type. Recurrence (RNN/LSTM) threads a hidden state through time to carry context — but serially, and it forgets across long gaps. Seq2seq maps one sequence to another through a single summary vector — which is itself a bottleneck. Attention removes it by letting the decoder look directly at every source position. The Transformer then throws out recurrence entirely — attention is all you need — buying full parallelism and constant-distance interaction between any two tokens. Finally pretraining learns the representation once on raw text and finetunes it everywhere, which is what made today’s large language models. A single quantity recurs as the villain — the product of Jacobians through time that makes vanilla RNNs forget — and the LSTM’s additive cell, attention’s direct paths, and the Transformer’s residual stream are all ways to keep it from vanishing. Prerequisites cashed in: backprop, the $\delta$ rule, $dz = a-y$, vanishing/exploding gradients, residual connections, softmax, Adam, layer/batch norm (track 11a); the chain rule and Jacobians (track 2); cross-entropy and MLE (tracks 3, 6); the SVD co-occurrence view (track 2).

1. Word vectors — meaning from distribution

• The representation problem. One-hot vectors (dimension = vocabulary size) are sparse, huge, and orthogonal — “motel” and “hotel” share nothing. Distributional semantics: a word’s meaning is given by the company it keeps (the words in a window around it). Goal: a dense vector per word encoding that context — a word embedding.
• Two families. Count-based (co-occurrence matrix + SVD: rows of $U$ in $X = U\Sigma V^T$ are embeddings) is fast but dominated by frequent words and tied to a fixed vocabulary. Prediction-based (word2vec) learns embeddings by an iterative prediction task and captures linear analogies. GloVe unifies them.
• word2vec / skip-gram. Each word has two vectors: $v_c$ when it is the center word, $u_o$ when an outside/context word (two vectors only to keep the math clean — a single vector creates an awkward $x^Tx$ term). Predict context from center via softmax over similarity (dot product):

$P(o \mid c) = \frac{\exp(u_o^T v_c)}{\sum_{w\in V}\exp(u_w^T v_c)}.$

Objective: minimize average negative log-likelihood over every center–context pair in the corpus, $J(\theta) = -\frac1T\sum_t \sum_{-m\le j\le m,\,j\ne0}\log P(w_{t+j}\mid w_t)$.

• The center-vector gradient — derive it (the canonical NLP derivation). Differentiate the log-probability:

$\frac{\partial}{\partial v_c}\log P(o\mid c) = \frac{\partial}{\partial v_c}\,u_o^T v_c \;-\; \frac{\partial}{\partial v_c}\log\sum_{w}\exp(u_w^T v_c).$

First term:

$\frac{\partial}{\partial v_c}\,u_o^T v_c = u_o.$

Second term, chain rule through the log then the sum. The outer log contributes $1/(\text{sum})$ times the derivative of the sum:

$\frac{\partial}{\partial v_c}\log\sum_{w}\exp(u_w^T v_c) = \frac{1}{\sum_w \exp(u_w^T v_c)}\cdot\frac{\partial}{\partial v_c}\sum_w \exp(u_w^T v_c).$

Differentiate the inner sum term by term, using $\frac{\partial}{\partial v_c}\exp(u_x^T v_c) = \exp(u_x^T v_c)\,u_x$:

$\frac{\partial}{\partial v_c}\sum_w \exp(u_w^T v_c) = \sum_x \exp(u_x^T v_c)\,u_x.$

Substitute back and move the denominator inside the sum:

$\frac{\partial}{\partial v_c}\log\sum_{w}\exp(u_w^T v_c) = \sum_x \frac{\exp(u_x^T v_c)}{\sum_w \exp(u_w^T v_c)}\,u_x.$

The ratio is exactly the softmax $P(x\mid c)$:

$\frac{\partial}{\partial v_c}\log\sum_{w}\exp(u_w^T v_c) = \sum_x P(x\mid c)\,u_x.$

Combine:

$\frac{\partial}{\partial v_c}\log P(o\mid c) = u_o - \sum_x P(x\mid c)\,u_x = \underbrace{u_o}_{\text{observed}} - \underbrace{\mathbb E_{x\sim P(\cdot\mid c)}[u_x]}_{\text{expected}}.$

Observed minus expected — the exact form of the logistic/softmax gradient ($dz = a-y$, track 11a). The update vanishes precisely when the model’s predicted context distribution matches the data.

• Numeric anchor: vocab $\{a,b,c\}$, center vector $v_c = (1,0)$, context vectors $u_a=(2,0),\ u_b=(0,1),\ u_c=(1,0)$. Scores $u^Tv_c = (2,0,1)$ → softmax $P = (0.67, 0.09, 0.24)$. If the true context word is $a$, the gradient is $u_a - \mathbb E[u] = (2,0) - (1.58, 0.09) = (0.42,\ -0.09)$ — it pushes $v_c$ toward the observed $u_a$, which is exactly observed − expected.
• Negative sampling (SGNS) — the softmax denominator sums over the whole vocabulary, far too expensive. Replace it with binary logistic discrimination of the true pair against $k$ noise words:

$J_t = \log\sigma(u_o^T v_c) + \sum_{i=1}^{k}\mathbb E_{j\sim P(w)}\big[\log\sigma(-u_j^T v_c)\big].$

Push the true pair’s dot product up, the noise pairs’ down. Sample noise from the unigram distribution raised to the $3/4$ power (then renormalized) to upsample rarer words. Skip-gram (predict context from center) vs CBOW (predict center from context bag).

• GloVe — fuse counts and prediction. Make the dot product approximate the log co-occurrence, so that ratios of co-occurrence probabilities (which encode meaning components) come out linear:

$J = \sum_{i,j=1}^{V} f(X_{ij})\,\big(w_i^T\tilde w_j + b_i + \tilde b_j - \log X_{ij}\big)^2,$

with $f$ a weighting that down-clips very frequent pairs. Trains fast on a global matrix and matches word2vec’s analogies.

• Evaluation. Intrinsic (analogy/similarity subtasks — exclude the input words, which otherwise win) vs extrinsic (downstream task, e.g. named-entity recognition). ~300 dimensions is the usual sweet spot. A single vector per word is a superposition of its senses, separable with enough data.

Implications

• The observed-minus-expected gradient is the same engine as logistic regression and softmax classification — one move across tracks.
• Static embeddings give one vector per type, blind to context — the limitation that the later sections (§4 onward, contextual representations) exist to fix.

Core competency set

• Derive the skip-gram center-vector gradient to $u_o - \mathbb E[u_x]$ and read off observed − expected.
• State the SGNS objective and why (denominator cost); give GloVe’s log-bilinear objective and what it unifies.

2. Recurrent neural networks

• Why recurrence. A fixed window over previous words can’t handle arbitrary length and grows in parameters with the window. An RNN applies the same weights at every timestep to a running hidden state, so it handles any length with fixed parameters:

$h^{(t)} = \sigma\big(W_h\,h^{(t-1)} + W_e\,e^{(t)} + b\big), \qquad e^{(t)} = E x^{(t)},$

and emits an output (e.g. a softmax over the vocabulary) at each step. Advantages: any length, fixed model size, symmetric weight use. Disadvantages: serial (slow), and in practice it struggles to use information from many steps back. Picture (draw it): the RNN unrolled through time — the same cell (same $W_h, W_e$) copied left-to-right, each copy taking the previous hidden state $h^{(t-1)}$ and the new embedding $e^{(t)}$ and passing $h^{(t)}$ rightward; the shared box repeated is weight sharing across time (the sequence analogue of convolution’s sharing across space, track 11a).

• Language modeling & training. Predict the next word at every position; loss is cross-entropy = negative log-likelihood:

$J^{(t)} = -\sum_{w\in V} y_w^{(t)}\log\hat y_w^{(t)} = -\log\hat y_{x_{t+1}}^{(t)}.$

Teacher forcing: feed the true previous word at each step (not the model’s own prediction) so training is one-step-ahead and stable. Evaluate with perplexity = $\exp(\text{cross-entropy})$ = geometric mean of inverse next-word probabilities; lower is better.

• Backprop through time (BPTT). $W_h$ is reused at every step, so $J^{(t)}$ depends on it through every hidden state $h^{(1)},\dots,h^{(t)}$. The multivariate chain rule sums the contribution down each of those paths:

$\frac{\partial J^{(t)}}{\partial W_h} = \sum_{i=1}^{t}\frac{\partial J^{(t)}}{\partial h^{(i)}}\cdot\left.\frac{\partial h^{(i)}}{\partial W_h}\right|_{\text{step }i\text{ only}},$

where the $i$-th term treats $W_h$ as appearing only at step $i$ (the other steps’ copies held fixed). Abbreviating that summand $\left.\tfrac{\partial J^{(t)}}{\partial W_h}\right|_{(i)}$:

$\frac{\partial J^{(t)}}{\partial W_h} = \sum_{i=1}^{t}\left.\frac{\partial J^{(t)}}{\partial W_h}\right|_{(i)}.$

In practice: compute the gradient at step $t$ and propagate it back through earlier steps, accumulating; apply the summed update once. For long sequences, truncate to the last ~20 steps.

• Vanishing / exploding through time. Backprop multiplies the recurrent Jacobian $\partial h^{(t+1)}/\partial h^{(t)}$ once per step, so gradient from far back is a long product (track 11a’s villain, now along the time axis): it shrinks toward 0 — the model learns only short-range effects and can’t connect distant words — or blows up. Exploding is patched by gradient clipping (rescale $\nabla$ when $\lVert\nabla\rVert$ exceeds a threshold). Vanishing needs an architecture fix → §3.
• Generation: seed with a start token and $h^{(0)}=0$, sample from the output softmax, feed the sample back in, repeat until an end token.
• Bidirectional RNNs concatenate a left-to-right and right-to-left pass so each position’s representation sees both sides — only usable when the whole input is available (not for generation). Multi-layer (stacked) RNNs let lower layers capture low-level features and higher layers abstract structure; a few stacked layers with a modest hidden size beat one huge layer.

Implications

• The RNN solves variable length and order with weight sharing across time — the same parameter-sharing idea as convolution across space (track 11a).
• BPTT’s summed gradient and its vanishing product are why vanilla RNNs forget, motivating the LSTM.

Core competency set

• Write the RNN recurrence and the LM cross-entropy / perplexity; explain teacher forcing.
• Derive BPTT’s gradient as a sum over timesteps and explain vanishing/exploding as a Jacobian product through time; name clipping.

3. LSTMs and GRUs — a gradient highway through time

• The fix: a protected memory. The LSTM adds a cell state $c^{(t)}$ alongside the hidden state $h^{(t)}$. The cell carries long-term information and is updated additively, which is what preserves the gradient across many steps. Three gates (each an $n$-vector in $[0,1]$ from a sigmoid) control the flow: forget $f$, input $i$, output $o$, plus a tanh candidate $\tilde c$. Each gate is a sigmoid of the same inputs $(h^{(t-1)}, x^{(t)})$ with its own weights, the candidate a tanh:

$f^{(t)} = \sigma(W_f h^{(t-1)} + U_f x^{(t)} + b_f), \quad i^{(t)} = \sigma(W_i h^{(t-1)} + U_i x^{(t)} + b_i), \quad o^{(t)} = \sigma(W_o h^{(t-1)} + U_o x^{(t)} + b_o)$

$\tilde c^{(t)} = \tanh(W_c h^{(t-1)} + U_c x^{(t)} + b_c)$

$c^{(t)} = f^{(t)}\odot c^{(t-1)} + i^{(t)}\odot \tilde c^{(t)}$

$h^{(t)} = o^{(t)}\odot \tanh(c^{(t)}).$

All gates and the candidate depend only on $(h^{(t-1)}, x^{(t)})$, so they compute in parallel.

• Why it beats vanishing. Differentiate the cell recurrence with respect to $c^{(t-1)}$. The candidate term $i^{(t)}\odot\tilde c^{(t)}$ touches $c^{(t-1)}$ only indirectly (through the gates, via $h^{(t-1)}$) and contributes little; the direct term dominates:

$\frac{\partial c^{(t)}}{\partial c^{(t-1)}} \approx \mathrm{diag}\big(f^{(t)}\big).$

The cell-to-cell Jacobian is just elementwise multiplication by the forget gate — no learned matrix. Chaining it back $k$ steps multiplies $k$ forget gates, so with $f \approx 1$ the gradient passes essentially unchanged:

$\frac{\partial c^{(t)}}{\partial c^{(t-k)}} \approx \mathrm{diag}\Big(\prod_{j} f^{(t-j)}\Big) \approx I.$

Contrast the vanilla RNN, whose $\partial h^{(t)}/\partial h^{(t-1)} = \mathrm{diag}(\sigma')\,W_h$ compounds into a vanishing/exploding matrix power. It is trivial for the LSTM to preserve information (set $f \approx 1$); learning when to forget is the harder, learned part. (Initialize the forget-gate bias positive so $f$ starts near 1.) This is the same additive gradient-highway idea as the ResNet skip connection, track 11a §6.

• GRU — a lighter variant with two gates (update $z$, reset $r$) and no separate cell state; it merges the forget/input gates into one and often matches the LSTM at lower cost. Default to the LSTM when in doubt, GRU when compute-bound.

Implications

• “Carry state through an additive path” is the recurring cure for vanishing gradients — LSTM cell, GRU, ResNet, and the Transformer residual stream are the same move.
• Gates are learned, soft read/write/erase controls — an early form of the content-based selection that attention generalizes.

Core competency set

• Write the LSTM cell update and explain why the additive $c^{(t)}$ preserves gradient (forget gate ≈ 1).
• Relate it to ResNet skips; state what the GRU simplifies.

4. Seq2seq and attention

• Encoder–decoder (seq2seq). Two RNNs: an encoder reads the source and its final hidden state initializes a decoder that generates the target. It is a conditional language model — the decoder predicts the next target word conditioned on the source:

$P(y\mid x) = \prod_{t=1}^{T} P\big(y_t \mid y_1,\dots,y_{t-1},\, x\big).$

Trained end-to-end with teacher forcing on a parallel corpus; used for translation, summarization, parsing, dialogue.

• The bottleneck. The entire source must be squeezed into one fixed encoder vector — long sentences overflow it. Attention fixes this: at each decoder step, look directly back at all encoder states and pull what’s relevant.
• Attention, mechanically. With encoder states $h_1,\dots,h_N$ and decoder state $s_t$:

$e^{t} = \big[s_t^T h_1,\ \dots,\ s_t^T h_N\big] \quad(\text{scores})$

$\alpha^{t} = \mathrm{softmax}(e^{t}) \quad(\text{attention distribution})$

$a_t = \sum_{i=1}^{N}\alpha_i^{t}\,h_i \quad(\text{weighted-sum output}),$

then concatenate $[a_t; s_t]$ to predict the word. Bonus: $\alpha^t$ is a soft alignment you can inspect for free.

• Numeric anchor: scores $e = s^Th = (2, 0, 1)$ over three encoder states → softmax $\alpha = (0.67, 0.09, 0.24)$; with $h_1=(1,0),\ h_2=(0,1),\ h_3=(1,1)$ the attention output is $a = \sum_i \alpha_i h_i = (0.91, 0.33)$ — dominated by $h_1$, the position the query best matches. Picture (draw it): the alignment heatmap — target words down the rows, source words across the columns, each cell shaded by $\alpha$; for translation it lights up roughly along the diagonal, bending where word order differs — a free, readable picture of what the decoder looked at.
• The general idea — query/key/value. Attention is a way to get a fixed-size summary of a set of values, weighted by how well each matches a query (like content-addressable memory). The recipe is always score → softmax → weighted sum; only the score function varies:
  • dot product $e_i = s^T h_i$ (needs equal dimensions);
  • multiplicative $e_i = s^T W h_i$ (learnable, but $W$ is large — dimension-squared);
  • reduced-rank multiplicative $e_i = (Us)^T(Vh_i)$ (low-rank $U,V$);
  • additive $e_i = v^T\tanh(W_1 h_i + W_2 s)$ (a small MLP).
• Decoding the target. Greedy (take the argmax each step) can’t undo a bad choice. Exhaustive search is exponential. Beam search keeps the $k$ best partial hypotheses by log-probability $\sum_i \log P(y_i\mid y_{<i}, x)$ ($k\sim5$–10); since longer hypotheses accrue more negative terms, normalize the score by length. Evaluate translation with BLEU (n-gram precision overlap with reference + brevity penalty) — useful but imperfect (penalizes valid paraphrases).

Implications

• Attention turns a fixed summary into a dynamic, query-dependent lookup — the conceptual leap that the Transformer makes the whole architecture.
• The QKV framing (set of values, a query, a weighted summary) is the reusable abstraction; the score function is the only knob.

Core competency set

• Write seq2seq as a conditional LM and state the bottleneck.
• Give the three-step attention computation and the QKV interpretation; list the score-function variants; explain beam search + length normalization and BLEU.

5. The Transformer

• Why drop recurrence. RNNs take $O(n)$ serial steps for two tokens $n$ apart to interact (long-range dependencies are slow and gradient-fragile), and they can’t parallelize across time (each hidden state waits for the previous). Replace recurrence with attention applied within one sequence: any two tokens interact in one step, and all positions compute in parallel.
• Self-attention. Draw queries, keys, values from the same sequence. From input vectors $x_i$ (output of the previous layer), project with learned matrices $Q, K, V \in R^{d\times d}$:

$q_i = Q x_i,\quad k_i = K x_i,\quad v_i = V x_i.$

Score every query against every key, softmax, weight the values. In matrix form with $X\in R^{T\times d}$:

$\text{output} = \mathrm{softmax}\!\big(XQ\,(XK)^T\big)\,XV \;\in R^{T\times d}.$

• Scaled dot product — derive the $\sqrt d$. A score is a dot product of a query and key over the head dimension $d_h = d/h$, $q^Tk = \sum_{l=1}^{d_h} q_l k_l$. Take the components independent, mean 0, variance 1; then each product $q_l k_l$ has mean 0 and variance 1, and the $d_h$ of them are independent, so the variances add:

$\mathrm{Var}(q^Tk) = \sum_{l=1}^{d_h}\mathrm{Var}(q_l k_l) = d_h,$

i.e. scores have standard deviation $\sqrt{d_h} = \sqrt{d/h}$ — growing with dimension. Large logits push the softmax toward one-hot, where its Jacobian $\mathrm{diag}(p) - pp^T \to 0$ and the gradient through attention vanishes. Divide the scores by $\sqrt{d/h}$ to pull the softmax input back to unit scale:

$\text{output} = \mathrm{softmax}\!\left(\frac{XQ\,(XK)^T}{\sqrt{d/h}}\right)XV.$

• Multi-head attention. One attention pattern can attend to only one thing at a time. Use $h$ heads, each with its own $Q_\ell, K_\ell, V_\ell \in R^{d\times d/h}$, run them independently, concatenate, and mix with a learned $Y\in R^{d\times d}$:

  Each head runs its own attention:

$\text{output}_\ell = \mathrm{softmax}\!\Big(\tfrac{XQ_\ell K_\ell^T X^T}{\sqrt{d/h}}\Big)XV_\ell.$

Concatenate the $h$ head outputs and mix with the learned $Y$:

$\text{output} = Y\,[\text{output}_1;\dots;\text{output}_h].$

(Splitting $d$ across heads keeps the total compute the same.)

• Position — because attention is order-blind. Self-attention operates on a set (permute the inputs, the outputs just permute) so sequence order must be injected. Add a position vector to each input: $x_i \leftarrow \tilde x_i + p_i$. Either fixed sinusoids (extrapolate to unseen lengths but not learnable) or learned $p\in R^{d\times T}$ (flexible but can’t extrapolate past $T$).
• Nonlinearity and stabilizers. Self-attention is only weighted averaging, so follow each block with a per-position feed-forward network $m_i = W_2\,\mathrm{ReLU}(W_1\,\text{output}_i + b_1) + b_2$. Two training essentials wrap every sublayer:
  • residual connection $X^{(i)} = X^{(i-1)} + \text{Layer}(X^{(i-1)})$ — learn the difference, smooth the landscape, keep a gradient highway (track 11a §6);
  • layer normalization — normalize each token vector to unit mean/variance (over its $d$ features, not the batch), with learned gain/bias: $\text{out} = \frac{x-\mu}{\sigma+\epsilon}\gamma + \beta$. (Per-token, so it doesn’t depend on batch composition — unlike batch norm.)
• Future masking. A decoder must not peek ahead, so before the softmax set scores to future positions to $-\infty$ (a causal lower-triangular mask) — done all at once, preserving parallelism.
• The blocks.
  • Encoder block: multi-head self-attention → add & norm → feed-forward → add & norm. Stack $N$ of them.
  • Decoder block: masked multi-head self-attention → add & norm → cross-attention (queries from the decoder, keys/values from the encoder output — the seq2seq memory access of §4) → add & norm → feed-forward → add & norm.
  • Picture (draw it): the block as a vertical stack — input → [multi-head attention] → ⊕ (residual add) → norm → [feed-forward] → ⊕ → norm → output, with the residual arrows skipping around each sublayer; stack $N$ identical blocks into a tower, embeddings + positional encodings entering at the bottom. The skip arrows are the gradient highway (11a §6); the alternation is mix across tokens (attention) → process each token (FFN).
• Cost and variants. All-pairs attention is $O(T^2 d)$ — quadratic in sequence length, prohibitive for very long documents. Efficient variants (e.g. BigBird) replace dense all-pairs attention with a mix of local-window, global, and random connections to get near-linear cost.

Implications

• The Transformer trades recurrence’s serial bottleneck for parallel, constant-distance interaction — the architecture that scaling rode to LLMs.
• Residual + layer norm + scaled dot product are the training tricks that make deep attention stacks trainable; they’re 11a’s Jacobian-product fixes ported to attention.

Core competency set

• Write self-attention in matrix form and derive the $\sqrt d$ scaling; explain multi-head and positional encoding.
• Give the encoder and decoder block layouts, the role of cross-attention and causal masking, and the $O(T^2d)$ cost.

6. Shapes, normalization, and the residual stream

The single biggest source of confusion in a transformer is bookkeeping the tensor shapes. One convention dissolves most of it.

• The $(B, T, d)$ mental model. Every activation flowing through a transformer has three axes:

  • $B$ = batch — independent sequences processed together;
  • $T$ = sequence length — tokens/positions in a sequence;
  • $d$ = model dimension ($d_{\text{model}}$) — the length of each token’s feature vector.

  So the working tensor $X$ is $(B, T, d)$: $B$ sequences, each $T$ tokens, each token a $d$-vector. (A plain MLP has no time axis: $(B, d)$; a convnet uses $(B, C, H, W)$, channels-then-spatial.)

• Operations split by which axis they touch:

Operation	Computes along	Mixes tokens?	Shape map
Linear / FFN	features $d$ (last axis)	no — per token	$(B,T,d)\to(B,T,d')$
Layer norm	features $d$	no — per token	$(B,T,d)\to(B,T,d)$
Attention	positions $T$	yes	$(B,T,d)\to(B,T,d)$
Batch norm	batch $B$ (+ $T$)	no — per feature	$(B,T,d)\to(B,T,d)$

The realization that clears most confusion: a linear layer only ever touches the last axis. $\text{Linear}(d, d')$ on $(B,T,d)$ is the same weight matrix applied independently to all $B\cdot T$ token vectors — it broadcasts over $B$ and $T$. So read any linear/FFN layer’s shape as just $d \to d'$; the batch and sequence axes ride along untouched. Only attention moves information between positions.

• Batch norm vs layer norm — it’s entirely about the axis. Both compute $(x - \mu)/\sigma$ then rescale by learned $\gamma, \beta$ of shape $(d,)$ (per-feature). They differ only in the set $\mu, \sigma$ are taken over:

  • Batch norm (track 11a §4): for each feature, average across the batch (and sequence) → statistics of shape $(d,)$, one mean/var per feature. “Normalize a feature down the batch.” Couples examples together → needs running averages at test time, degrades with small batches, and is awkward when $T$ varies.
  • Layer norm: for each token, average across its own $d$ features → statistics of shape $(B, T)$, one mean/var per token vector. “Normalize a token across its features.” Each token is normalized using only itself — no batch coupling, identical at train and test, indifferent to $T$. That is why transformers use layer norm, not batch norm.

  Concretely, for one token vector $x = (x_1, \dots, x_d)$, layer norm computes $\mu = \frac1d\sum_j x_j$, $\sigma = \sqrt{\frac1d\sum_j (x_j - \mu)^2}$, output $\gamma\odot\frac{x-\mu}{\sigma+\epsilon} + \beta$ — run independently for every one of the $B\cdot T$ tokens.

• Pre-norm vs post-norm — where the LayerNorm sits relative to the residual add. Each sublayer (attention, FFN) is wrapped in a residual connection (track 11a §6); the only question is whether the norm goes before or after the add:

  • Post-norm (original 2017 Transformer): normalize the block’s output, after the residual add —

$x \leftarrow \text{LayerNorm}\big(x + \text{Sublayer}(x)\big).$

• Pre-norm (GPT-2 onward, now standard): normalize the sublayer’s input, inside the residual branch; the residual stream itself is never normalized —

$x \leftarrow x + \text{Sublayer}\big(\text{LayerNorm}(x)\big).$

Why pre-norm won — look at the Jacobian each adds to the backward path. Pre-norm’s update $x \leftarrow x + \text{Sublayer}(\text{LN}(x))$ differentiates to

$\frac{\partial x_{\text{out}}}{\partial x} = I + \frac{\partial\,\text{Sublayer}(\text{LN}(x))}{\partial x},$

whose identity term is a clean pass-through, so a gradient travels from the last block to the first essentially undiminished (the §3 / 11a gradient-highway argument). Post-norm’s update $x \leftarrow \text{LN}(x + \text{Sublayer}(x))$ instead wraps the whole sum in the norm:

$\frac{\partial x_{\text{out}}}{\partial x} = J_{\text{LN}}\Big(I + \frac{\partial\,\text{Sublayer}(x)}{\partial x}\Big),$

so every block multiplies the backward signal by a LayerNorm Jacobian $J_{\text{LN}} \ne I$; across $N$ blocks that is a product of $N$ such Jacobians — the vanishing/exploding-product problem again, which is why deep post-norm transformers are unstable and need learning-rate warmup. Pre-norm leaves an unbroken residual stream from input to output, so very deep stacks train stably. Its one cost: the residual stream’s variance grows with depth, fixed by a single final LayerNorm before the output head.

• The FFN, spelled out. The position-wise feed-forward network is a 2-layer MLP applied to each token independently. Start from input $x$ of shape $(B,T,d)$. Expand with $W_1 \in R^{d\times d_{ff}}$ (conventionally $d_{ff} = 4d$, e.g. $d=512 \Rightarrow d_{ff}=2048$):

$h = \text{ReLU}(xW_1 + b_1), \qquad \text{shape } (B, T, d_{ff}).$

Contract back with $W_2 \in R^{d_{ff}\times d}$:

$\text{out} = hW_2 + b_2, \qquad \text{shape } (B, T, d).$

“Position-wise” means the same $W_1, W_2$ hit every token, and no information crosses positions (attention already did that). Parameters $\approx 2\,d\,d_{ff}$ — about two-thirds of a transformer’s weights live here. The rhythm of a transformer block is therefore mix across tokens (attention) → process each token nonlinearly (FFN), repeated $N$ times.

• Full shape walkthrough of one block. Concrete numbers: $B=2$ sequences, $T=4$ tokens, $d=8$, $h=2$ heads (so head size $d_h = d/h = 4$), $d_{ff}=32$.

  • input $X$: $(2,4,8)$.
  • $Q,K,V = XW_{Q},XW_K,XW_V$, each $W$ is $(8,8)$ → $Q,K,V$: $(2,4,8)$.
  • split heads: $(2,4,8) \to (2,4,2,4) \to$ transpose $\to (2,2,4,4) = (B, h, T, d_h)$.
  • scores $QK^T$ over the last two axes: $(2,2,4,4) = (B,h,T,T)$; scale by $1/\sqrt{d_h} = 1/2$; causal-mask the $4\times4$ block; softmax over the last (key) axis.
  • weighted values (scores)$V$: $(2,2,4,4) = (B,h,T,d_h)$.
  • merge heads: $(2,2,4,4) \to (2,4,2,4) \to (2,4,8)$; output projection $(8,8)$ → $(2,4,8)$.
  • residual add + layer norm → $(2,4,8)$.
  • FFN: $(2,4,8) \xrightarrow{W_1\,(8,32)} (2,4,32) \xrightarrow{\text{ReLU}} \xrightarrow{W_2\,(32,8)} (2,4,8)$; residual add + norm → $(2,4,8)$.
  • output $(2,4,8)$ — the same shape as the input, which is exactly what lets you stack $N$ blocks.

  That invariant $(B,T,d)$ tensor is the residual stream: a fixed-width bus running through the whole network, which every block reads from and adds its update back to.

Implications

• Read shapes by axis role, not by memorizing: linear/FFN/layer-norm act on the last ($d$) axis per token; attention is the only thing that mixes the $T$ axis; batch norm is the only thing that mixes the $B$ axis.
• Layer norm beats batch norm in sequence models precisely because it is per-token (batch- and length-independent); pre-norm beats post-norm because it preserves the residual gradient highway.

Core competency set

• State the $(B,T,d)$ convention and classify each op by the axis it touches; explain why a linear layer reads as $d\to d'$.
• Contrast batch vs layer norm by the axis their statistics are computed over (and the shape of those statistics); say why transformers use layer norm.
• Write pre-norm vs post-norm and explain why pre-norm trains deep stacks stably.
• Spell out the FFN ($d \to d_{ff} \to d$, $d_{ff}=4d$, position-wise) and walk one block’s shapes start to finish.

7. Pretraining and large language models

• Subword tokenization (BPE). A fixed word vocabulary makes every novel word UNK. Byte-pair encoding starts from characters and greedily merges the most frequent adjacent pair into a new subword, repeating to a target vocab — common words stay whole, rare words split into pieces. Novel words decompose into known subwords (so “word” embeddings are really subword embeddings).
• The paradigm shift. Previously: pretrained word vectors + a large randomly-initialized model trained per task. Now: pretrain almost all parameters on raw text by hiding parts of the input and reconstructing them (a scaled-up word2vec idea), then finetune on the task. Why it helps: SGD finetuning stays near the good pretrained $\hat\theta$, landing in minima that generalize better with friendlier gradients. Pretraining is compute-heavy; finetuning is cheap.
• Three architectures, three objectives.
  • Decoders (GPT) — autoregressive language modeling: predict the next token from the past. Plenty of data (any text). Finetune by adding a linear head on the final hidden state and backpropagating through the whole net; better still, reframe the task as text (e.g. natural-language inference fed as Start … Delim … Extract) and keep the architecture fixed. Can’t condition on the future (that’s required to pretrain as an LM); easy to sample from.
  • Encoders (BERT) — bidirectional, so they can’t do next-token LM. Instead masked language modeling: replace ~15% of tokens and predict them, with the 80/10/10 trick (80% [MASK], 10% random token, 10% unchanged) so the model can’t shortcut by only working on [MASK] positions and must build a full representation. (BERT’s original next-sentence-prediction objective was later judged unhelpful; RoBERTa = BERT trained longer without it; SpanBERT masks contiguous spans.) Great for understanding tasks; awkward for generation.
  • Encoder–decoders (T5) — span corruption: replace random spans with placeholders and decode the removed spans. Combines bidirectional encoding with generation; strong on QA after finetuning.
• Emergent in-context learning (GPT-3). Very large LMs perform tasks without gradient updates, just from examples placed in the prompt — the in-context examples specify the task and the conditional distribution mimics performing it. Not finetuned, still not fully understood.
• Where the field went after these notes (2020–21), for orientation (verified against current ground truth, not in the source): decoder-only models came to dominate; instruction tuning + RLHF (reinforcement learning from human feedback) aligns raw LMs into assistants; scaling laws made size/data/compute the main lever; and context lengths, retrieval augmentation, and mixture-of-experts pushed the efficient-attention agenda of §5 much further.

Implications

• Pretraining decouples learning the representation (once, on cheap unlabeled text) from using it (cheap finetuning) — the economic shift behind modern NLP.
• Decoder/encoder/encoder-decoder is the same Transformer with a different masking + reconstruction objective; the objective, not the architecture, sets what the model is good at.

Core competency set

• Explain BPE and the pretrain→finetune paradigm and why it helps.
• Contrast GPT (autoregressive decoder), BERT (masked encoder + 80/10/10), T5 (span-corruption encoder-decoder) by objective and capability; state GPT-3 in-context learning.

8. Memorize cold

• Distributional semantics: meaning = context; embeddings are dense vectors replacing orthogonal one-hots.
• Skip-gram $P(o\mid c) = \dfrac{\exp(u_o^T v_c)}{\sum_w \exp(u_w^T v_c)}$; center gradient $\dfrac{\partial \log P}{\partial v_c} = u_o - \sum_x P(x\mid c)u_x$ = observed − expected.
• SGNS $J_t = \log\sigma(u_o^T v_c) + \sum_{i=1}^k \mathbb E_{j\sim P}[\log\sigma(-u_j^T v_c)]$; noise $\propto$ unigram$^{3/4}$. GloVe $\sum f(X_{ij})(w_i^T\tilde w_j + b_i + \tilde b_j - \log X_{ij})^2$.
• RNN $h^{(t)} = \sigma(W_h h^{(t-1)} + W_e e^{(t)} + b)$; LM loss CE $= -\log\hat y_{x_{t+1}}^{(t)}$; perplexity $= \exp(\text{CE})$. BPTT: $\partial J^{(t)}/\partial W_h = \sum_{i\le t}(\cdot)|_{(i)}$; clip exploding gradients.
• LSTM cell $c^{(t)} = f\odot c^{(t-1)} + i\odot\tilde c^{(t)}$, $h^{(t)} = o\odot\tanh(c^{(t)})$ — additive cell ⇒ gradient highway (≈ ResNet skip). GRU = update + reset gate, no cell.
• Attention: scores $e_i = s^Th_i$ → $\alpha = \mathrm{softmax}(e)$ → output $\sum_i \alpha_i h_i$. QKV = weighted summary of values by query-key match. Beam search keeps top-$k$ by length-normalized log-prob; BLEU = n-gram precision + brevity penalty.
• Self-attention $\mathrm{softmax}\!\big(\frac{XQ(XK)^T}{\sqrt{d/h}}\big)XV$; $\sqrt{d/h}$ stops the softmax saturating. Multi-head: $h$ heads of dim $d/h$, concat, mix with $Y$. Positional encoding added (sinusoid/learned). Causal mask = future scores $-\infty$.
• Transformer encoder block: self-attn → add&norm → FFN → add&norm. Decoder: masked self-attn → add&norm → cross-attn (Q decoder, K/V encoder) → add&norm → FFN → add&norm. Cost $O(T^2 d)$.
• Shapes: tensor is $(B,T,d)$; linear/FFN/layernorm act per-token on the last axis ($d\to d'$, broadcast over $B,T$); attention mixes $T$; batch norm mixes $B$. Block maps $(B,T,d)\to(B,T,d)$ — the residual stream.
• Batch vs layer norm (both rescale by $\gamma,\beta\in R^d$): batch norm averages per feature across the batch (stats $(d,)$, needs test-time running stats); layer norm averages per token across its $d$ features (stats $(B,T)$, batch/length-independent → used in transformers).
• Pre-norm $x\leftarrow x+\text{Sublayer}(\text{LN}(x))$ (clean residual highway, trains deep, standard) vs post-norm $x\leftarrow \text{LN}(x+\text{Sublayer}(x))$ (original, needs warmup).
• FFN: $x \xrightarrow{W_1 (d\times d_{ff})} \text{ReLU} \xrightarrow{W_2 (d_{ff}\times d)}$, position-wise, $d_{ff}=4d$, $\approx 2dd_{ff}$ params (~⅔ of the model).
• Pretraining: BPE subwords; GPT = autoregressive decoder; BERT = masked LM, 15% with 80/10/10; T5 = span corruption. GPT-3 = in-context learning, no gradient steps.

Named moves (cross-track glossary): distributional-meaning (context defines the vector); observed-minus-expected (word2vec gradient = softmax/logistic gradient, track 11a); negative-sampling (turn a big softmax into binary discrimination); share-weights-across-time (RNN, cf. convolution’s share-across-space); Jacobian-product-through-time (the vanishing-gradient villain); additive-gradient-highway (LSTM cell / GRU / ResNet skip / Transformer residual — one idea); score-softmax-weighted-sum (the attention recipe); query-key-value (content-addressable summary); attention-is-all-you-need (drop recurrence for parallel all-pairs); scale-by-$\sqrt d$ (keep the softmax unsaturated); read-shapes-by-axis ($(B,T,d)$: linear/FFN/LN per-token, attention mixes $T$, batch norm mixes $B$); normalize-per-token-not-per-batch (layer norm vs batch norm); pre-norm-residual-highway (LN inside the branch, stream unbroken); residual-plus-layernorm (train deep stacks); causal-mask (no peeking ahead); pretrain-then-finetune (learn the representation once); objective-not-architecture (GPT/BERT/T5 differ by masking + reconstruction goal); in-context-learning (task from the prompt, no gradients).