Deep Learning: Training Dynamics & Vision

A deep network is nothing but a composition of affine maps and elementwise nonlinearities; training it is plain gradient descent. Two facts make that practical: backprop computes every gradient in one backward sweep (reverse-mode autodiff on the computation graph), and a stack of techniques — initialization, normalization, adaptive optimizers, regularization — keeps that gradient well-scaled so the net actually converges and generalizes. A single thread runs through the whole subject: in a deep net the gradient to an early layer is a product of Jacobians, and almost every trick here (ReLU, He init, batch norm, residual connections) exists to keep that product near 1. The second half specializes the architecture to images — convolution bakes in locality and translation equivariance — and closes on a unifying capstone: backprop differentiates w.r.t. anything in the graph, including the input, which gives adversarial examples, saliency, style transfer, and GANs. Prerequisites cashed in: logistic regression and cross-entropy (tracks 4, 6); the chain rule, Jacobians, $A^TA$, norms (track 2); SGD and the convex/non-convex split (track 10); bias–variance (track 6); MLE (track 3).

1. From a neuron to a deep network

• A neuron is logistic regression. Pre-activation $z = w^Tx + b$, activation $a = g(z)$. With $g = \sigma$ this is exactly the logistic unit of track 4. Stack neurons side by side into a layer (vectorize: $z^{[1]} = W^{[1]}x + b^{[1]}$, rows of $W^{[1]}$ are the individual neurons’ weights), and stack layers in sequence. The whole network is a composition

$\hat y = f^{[L]}\!\circ \cdots \circ f^{[1]}(x), \qquad f^{[l]}(a) = g^{[l]}\!\big(W^{[l]}a + b^{[l]}\big).$

Layer indices are bracketed $[l]$; training examples parenthesized $(i)$; stacking examples as columns of $X$ vectorizes everything ($Z^{[1]} = W^{[1]}X + b^{[1]}$, broadcasting $b$).

• Why a nonlinearity is non-negotiable. Compose two affine maps and you get an affine map: $W_2(W_1x + b_1) + b_2 = (W_2W_1)x + (\cdots)$. Without $g$, an $L$-layer net collapses to a single linear layer — depth buys nothing. The nonlinearity is the entire source of expressive power.
• Why depth — hierarchical representation. Each layer’s output is a richer encoding of the input than its own input: layer 1 fires on edges, layer 2 on parts (eyes, corners), layer 3 on objects. Depth lets features be reused compositionally, which is exponentially more parameter-efficient than width for the functions that occur in practice. “Deep learning” is representation learning: the net discovers the intermediate features instead of being handed them.
• Activations, judged by their derivative (because $g'(z)$ multiplies into every backprop step, §2 — a saturating $g$ throttles learning):
  • sigmoid $\sigma(z) = \tfrac{1}{1+e^{-z}}$, derivative $\sigma' = \sigma(1-\sigma) \le \tfrac14$. Saturates to $\approx 0$ gradient in both tails, and its output is not zero-centered (biases the next layer’s gradients). Use only at a binary output, never in hidden layers.
  • tanh $\tanh(z)$, derivative $1 - \tanh^2$. Zero-centered (strictly better hidden unit than sigmoid) but still saturates in the tails.
  • ReLU $\max(0, z)$, derivative $\mathbb 1[z>0]$. No saturation for $z>0$ — the gradient passes through undamped — so learning is fast; this single change (sigmoid → ReLU) is the biggest reason deep nets became trainable. Risk: a unit stuck at $z<0$ is “dead” (zero gradient forever); Leaky ReLU keeps a small negative slope to avoid it.
• Output layer matches the task: sigmoid + binary cross-entropy (two classes), softmax + cross-entropy (multiclass), linear + squared error (regression).

Implications

• A deep net is a parameterized function class; everything downstream is fitting it (§2–5) or structuring it (§6).
• The activation choice is a gradient-flow decision, not a modeling nicety — it reappears as the vanishing-gradient story in §2 and motivates ReLU, batch norm, and residual connections.

Core competency set

• Write a layer’s forward map and the full-network composition; explain why removing $g$ collapses depth.
• State $\sigma', \tanh', \text{ReLU}'$ and rank the three by gradient flow; say why ReLU fixed trainability.

2. Backpropagation = reverse-mode autodiff

• The training loop. Forward pass: evaluate $\hat y$ and the scalar cost $J$. Backward pass: compute $\nabla_\theta J$. Update: $\theta \leftarrow \theta - \alpha\nabla_\theta J$. Backprop is just the efficient way to do the middle step.
• The computation graph. Write $J$ as a DAG of elementary operations. The forward pass evaluates it left-to-right; the backward pass applies the chain rule right-to-left, caching and reusing every intermediate. Key efficiency fact: one backward pass costs the same as one forward pass, versus $O(\#\text{params})$ separate evaluations for finite differences — this is why training billion-parameter nets is feasible at all.
• One layer, with the $\delta$ trick. Setup: $z = W a_{\text{prev}} + b$, $a = g(z)$, scalar loss $J$. Define the local error signal $\delta = \partial J/\partial z$ — the gradient sitting just below the nonlinearity. The upstream gradient $\partial J/\partial a$ arrives from the layer above. Push it through the activation:

$\delta = \frac{\partial J}{\partial a}\odot g'(z)$

($\odot$ = elementwise, since $a_i = g(z_i)$ acts coordinatewise). Then to the parameters:

$\frac{\partial J}{\partial W} = \delta\, a_{\text{prev}}^T$

$\frac{\partial J}{\partial b} = \delta$

and the signal handed down to the layer below:

$\frac{\partial J}{\partial a_{\text{prev}}} = W^T\delta.$

That last line is the next layer’s upstream gradient — backprop is this block repeated down the stack.

• The shape convention. A textbook Jacobian of a scalar w.r.t. a matrix $W$ would be a row vector of length $n\cdot m$ — useless for the update $W \leftarrow W - \alpha\,\partial J/\partial W$. NN convention: the gradient takes the shape of its parameter, so $\partial J/\partial W$ is $n\times m$ and the update is elementwise. The outer product $\delta a_{\text{prev}}^T$ is exactly what makes the dimensions line up (error signal $\times$ input signal).
• Reverse-mode in general. Initialize the output gradient to 1, visit nodes in reverse topological order; each node receives the upstream $\partial J/\partial(\text{out})$, multiplies by its local Jacobian $\partial(\text{out})/\partial(\text{in})$, and passes the result down. Gate behavior: a + node distributes the upstream gradient unchanged, a max node routes it to the arg-max input only, a × node switches the inputs; fan-out sums (a value used in two places sums the two upstream gradients).
• Worked: logistic / softmax gradient. With cross-entropy on top of sigmoid (or softmax), the messy $\partial J/\partial a$ and $g'$ collapse:

$dz = a - y,$

i.e. the error signal is just predicted minus observed. Then $\partial L/\partial w_j = x_j\,dz$. (This observed − expected form recurs everywhere — softmax, and the word2vec gradient of 11b.)

• Numeric anchor (one logistic neuron): $x=(2,-1)$, $w=(0.5,1)$, $b=0$, label $y=1$. Forward: $z = 0.5{\cdot}2 + 1{\cdot}(-1) = 0$, $a = \sigma(0) = 0.5$. Backward: $dz = a-y = -0.5$, $\nabla_w = x\,dz = (-1,\ 0.5)$. One step ($\alpha = 0.1$): $w \to (0.6,\ 0.95)$ — nudging $\hat y$ up toward $y=1$, as it should.
• Picture (draw it): the computation graph — a DAG with inputs and weights as source nodes and operations as interior nodes; the forward pass flows values left→right to the scalar loss, the backward pass flows gradients right→left, each node multiplying the upstream gradient by its local derivative. A + node copies the gradient to both inputs, a max routes it to the winner only, a × swaps the two inputs.
• Vanishing / exploding gradients. Unrolling backprop, the gradient reaching an early layer is a product of Jacobians

$\frac{\partial J}{\partial a^{[1]}} \;\sim\; \prod_{l} W^{[l]T}\,\mathrm{diag}\!\big(g'(z^{[l]})\big).$

Repeated multiplication is unstable: factors with magnitude $<1$ drive the product (and the early-layer gradient) to $0$ — those layers stop learning, and signal from far back is lost; factors $>1$ blow it up to overflow. Exploding is patched bluntly by gradient clipping (rescale $\nabla$ if $\lVert\nabla\rVert$ exceeds a threshold); vanishing is the harder one, and §1’s ReLU plus §4’s init/batch-norm and §6’s residual connections are all aimed at keeping this product near 1.

Implications

• Backprop is mechanical once the forward graph is written — modern frameworks autodiff it; the skill is reading off $\delta = \frac{\partial J}{\partial a}\odot g'$, $\partial J/\partial W = \delta a_{\text{prev}}^T$, $\partial J/\partial a_{\text{prev}} = W^T\delta$.
• The Jacobian-product view is the single lens unifying the rest of the guide’s training tricks.

Core competency set

• Derive the four backprop equations for one layer with the $\delta$ trick; state the shape convention and why $\partial J/\partial W = \delta a_{\text{prev}}^T$.
• Give the gate rules (+, max, ×, fan-out) and the $dz = a-y$ simplification.
• Explain vanishing/exploding as a Jacobian product and name clipping vs the structural fixes.

3. Optimization — escaping saddles, not local minima

• The loss surface is non-convex — but that’s not the obstacle you’d expect. Unlike track 10, $J(\theta)$ has no convexity guarantee. Yet gradient descent works, because in high dimensions a critical point is almost never a local minimum: being a min requires the curvature to be positive in all $n$ directions, and with $n$ in the millions the overwhelmingly likely critical points are saddle points (mixed curvature) and plateaus (near-flat regions). So the optimizer’s real job is to cross saddles and plateaus quickly — which is what momentum and adaptive methods buy. Picture (draw it): a saddle — a surface curving up along one axis and down along another (a Pringle); the gradient vanishes at the center but it is not a minimum. In high dimensions almost every flat point looks like this, so progress means sliding off along the down direction — which momentum’s accumulated velocity carries you through.
• Minibatch SGD. Full-batch GD is one exact step per pass over all $m$ examples — too slow at scale. Pure SGD (batch = 1) throws away vectorization and jitters wildly. Minibatches of 64–512 are the practical middle: vectorized speed plus frequent, slightly noisy steps (the noise itself helps jump saddles). One epoch = one pass through the data; the cost curve is jittery, not monotone, since each step sees a different batch. Batch size a power of 2 to fit hardware; $m \lesssim 2000$ → just use full batch.
• Exponentially weighted average — the shared building block.

$v_t = \beta v_{t-1} + (1-\beta)\theta_t$

averages roughly the last $1/(1-\beta)$ values ($\beta = 0.9 \Rightarrow$ ~10). Initialized at $v_0 = 0$ it underestimates early; bias-correct with $v_t / (1-\beta^t)$ (denominator $\to 1$ as $t$ grows).

• Momentum — average the gradient, then step. On iteration $t$:

$v_{dW} = \beta v_{dW} + (1-\beta)\,dW$

$W \leftarrow W - \alpha\, v_{dW}.$

On an elliptical bowl GD zig-zags across the narrow axis; averaging cancels the oscillating component while the consistent down-valley component accumulates — faster, smoother. (Mechanical picture: $dW$ is acceleration, $v$ velocity, $\beta$ friction.)

• RMSprop — equalize step sizes per coordinate by dividing out the recent gradient magnitude:

$s_{dW} = \beta_2 s_{dW} + (1-\beta_2)\,dW^2 \quad(\text{elementwise})$

$W \leftarrow W - \alpha\,\frac{dW}{\sqrt{s_{dW}} + \epsilon}.$

Large-gradient directions get damped, small-gradient directions sped up.

• Adam = momentum + RMSprop with bias correction. On iteration $t$:

$v = \beta_1 v + (1-\beta_1)\,dW, \qquad s = \beta_2 s + (1-\beta_2)\,dW^2$

$\hat v = \frac{v}{1-\beta_1^t}, \qquad \hat s = \frac{s}{1-\beta_2^t}$

$W \leftarrow W - \alpha\,\frac{\hat v}{\sqrt{\hat s} + \epsilon}.$

Defaults $\beta_1 = 0.9,\ \beta_2 = 0.999,\ \epsilon = 10^{-8}$; tune $\alpha$. The default first choice for most nets.

• Learning-rate decay, e.g. $\alpha = \alpha_0 / (1 + \text{decay}\cdot\text{epoch})$: big steps early to traverse the landscape, small steps late to settle into a tight neighborhood instead of orbiting the minimum.

Implications

• Non-convexity is survivable because saddles, not bad minima, dominate — and noise + momentum cross saddles.
• Adam is EWMA applied twice (to the gradient and to its square); knowing the EWMA primitive collapses momentum/RMSprop/Adam into one idea.

Core competency set

• Argue why high-dimensional critical points are saddles, and why minibatch noise helps.
• Write EWMA + bias correction, and the momentum / RMSprop / Adam updates from it; give Adam’s defaults.

4. Making training work — initialization & normalization

• The init problem, straight from §2. Forward activations and backward gradients both scale like a product of $W$‘s; too-large weights explode them, too-small weights vanish them. Goal: choose the initial weight scale so the variance is preserved across layers, forward and backward.
• The variance argument. For $z = \sum_{i=1}^{n} w_i x_i$ with independent, zero-mean, unit-variance inputs and weights,

$\mathrm{Var}(z) = n\,\mathrm{Var}(w).$

To hold $\mathrm{Var}(z) = 1$, set $\mathrm{Var}(w) = 1/n_{\text{in}}$:

• Xavier (tanh/sigmoid): $\mathrm{Var}(w) = 1/n_{\text{in}}$, or $2/(n_{\text{in}}+n_{\text{out}})$ to balance the forward and backward passes at once.
• He (ReLU): $\mathrm{Var}(w) = 2/n_{\text{in}}$ — the extra factor 2 because ReLU zeros roughly half its inputs, halving the variance that survives.
• In code: W = randn(shape) * sqrt(2 / n_prev). Numeric anchor: a layer with $n_{\text{in}} = 512$ ReLU inputs → $\mathrm{Var}(w) = 2/512 \approx 0.0039$, i.e. initialize from $\mathcal N(0,\ 0.0039)$, sd $\approx 0.063$. (Xavier would use $1/512$, sd $\approx 0.044$ — smaller, matched to tanh, which doesn’t zero half its inputs.)
• Input normalization. Subtract the mean, divide by the std (statistics from the training set, applied identically to dev/test). Unnormalized features stretch the loss contours into long ellipses, so GD zig-zags and needs a tiny learning rate; normalized features round the contours and let GD head almost straight to the minimum.
• Batch normalization — normalize the pre-activations of each layer within the minibatch, then give the network back the freedom to rescale. For layer $l$ over a minibatch:

$\mu = \frac1m\sum_i z^{(i)}, \qquad \sigma^2 = \frac1m\sum_i (z^{(i)} - \mu)^2$

$z_{\text{norm}}^{(i)} = \frac{z^{(i)} - \mu}{\sqrt{\sigma^2 + \epsilon}}$

$\tilde z^{(i)} = \gamma\, z_{\text{norm}}^{(i)} + \beta.$

$\gamma, \beta$ are learned (so the layer can recover any mean/variance it needs — even undo the normalization by $\gamma = \sqrt{\sigma^2+\epsilon},\ \beta = \mu$); the bias $b$ becomes redundant and is dropped. Which axis: the statistics are computed per feature, across the batch (and any spatial/sequence positions), so $\mu, \sigma^2$ have one entry per feature — this batch-coupling is exactly why there’s no batch to use at test time. Why it helps: each layer’s input distribution stays stable as earlier layers update (less internal covariate shift), so deep layers aren’t chasing a moving target; it also smooths the loss landscape and adds mild regularization (each batch’s stats are noisy). At test time there’s no batch — use the exponentially weighted running $\mu, \sigma^2$ accumulated during training. (Contrast layer norm, which normalizes each example across its own features — batch-independent, the right choice for sequences; full comparison and the pre/post-norm placement question in 11b §6.)

• Gradient checking (debugging only, never training): centered difference $\frac{J(\theta+\epsilon) - J(\theta-\epsilon)}{2\epsilon}$ has error $O(\epsilon^2)$ (vs $O(\epsilon)$ one-sided); compare to backprop and expect $\sim 10^{-7}$. Include any regularizer; turn off dropout (its randomness breaks the check).

Implications

• Init, input norm, and batch norm are the same move at different points — hold the signal’s variance near 1 so the Jacobian product of §2 doesn’t drift.
• Batch norm decouples layers, which is why it lets you train deeper nets with larger learning rates and less init care.

Core competency set

• Derive $\mathrm{Var}(w) = 1/n_{\text{in}}$ and give Xavier vs He (and why He’s factor 2).
• Write the four batch-norm equations, say why $\gamma,\beta$ and why $b$ drops, and what happens at test time.

5. Generalization & regularization

• The deep-learning puzzle. A large net can memorize random labels (zero training error) yet still generalize on real data — the classic bias–variance U-curve doesn’t cleanly describe the over-parameterized regime. Practical diagnosis still uses the two numbers: high training error → underfitting / bias; low training but high dev error → overfitting / variance; both high → both problems.
• The orthogonal recipe. Fix bias first (look at training fit: bigger net, train longer, better architecture), then variance (look at the dev gap: more data, regularize). Different levers for different problems — diagnose before treating. In DL the two are largely decoupled: a bigger net cuts bias and more data cuts variance, each at little cost to the other.
• L2 / weight decay. Add $\frac{\lambda}{2m}\sum_l \lVert W^{[l]}\rVert_F^2$ to $J$. Its gradient contributes $\frac{\lambda}{m}W$, so the update becomes

$W \leftarrow \Big(1 - \frac{\alpha\lambda}{m}\Big)W - \alpha\,(dW)_{\text{data}}$

— $W$ is multiplied by a factor just under 1 each step (“decay”) before the data gradient. Shrinking weights yields a smaller effective network (many units near-off) and pushes tanh units into their near-linear region → a smoother decision boundary.

• Dropout. At training, zero each unit independently with probability $1-p$; inverted dropout divides the surviving activations by $p$ so the expected activation is unchanged. At test, run the full network (no dropout). Two readings: (1) it trains an ensemble of weight-sharing subnetworks; (2) it prevents co-adaptation — a unit can’t count on any specific partner surviving, so weight spreads across units (an L2-like shrinkage). Heavy in computer vision (chronically data-starved); elsewhere reach for it once you see overfitting. Costs the monotone-cost debugging signal (turn it off to sanity-check $J$).
• Early stopping: halt at the dev-error minimum. Equivalent to limiting how far $W$ travels from its small init (≈ implicit L2). Downside: it couples fitting and regularizing into one knob (non-orthogonal), so L2 + full training is often cleaner.
• Data augmentation: cheap label-preserving input transforms (mirror, crop, color shift) enlarge the dataset — but only label-preserving ones (rotating a handwritten 6 into a 9 corrupts the label).

Implications

• L2, dropout, early stopping, and augmentation all reduce effective capacity or enlarge effective data — the two halves of the variance fix.
• The “memorize yet generalize” phenomenon is why DL regularization is applied empirically (watch the dev gap), not derived from a capacity bound.

Core competency set

• Diagnose bias vs variance from train/dev errors and give the orthogonal recipe.
• Show L2 = weight decay $(1-\alpha\lambda/m)W$; explain inverted dropout (÷$p$, off at test) and its two interpretations.

6. Convolutional networks

• Why not fully connected on images. A $1000\times1000\times3$ image is 3M inputs; a single FC layer to even 1000 units is billions of weights — untrainable and hopeless on limited data. Two structural priors fix it:

  • parameter sharing — a feature detector useful at one location is useful everywhere, so slide one small filter across the whole image (one weight set for all positions);
  • sparse connectivity — each output depends only on a small local receptive field.

  Together they give translation equivariance (shift the input → the feature map shifts) and slash the parameter count.

• Convolution mechanics. A filter of size $f\times f\times n_c$ slides over the input volume; at each position take the elementwise product-and-sum to one scalar. The filter’s depth always equals the input’s channel count; each filter produces one 2D feature map; stacking $n_c'$ filters gives output depth $n_c'$. Worked (vertical-edge filter): convolving a left-bright/right-dark region with the filter $\left[\begin{smallmatrix}1&0&-1\\1&0&-1\\1&0&-1\end{smallmatrix}\right]$ yields a bright stripe exactly at the edge (elsewhere the $\pm$ products cancel) — and these filter weights are learned, not hand-set.

• Conv arithmetic (memorize). Output side length

$n_{\text{out}} = \left\lfloor \frac{n + 2p - f}{s} \right\rfloor + 1.$

Padding: valid ($p=0$, image shrinks and edge pixels are under-counted) vs same ($p=(f-1)/2$ with odd $f$, preserves size). Parameters in a conv layer: $(f\cdot f\cdot n_c^{\text{prev}} + 1)\cdot n_c'$ — independent of image size (the sharing payoff).

• Numeric anchor (LeNet conv-1): input $32\times32\times3$, $f=5$, $s=1$, $p=0$, $6$ filters. Output side $\lfloor(32-5)/1\rfloor + 1 = 28$ → volume $28\times28\times6$. Parameters $(5{\cdot}5{\cdot}3 + 1)\cdot 6 = 76\cdot 6 = 456$ — versus a fully-connected layer to the same $28{\cdot}28{\cdot}6$ outputs, which would need $\sim\!1.4\times10^{10}$ weights. That gap is the sharing payoff.
• Picture (draw it): one small $f\times f\times n_c$ filter sliding across the input volume, dotting into a single scalar at each stop to paint one 2D feature map; stack $n_c'$ such filters (each its own slid kernel) for the output depth. The same weights at every position is the parameter sharing; the small window is the sparse connectivity.
• Pooling: downsample each channel over an $f\times f$ window with stride $s$ (max — keep the strongest activation; or average), no learnable parameters, output $\lfloor (n-f)/s\rfloor + 1$. Shrinks H×W and adds a little translation invariance.
• Typical architecture: $[\text{CONV} \to \text{ReLU} \to \text{POOL}]\times k \to \text{flatten} \to \text{FC} \to \text{softmax}$; spatial size shrinks while channel depth grows.
• The classic ladder.
  • LeNet-5 (~60k params): the conv–pool–conv–pool–FC template, on digits.
  • AlexNet (~60M): much bigger, ReLU — launched the deep-vision era.
  • VGG: uniform $3\times3$ same-convs, doubling channels / halving spatial at each block — depth through uniformity (~138M params).
  • ResNet — the key idea. A residual block adds a skip connection around two layers:

$a^{[l+2]} = g\big(z^{[l+2]} + a^{[l]}\big) = g\big(W^{[l+2]}a^{[l+1]} + b^{[l+2]} + a^{[l]}\big).$

Now the identity is *free*: $W^{[l+2]} = 0$ gives $a^{[l+2]} = a^{[l]}$, so extra layers can never hurt and the block only has to learn a *residual* on top of the identity. The skip is also a direct gradient path back (the additive term keeps the Jacobian product near 1), which is what finally made 100+-layer nets train — training error decreases monotonically with depth instead of the empirical U-curve. (Dimension mismatch on the skip: insert a $W_s a^{[l]}$.) *Same additive gradient-highway idea as the LSTM cell, 11b.*

• $1\times1$ convolution (“network in network”): a per-pixel FC across depth — multiplies/sums across channels. Cheap way to reduce the channel count (or just add a nonlinearity at fixed depth).
• Inception: run $1\times1$, $3\times3$, $5\times5$, and pooling in parallel and concatenate — let the net pick the filter size. A $1\times1$ “bottleneck” before the big filters cuts compute ~10× (e.g. $28{\times}28{\times}192 \to 16$ channels $\to 5{\times}5$ conv).
• Transfer learning — almost always the right move. Take an ImageNet-pretrained net, replace the final layer with your classes, freeze early layers (generic edges/textures), and train the rest; with more data, unfreeze more. Strong performance even on small datasets.
• Application — detection. Localization adds bounding-box outputs to the class: target $y = (p_c,\ b_x, b_y, b_h, b_w,\ c_1,\dots)$, where $p_c$ flags object-present (and when $p_c=0$ the rest is “don’t care”). YOLO lays a grid on the image and predicts one such vector per cell in a single forward pass (object assigned to the cell holding its midpoint). IoU = intersection/union of boxes scores overlap (correct if $\ge 0.5$). Non-max suppression keeps the highest-$p_c$ box and discards others overlapping it (IoU $> 0.5$), removing duplicate detections. Anchor boxes give each cell several box shapes so it can predict multiple overlapping objects.
• Application — face recognition. It’s a one-shot problem (recognize a person from a single enrolled photo), so learn an encoding, not a classifier. A Siamese network maps each image to $f(x)\in R^{128}$; similarity is $d(x_i,x_j) = \lVert f(x_i) - f(x_j)\rVert_2^2$. Triplet loss trains it on (Anchor, Positive, Negative):

$L(A,P,N) = \max\Big(\lVert f(A) - f(P)\rVert^2 - \lVert f(A) - f(N)\rVert^2 + \alpha,\ 0\Big).$

Pull anchor–positive together, push anchor–negative apart, by a margin $\alpha$ — the margin is what blocks the trivial all-zero encoding (same margin idea as the SVM, track 6). Mine hard triplets ($d(A,P)\approx d(A,N)$) or gradient descent has nothing to do.

Implications

• Convolution is an architectural prior — sharing + locality — that encodes “images are translation-equivariant,” cutting parameters by orders of magnitude.
• ResNet’s skip connection is the §2 Jacobian-product fix made structural; it recurs as the LSTM cell highway in 11b.

Core competency set

• State the two CNN priors and the conv output-size and parameter-count formulas; same vs valid padding.
• Explain the residual block: why identity is free and why it fixes deep training.
• Give the detection toolkit (YOLO/IoU/NMS/anchors) and the Siamese + triplet-loss recipe with the role of $\alpha$.

7. Gradients beyond the weights

• The unifying move. Backprop computes the gradient w.r.t. any node of the graph. Freeze the weights and differentiate w.r.t. the input $x$, and a whole family of techniques drops out.
• Adversarial examples. Find an input classified wrongly while staying near a real image. The fast gradient sign method perturbs every pixel by $\pm\epsilon$ along the loss gradient’s sign:

$x^* = x + \epsilon\,\mathrm{sign}\big(\nabla_x J(W, x, y)\big).$

Why a tiny perturbation flips the prediction: its effect on a logit $w^Tx$ is $\approx \epsilon\sum_i |w_i|$, which grows with dimension — so a change imperceptible per-pixel moves the output a lot in high dimensions. Defense: adversarial training, adding $\lambda\,L(W, x_{\text{adv}}, y)$ to the loss.

• Interpretability via input gradients.
  • Saliency map: $\big|\partial\,\text{score}_c / \partial x\big|$ over pixels — use the pre-softmax score (it depends only on class $c$‘s evidence, whereas post-softmax mixes in other classes); the magnitude highlights which pixels matter, drawing the object’s silhouette.
  • Occlusion sensitivity: slide a gray patch across the image and watch the class probability drop where the object is.
  • Class activation map (CAM): replace the flatten+FC head with global-average-pool → FC → softmax; the FC weights say which feature maps drive the class, and each map localizes back to image regions — fast enough for real time.
• Neural style transfer. Optimize the image (not the weights) to carry the content of $C$ and the style of $S$, using a fixed pretrained net:

$J(G) = \alpha\,J_{\text{content}}(C, G) + \beta\,J_{\text{style}}(S, G), \qquad G \leftarrow G - \frac{\partial J}{\partial G}.$

Content matches mid-layer activations: $J_{\text{content}} = \tfrac12\lVert a^{[l](C)} - a^{[l](G)}\rVert^2$. Style matches the Gram matrix of channel correlations $G_{kk'}^{[l]} = \sum_{i,j} a_{ijk}\,a_{ijk'}$ — style is how features co-occur, with spatial layout discarded.

• GANs. A generator $G$ maps noise $z$ to fake samples; a discriminator $D$ classifies real vs fake; they play a minimax game.
  • $D$ minimizes binary cross-entropy: reward $D(x_{\text{real}})\to 1$ and $D(G(z))\to 0$.
  • $G$ wants to fool $D$. The naive objective $\min \log(1 - D(G(z)))$ saturates early — when $D$ easily wins, $D(G(z))\approx 0$ and the gradient is tiny. Fix with the non-saturating loss

$J^{(G)} = -\frac{1}{m}\sum_i \log D\big(G(z^{(i)})\big),$

same optimum, large gradient early in training. Train $D$ and $G$ alternately (sometimes $D$ for $k$ steps per $G$ step). The loss value does *not* track sample quality (it's relative to $D$'s current strength) — judge by inspecting samples.

Implications

• “Differentiate w.r.t. the input” is one idea behind attacks, explanations, art, and (with a second network) generative modeling.
• Adversarial vulnerability is a property of high-dimensional linear-ish models, not a bug to be fully patched — it reframes robustness as part of the loss.

Core competency set

• State FGSM and the dimension argument for why small perturbations flip predictions.
• Give saliency (pre-softmax, $|\partial\text{score}/\partial x|$), CAM, the style-transfer content/Gram split, and the GAN $D$/$G$ losses with the non-saturating fix.

8. Memorize cold

• Network = composition $a^{[l]} = g^{[l]}(W^{[l]}a^{[l-1]} + b^{[l]})$; drop $g$ and the whole net collapses to one affine map.
• Activation derivatives: $\sigma' = \sigma(1-\sigma)$, $\tanh' = 1 - \tanh^2$, $\text{ReLU}' = \mathbb 1[z>0]$. ReLU avoids tail saturation → fixes vanishing gradients.
• Backprop, one layer: $\delta = \dfrac{\partial J}{\partial a}\odot g'(z)$; $\ \dfrac{\partial J}{\partial W} = \delta\,a_{\text{prev}}^T$; $\ \dfrac{\partial J}{\partial b} = \delta$; $\ \dfrac{\partial J}{\partial a_{\text{prev}}} = W^T\delta$. Cross-entropy+sigmoid/softmax: $dz = a - y$.
• Vanishing/exploding = product of Jacobians $\prod_l W^{[l]T}\mathrm{diag}(g')$ drifting from 1; clip to tame exploding.
• EWMA $v_t = \beta v_{t-1} + (1-\beta)\theta_t$ (~$1/(1-\beta)$ window), bias-correct $/(1-\beta^t)$. Momentum = avg gradient; RMSprop = ÷$\sqrt{s}$; Adam = both + correction ($\beta_1{=}0.9,\ \beta_2{=}0.999,\ \epsilon{=}10^{-8}$).
• Init variance: $1/n_{\text{in}}$ (Xavier, tanh), $2/n_{\text{in}}$ (He, ReLU).
• Batch norm: per-batch $z_{\text{norm}} = \frac{z-\mu}{\sqrt{\sigma^2+\epsilon}}$, then $\tilde z = \gamma z_{\text{norm}} + \beta$ ($\gamma,\beta$ learned, $b$ dropped); test uses running $\mu,\sigma^2$.
• L2 = weight decay $(1-\alpha\lambda/m)W$. Dropout = ensemble / anti-co-adaptation (÷$p$ at train, off at test).
• Conv output $\lfloor(n+2p-f)/s\rfloor + 1$; same pad $p=(f-1)/2$; layer params $(f^2 n_c^{\text{prev}} + 1)\,n_c'$, image-size-independent.
• ResNet skip $a^{[l+2]} = g(z^{[l+2]} + a^{[l]})$ → identity free, gradient highway.
• Triplet loss $\max(\lVert f(A)-f(P)\rVert^2 - \lVert f(A)-f(N)\rVert^2 + \alpha,\ 0)$.
• FGSM $x^* = x + \epsilon\,\mathrm{sign}(\nabla_x J)$.
• GAN: $D$ maximizes real/fake CE; $G$ non-saturating $-\log D(G(z))$.

Named moves (cross-track glossary): compose-affine-and-nonlinearity (the network); reverse-mode-autodiff (the $\delta$ backward sweep); observed-minus-expected ($dz = a-y$, shared with softmax and word2vec, 11b); keep-the-Jacobian-product-near-1 (ReLU / He init / batch norm / residual all unify here); EWMA-the-gradient (momentum/RMSprop/Adam); normalize-the-activations (input norm + batch norm); decay-the-weights (L2); ensemble-by-dropout; share-and-localize (convolution); skip-connection (ResNet, and the LSTM highway in 11b); bottleneck-with-$1\times1$; differentiate-w.r.t.-the-input (adversarial / saliency / style transfer); minimax-game (GAN); margin (triplet loss, SVM track 6).