Learn machine learning · deep dive D9 · optional
Layer norm and residuals
Two small rows, normalised by hand, with the epsilon as a switch you can see the effect of. Then an eight-block stack with the residuals and the norms taken out, one combination at a time.
This is optional. It opens once 8.5 The transformer block is done, and nothing in the track waits on it. The gate is six numbers within 0.001, the right reading of the epsilon, four predictions recorded before any row plays, and the best row played.
01 · The question
8.5 assembled the block and named its eight components. Two of them are the ones a deep stack cannot do without, and this dive takes them out to see which is which. Before that, the arithmetic: layer norm is a mean, a variance and a division, and the epsilon under the square root is not decoration.
The hand computations are live. The ablation rows are training runs the course script produced once and shipped, because nothing in this track trains an eight-block stack on your phone.
Commit first, then compute.
02 · Predict
Before anything runs
On an eight-block stack that has both removed, which single component recovers more when you put it back?
Choose something first.
03 · Try it
Layer norm, by hand
Mean, variance, then divide. The switch adds the epsilon of 1e-5 under the square root, so the difference is something you read rather than something you take on trust.
Layer norm, one row at a time
computed in this browsereps 0.00001Row [1, 2, 3]
- 1. Mean: add the entries and divide by 3, which gives 2.000000.
- 2. Variance: average the squared distances from that mean, which gives 0.666667.
- 3. Divide each distance by the square root of the variance plus 0.00001, which is 0.816503.
Out: [-1.224736, 0.000000, 1.224736]
Row [2, 2, 2, 6]
- 1. Mean: add the entries and divide by 4, which gives 3.000000.
- 2. Variance: average the squared distances from that mean, which gives 3.000000.
- 3. Divide each distance by the square root of the variance plus 0.00001, which is 1.732054.
Out: [-0.577349, -0.577349, -0.577349, 1.732048]
The row where the epsilon earns its place
[4, 4, 4, 4] has a variance of exactly zero. With the epsilon off, every entry is divided by zero and the row comes back as [not a number, not a number, not a number, not a number]. With it on, the row comes back as [0.000000, 0.000000, 0.000000, 0.000000]. Right now the switch is on, so this row reads [0.000000, 0.000000, 0.000000, 0.000000].
[1, 2, 3] has mean 2.000000 and variance 0.666667; at eps 0.00001 it normalises to [-1.224736, 0.000000, 1.224736]. [2, 2, 2, 6] has mean 3.000000 and variance 3.000000; at eps 0.00001 it normalises to [-0.577349, -0.577349, -0.577349, 1.732048].
Mean, variance and the normalised row, computed in this browser at the epsilon shown.
Show the steps as a table
| row | mean | variance | denominator | normalised |
|---|---|---|---|---|
| [1, 2, 3] | 2.000000 | 0.666667 | 0.816503 | -1.224736, 0.000000, 1.224736 |
| [2, 2, 2, 6] | 3.000000 | 3.000000 | 1.732054 | -0.577349, -0.577349, -0.577349, 1.732048 |
Six numbers
Tolerance is 0.001. All six have to land.
0.001
Attempts: 0
0.001
Attempts: 0
0.001
Attempts: 0
0.001
Attempts: 0
0.001
Attempts: 0
0.001
Attempts: 0
One reading to choose
What is the epsilon under the square root for?
Choose one before locking it in.
The ablation
Four combinations of the two components, on the same eight-block stack, on the copy task at length 16. Predict each row before you play it. What comes out is what these four runs did, at this depth, at this seed, and nothing more general than that.
The ablation at depth 8
recorded runrecord-runs.mjs · the-transformer-block.jsonLoading the recorded rows.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. The answer: available after 5 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: an outcome you produce
not yetSix hand computations within 0.001, the right reading of the epsilon, all four predictions recorded before any row plays, and the best row played.
Not met yet. Keep working the session.
0 of 6 numbers checked. 0 of 4 predictions recorded. The best row has not been played. The ablation rows are not in the 8.5 recording yet, so this gate cannot close today.
The target is read from the recording, not written here: it is the best of the four rows, whatever that turns out to be.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.