Learn machine learning · part 8 · session 5
The transformer block
Every part of it is something you have already built. What is left is the order, and the order is checkable: most wrong orders produce a shape that cannot be multiplied, and the rest produce a number you can measure.
After this session you will be able to assemble a transformer block from its eight parts and say what the residual path is measured to do.
The gate is two things you produce: an assembly that passes the shape calculator and matches the order, and one wrong order whose failure you can name.
01 · The question
Here is the whole , in parts, for the first time. You have built every part already: the weighted average in 8.2, the positional rows in 8.4, the normalisation and the residual path from Part 5 and Part 6.
Each block has two bypass lanes that let the signal skip past the work, and two steps that rescale it. The lane is a and the rescaling is .
Assemble it, then play the two runs I recorded on an eight-deep stack, one with the residual paths and one without. The result is not the one this topic is usually given, and I have left it as measured. Thirty five minutes.
This track normalises after each part rather than before; other arrangements exist and the difference is a design choice.
02 · Predict
Each block has two bypass lanes that let the signal skip past the work, and two steps that rescale it. Take the bypass lanes out of a stack of eight blocks and leave the rescaling in. What happens to the learning signal that reaches block 1?
Before anything runs
Take the two residual adds out of an eight-block stack, leaving the layer norms in. What happens to the size of the gradient at block 1?
Choose something first.
03 · Try it
One block, with its bypass lanes
illustrativethe order the code runs, drawn for the lessonThe signal goes in: one row per position, and the positional rows are already added.
Follow the tensor and its bypasses
illustrativeshape diagram · batch 1Select a step to follow eight positions through this post-norm block. Each small cell is one number; the diagram shows dimensions, not activation values. The feed-forward component is opened into its two matrices here.
Embed + position · output shape
8 positions × 16 features
Same scale at every step: full width = 64 columns.
x = token rows + positional rows
Eight token IDs become eight rows of 16 features. Save x for the first residual add.
Order the eight components
The toy is one block at batch 1, sequence 8, d 16, two heads of eight, feed-forward 64 and a vocabulary of 32. Every step in the block expects numbers arranged in a particular grid, and that arrangement is its shape. Most wrong orders hand a step a grid it cannot work with, and the calculator stops at the first one.
The eight components
Focus a card, press Enter to lift it, move it with the arrow keys, press Enter to drop it and Escape to put it back. The two move buttons beside a lifted card do the same thing with a thumb. Two labels appear twice, and the checker treats the two of a kind as interchangeable.
Put the eight components in the order the block runs them
Every card has to be used, and only the order changes.
The gate asks for one wrong order as well as the right one, so run a wrong one deliberately if you got it first time.
Stack eight of them
I built the same block at every depth from 1 to 8, ran one forward and one backward pass over a fixed batch, and measured the size of the gradient at the embedding. Then I did it again with the layer norms taken out, because the first pair on its own would have told you something misleading.
Loading the recorded pair.
The same stack, trained four ways
Gradient size is not quality, so here is the same eight-block stack trained on the copy task at length 16, 400 steps at seed 7, with the residual adds and the layer norms switched on and off. Predict each row before you open it. This is the table the deep dive attached to this session finishes.
Four rows, predicted then opened
recorded runLoading the recorded rows.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on a two-component block: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: an outcome you produce
not yetTwo things, in one session. An assembly that passes the shape calculator and matches the eight-step order, and one wrong order submitted at least once with the right explanation chosen from the four generated for it.
Not met yet. Keep working the session.
Where the two halves stand
Accepted order: not yet. A wrong order explained: not yet.
The checker compares the ordered list of components, so an order that happens to produce legal shapes but puts a residual add around the wrong operation is rejected with that reason named.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.