Learn machine learning · part 8 · session 6
Attention versus LSTM
Same task, same data, same budget, two models. One passes the symbol through every intervening step; the other reaches it in one weighted average. Twelve runs recorded once, played back here.
After this session you will be able to say why attention displaced recurrence for long sequences, in terms of path length rather than capability.
The gate is three calls: which curve is which, where they separate, and why.
01 · The question
Same task, same data, same budget, two models. The task is the one you set by hand in 7.4: carry the symbol at position 3 to the end. The recurrent model has to pass it through every intervening step. The attention model reaches it in one weighted average regardless of how far back it is.
Below you can raise the sequence length and play the pair of runs I recorded at that setting, and watch what the difference costs. Twenty five minutes. The gate asks for the length where the two curves separate and the reason, and the reason is about path length, not about one model being better.
02 · Predict
Before anything runs
At which sequence length do you expect the attention model's held-out accuracy to overtake the recurrent model's?
Choose something first.
03 · Try it
Six lengths, twelve runs
Both models were trained with the same engine, on the same batches, for 500 steps at seed 7, with held-out exact match measured every 25 steps: the fraction of 200 unseen sequences it gets exactly right. Nothing trains on this page.
The sequence length
Six settings were recorded, and this picker steps through them. Both models were trained at every one of them with the same data, the same budget and the same seed.
Position 6 of 6
Length 34. Attention model 2440 parameters, recurrent model 3368.
Loading the recorded runs.
The two paths from position 3 to the answer
illustrativeRecurrent: 31 sequential steps, which is the length minus 3. Attention: 1 weighted average, at every length.
At length 34 the recurrent path from position 3 to the answer is 31 sequential steps, and the attention path is 1.
Both rows are the same 34 positions. The upper row is the chain the recurrent model passes the symbol along; the lower is the single edge the weighted average draws.
The table you are filling
A row appears for each length you play. The difference column is the attention model's held-out exact match minus the recurrent model's.
| length | attention | recurrent | difference | recurrent path |
|---|
Play a length to put its row in the table.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on a different task: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: predict results on new examples
not yetThree calls on the recorded grid: which series is the attention model, where the two separate, and which explanation matches the numbers. Two of the three passes.
Variant 1 of 3
The two curves with their labels hidden. Which series is the attention model?
Call it before you run it.
Variant 2 of 3
At which recorded length does the attention model first lead by more than the width of the measurement?
Call it before you run it.
Variant 3 of 3
Which explanation of the difference matches the numbers on this page?
Call it before you run it.
Called 0 of 3 variants so far.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.