Learn machine learning · part 6 · session 2
Three regularisers
L2, dropout and early stopping are not interchangeable. Each one changes a different thing, and the weight histogram tells you which one you actually used.
After this session you will be able to tell three regularisers apart by what each one changes rather than by what each one is called.
The gate is three calls: name the right control for two of the three failed configurations.
01 · The question
Three , three different mechanisms. L2 adds the size of the weights to the loss, so the training is charged a fee for every large number it holds. It will still take a large weight if the accuracy gained is worth the fee, and otherwise it shrinks it. removes a random share of the hidden units on every step, so no single unit can carry a result alone. Early stopping changes nothing about the model and takes the weights from an earlier step.
Watch the weight histogram while you move each one, because it tells you which of the three you actually used. I trained all twenty four settings once and shipped them, so each control loads a run and the histogram moves between them. Commit first: which control narrows the histogram most?
02 · Predict
Before anything runs
Which control leaves the model with the smallest weights?
Choose something first.
03 · Try it
Move one control at a time
Start with all three at their off value, which is the baseline cell. Then take L2 to 0.01 on its own, dropout to 0.2 on its own, and early stopping to on with the other two at zero. Say what will happen before each cell loads.
Units switching off
illustrativedropout 0.2, masks from mulberry32 at seed 7A layer of 16 units, all present, all wired in.
Three controls, twenty four recorded cells
syntheticloan seed 7With no regulariser at all, how large do you expect the largest weight in the three matrices to be?
Three failures to call
Each of these is a real configuration that record-runs.mjs trained once and shipped in the same file. Read what each one left behind, then make the three calls at the gate. A wrong call still opens the run.
Failure A
recorded runrecord-runs.mjs · seed 7 · undefined1,200 training rows on a wider network, a large training-validation gap, and a weight histogram with a long tail.
Failure B
recorded runrecord-runs.mjs · seed 7 · undefined200 training rows on the base network. Validation loss reaches its minimum early and rises for the rest of the run.
Failure C
recorded runrecord-runs.mjs · seed 7 · undefined1,200 training rows on a much wider network with L2 already in the loss. The gap is small and the weights are moderate, but zeroing one hidden unit at test time costs more than twice what the next unit costs, and most of the units cost nothing at all.
This is not dropout. Dropout happens during training; here the training is finished and we are switching one unit off to see how much the finished model depended on it. The probe below is the evidence for this card: read it before you make the call.
The unit-drop probe
recorded runrecord-runs.mjs · seed 7 · failure Ccomputed in this browser400 held-out rowsPreparing the held-out rows.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on a different network: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: predict results on new examples
not yetThree failed configurations, three controls. Name the right control for two of the three. Each call is locked in before its note is revealed.
Variant 1 of 3
Failure A. 1,200 training rows on a wider network, a large training-validation gap, and a weight histogram with a long tail.
Call it before you run it.
Variant 2 of 3
Failure B. 200 training rows on the base network. Validation loss reaches its minimum early and rises for the rest of the run.
Call it before you run it.
Variant 3 of 3
Failure C. 1,200 training rows on a much wider network with L2 already in the loss. The gap is small and the weights are moderate, but zeroing one hidden unit at test time costs more than twice what the next unit costs, and most of the units cost nothing at all.
Call it before you run it.
Called 0 of 3 variants so far.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.