Learn machine learning · part 5 · session 3
Activations matter
A five-layer network here holds two hundred and forty nine parameters against the three in the line you fitted in 4.1. With identity on every hidden layer it draws the same line, and one multiplication on paper says why.
After this session you will be able to say what the activation function is for, and show that without one a deep network collapses to a line.
The gate: two of three calls on unseen variants, each one a cell of the same recording.
01 · The question
A five-layer network here holds 249 parameters, meaning 249 separate numbers the training is allowed to move, against the three in the line you fitted in 4.1, so it should be able to do more. Set every squashing step, the step this track calls the , to identity, which means no squash at all: the layer hands its total straight on. Now each layer only multiplies and adds, and it does not do more. It draws the same line, and I overlay them so you can see they coincide.
The reason is one multiplication you can do on paper. Commit first: with identity on every hidden layer, is held-out accuracy higher than, the same as, or lower than the single line.
What was recorded
recorded runrecord-runs.mjs · seed 7Ten runs of 600 steps: four activations at width 8 and at width 64, plus the two variants the gate asks about. The script is in the repository at scripts/ml/grids/activations-matter.mjs. The 4.1 line is not recorded: it is refitted here on the same 300 training rows.
02 · Predict
Before anything runs
With identity on every hidden layer, held-out accuracy against the 4.1 line will be:
Choose something first.
03 · Try it
1 and 3 · play the identity cell, then the tanh cell
Moving either select loads a cell. It never starts one: play is always a button press.
Loading the recorded runs.
Play is disabled until the commitment above is recorded. That order is the whole mechanic.
2 · multiply the chain by hand
Five layers, one multiplication
illustrativethe hand fixture on this pageFive layers, one weight each, and 0.80 arriving at the first.
One unit per layer, five layers, weights 2, 0.5, -3, 1.5, 0.5 and every bias at zero. What single weight does that chain amount to?
to two decimals, within 0.01
Attempts: 0
With that chain, an input of 0.80 leaves the stack as −1.80.
4 · inspect the pre-activations
The histogram inside the player above reads the first hidden layer's pre-activations at each of the eight recorded checkpoints. The checkpoint select and the step scrubber both move it.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, two layers in two dimensions: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: predict results on new examples
not yetThree variants, two calls right. Each variant is a cell of the same recording, and you can open it in the player after you have called it.
Variant 1 of 3
Identity on all four hidden layers, width raised from 8 to 64. That is 12,737 parameters against 249. What does the boundary do?
Call it before you run it.
Variant 2 of 3
tanh on the first hidden layer only, identity on the other three. What does the boundary do?
Call it before you run it.
Variant 3 of 3
ReLU everywhere, with every first-layer bias held at −12 and the inputs inside [−2, 2]. What does the boundary do?
Call it before you run it.
Called 0 of 3 variants so far.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.