Learn machine learning · part 5 · session 5
Initialisation and batches
Two networks with the same architecture, the same data and the same loss can behave completely differently. The only difference is the numbers they started from and how many rows go into each update.
After this session you will be able to read a loss curve for a setup fault rather than a modelling fault.
The gate: two of three calls on curves you have not played.
01 · The question
Two networks with the same architecture, the same data and the same loss can behave completely differently, and the only difference is their , the numbers they started from, and the , how many rows go into each update. That is not a footnote. Most of the failed training runs I have looked at were set up wrong rather than modelled wrong.
I trained all nine combinations of three starting schemes and three batch sizes once and shipped them, so your phone plays the curves instead of grinding them out. Your job is to say what each one will do before you press play. Commit first: with every weight and bias at exactly zero, what does the training loss do over the run.
The three schemes, as numbers
syntheticloan seed 7, 1,200 training rowsFan-in is how many wires arrive at a unit. randn draws a random number centred on zero, and mulberry32 is just the random-number generator, named so you can reproduce this. std is how spread out those random numbers are.
| scheme | weights | biases | std, fan-in 15 | std, fan-in 8 |
|---|---|---|---|---|
| zeros zeros | every weight exactly 0 | 0 | 0.0000 | 0.0000 |
| too-large large | 3.0 × randn(mulberry32(seed)) | 0 | 3.0000 | 3.0000 |
| sensible (Xavier) xavier | sqrt(1 / fan-in) × randn(mulberry32(seed)) | 0 | 0.2582 | 0.3536 |
Xavier scaling is the standard choice for tanh units. The ReLU counterpart uses sqrt(2 / fan_in). The one-hot encoding widens the loan input from nine named features to fifteen columns, which is the fan-in the first layer actually sees.
02 · Predict
Before anything runs
With every weight and bias at exactly zero, over the 600 recorded steps the training loss will:
Choose something first.
03 · Try it
1 · mark the chart, then play the three schemes
Before play is enabled, put a line where you expect the training loss to sit at the last step. The mark stays on the chart while the run plays.
Play is enabled once the mark is on the chart.
2 · trace zeros by hand on fixture N221
All nine parameters at zero, SGD at 1.0, the same 2-2-1 network as 5.4. Three values.
within 0.0005
Attempts: 0
within 0.001
Attempts: 0
within 0.0005
Attempts: 0
The trace, three steps
computed in this browsermicrograd.ts, fixture N221| step | h1, h2 | ŷ | L | ∂L/∂w11, ∂L/∂w21 | v1, v2 after |
|---|---|---|---|---|---|
| 1 | 0.5000, 0.5000 | 0.5000 | 0.1250 | 0.0000, 0.0000 | 0.0625, 0.0625 |
| 2 | 0.5000, 0.5000 | 0.5467 | 0.1027 | −0.0018, −0.0018 | 0.1187, 0.1187 |
| 3 | 0.5010, 0.5010 | 0.5881 | 0.0848 | −0.0030, −0.0030 | 0.1686, 0.1686 |
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment on a smaller network: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: predict results on new examples
not yetThree curves, two calls right. Each one is a cell of the same recording, and you can open it in the player above after you have called it.
Called 0 of 0 variants so far.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.