Learn machine learning · part 9 · session 3
What scale buys
Four models, one script, one corpus, four sizes. A line through their endpoints is the useful part of scaling, and the second half of this page is about what it does not give you.
After this session you will be able to fit a line through four runs of one script and say what it predicts and what it does not.
To pass: two of three calls on unseen variants, each revealed with the rows behind it.
01 · The question
Four models, one script, one corpus, four sizes. None of them was trained here. The curves and samples on this page are the values the script produced, stored in the repository, and the script that wrote them sits beside them. Plot each model's final loss against its parameter count on a logarithmic axis and the four points sit close to a line. That line is the useful part of scaling. What it does not give you is a way to read a capability off the axis.
The largest of the four is the model 9.2 ran: about 112,000 parameters.
Pretraining at a size where the word scale means anything is out of reach here, and this page makes no claim that needs it.
02 · Predict
Loss here is how surprised the model is by the next character, in nats: 0 means certain and right, and about 4.2 means it is guessing evenly among the 64 characters this script uses. Lower is better. The largest of the four models below is 111,808 parameters, so the question is where a model about ten times bigger would land.
Before anything runs
Estimate the final loss this script would reach at about 1 million parameters.
03 · Try it
Loading the stored curves.
04 · Compare
Call the three variants below and the verdict lands here, against the number you committed to.
05 · Check your understanding
Your check: predict results on new examples
not yetTwo of three. The held-out reading is accepted within 0.08 of the stored value, and every reveal opens the rows behind it.
Called 0 of 0 variants so far.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question about the fit: available after 3 evaluated runs.
2. What the second variant is asking: available after 5 evaluated runs.
3. A different two-point fit, computed in full: available after 7 evaluated runs.
4. The three calls with the stored values: available after 9 evaluated runs.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.
10.4 cites this fit when it places the data-hunger axis.