Learn machine learning · part 7 · session 5
Train a character model
The first model in this track that produces text. It has 6,272 parameters, it sees one character at a time, and its whole training run is recorded so your phone draws a curve instead of grinding through the work.
After this session you will be able to read a character language model's training run and say what its samples do and do not show.
The gate: name the first checkpoint whose samples take word shape, within one, and get the six-statement checklist right.
01 · The question
This is the first model in the track that produces text. It has 6,272 parameters and it sees one character at a time. I trained it once with the same engine this page runs and recorded every step, 1,500 of them, keeping twelve samples along the way, closer together early where the text changes fastest.
Your phone plays that recording rather than repeating the work. From random weights the samples are noise. Within the first hundred steps they already have runs of letters with spaces in plausible places and almost no real words, and by the end they still have almost no real words. That gap is the lesson, and beside it I have put samples from the same code at a larger size, trained offline, so you can see what more parameters and more steps buy. Forty minutes. The gate is a checkpoint and a list.
1 · read the run's settings
recorded runOne LSTM layer, hidden 24, vocabulary 32, one-hot input, 6,272 parameters, recorded by scripts/ml/grids/train-a-character-model.mjs. The text is cut into 17-character windows; nine out of ten train the model, the rest are kept back to score it, split at seed 7.
Loss is measured in nats per character: how surprised the model is by each character it has to predict. Lower is less surprised. Guessing blindly among 32 characters costs 3.47.
The alphabet is 32 symbols: 26 letters, space, full stop, comma, apostrophe, newline and one out-of-vocabulary symbol. A loss in nats per character is only comparable within the same alphabet and corpus, so no number here can be set beside a published model's loss or beside a loss from Parts 8 or 9.
Text: The Wonderful Wizard of Oz, L. Frank Baum (1856-1919), 1900. Public domain.
02 · Predict
Before anything runs
Checkpoint 5 of this run is step .... What will its sample look like?
Choose something first.
The commitment is taken before the player is unlocked, so the first press of play is a test of it.
03 · Try it
2 · play the run
The player is locked until the commitment above is recorded. That order is the whole mechanic.
Writing one character at a time
illustrativethe loop, drawn for the lessonThe text so far is the prompt, and the model gives one number per character.
Word shape is the fraction of chunks between spaces that look word-shaped: 1 to 12 characters with a vowel. Random letters score about 0.30, the book itself 0.99.
Each checkpoint below shows two samples: two ways of drawing the same text, where the left plays it safe and the right takes more risks. Part 9 makes that dial explicit.
3 · read the checkpoints
recorded runrecord-runs.mjs, both samples storedThe recorded run has not arrived yet, so there are no samples to read.
No checkpoint loaded.
For scale: uniform random characters over this alphabet score about 0.30, and the corpus itself scores about 0.99.
How word shape is defined
Split the sample on spaces and newlines, discard the first and last fragment, and report the fraction of the rest that are 1 to 12 characters long and hold at least one vowel, counting "y" as a vowel. It is a statistic about the shape of the fragments and not a quality score. Uniform random characters over this alphabet score about 0.30 and the corpus itself scores about 0.99.
The rates printed under each sample are recomputed here from the stored text, so you can check the definition against what is on screen.
4 · the same code at a larger size
Loading the offline samples.
5 · what it acquired
computed in this browserchecked in this browserTick the statements that are true of this 6,272-parameter run. Three of the six are.
Ticked 0 of 6
6 · train this model on your own text
Optional, and off the gate. The same model trains here on text you paste, up to 8,192 bytes, in a worker so the page stays usable while it runs. 600 steps is about 10 seconds here on a mid-range phone, and the panel states the wait again before anything starts. The gate reads the recorded run either way.
Name the crossing checkpoint
Which is the first checkpoint whose temperature 0.8 sample has a word-shape rate at or above 0.60? An answer within one of the recorded one passes.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on the smaller model from 7.2: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: reach the stated target
not yetBoth parts in one session, read from the recorded run: the first checkpoint whose temperature 0.8 sample has a word-shape statistic at or above 0.60, answered within one checkpoint of the recorded answer, and the six-statement checklist with its three correct statements selected. The loss target stays on screen as the reference line: held-out loss at or below 2.30 nats per character, which this run reaches before its last checkpoint.
Not met yet. Keep working the session.
Checkpoint call: Name the checkpoint before the gate can read it. Checklist: not checked yet.
Each sample is 200 characters continued from the prompt "the ", drawn with a generator seeded from the step number, so the same checkpoint reproduces the same text.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.
The shelf copy is this run's own weights block, int8 with one scale per matrix, the same bytes the recording ships, so nothing is retrained to produce it. It is a character language model of 6,272 parameters.