Learn machine learning · part 9 · session 2
One step, one distribution
Given the text so far, the model produces one score for every character in its vocabulary, a full set of odds for what comes next, adding up to 1. It computes nothing beyond that one step.
After this session you will be able to read a next-character distribution and say what temperature does to it.
To pass: produce the target string exactly, with every character selected from the distribution list rather than sampled, and the right call on why five runs differ.
01 · The question
This model was trained by a script in the repository, not in your browser, and it predicts the next character rather than the subword tokens you built in the last session. Production models work on subword tokens, for the reason that session showed, and the operation is the same one either way: . One step takes a prompt of at most 64 characters and returns one number for every character in the vocabulary. On this page you are the sampler: read the distribution and pick, or hand the pick to and see what it costs you.
The model has about 112,000 numbers inside, which is tiny.
The bundled model's shape
2 blocks, d = 64, 4 heads, context 64, a 64-character vocabulary, about 112,000 parameters, stored as int8 with per-row scales.
The Wonderful Wizard of Oz, L. Frank Baum (1856-1919), 1900. Public domain.
02 · Predict
Before anything runs
After the prompt `the `, how much probability will the single most likely next character hold?
Choose something first.
03 · Try it
Steer it to the target
Each target came out of this model at the temperature recorded in the file under a recorded seed, so every character of it is inside the top twelve at its own step. Leaving the target is allowed: the matched count drops and nothing is blocked.
One step at a time
computed in this browserweights trained offline by the course scriptLoading the bundled model.
Every chatbot you have used is this loop: produce the odds, pick one piece, put it back on the end of the input, produce the odds again.
What a language model does
computed in this browserone forward pass, weights trained offline by the course scriptThe text so far sits in a slot.
Five seeds at temperature 1
Five seeds, one prompt
computed in this browsertemperature 1No continuations drawn yet.
Why they differ
Call the reason before you submit.
04 · Compare
Submit the run and the verdict lands here, against the position you took above.
05 · Check your understanding
Your check: an outcome you produce
not yetIn one submission: the produced continuation matches the target string exactly, every appended character was selected from the distribution list rather than sampled, and the multiple-choice reason is correct.
Not met yet. Keep working the session.
Where you are
- Target matched: not yet.
- Sampled characters in the run: 0.
- The reason: not called yet.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question about your own trace: available after 3 evaluated runs.
2. The list is never the problem: available after 5 evaluated runs.
3. A different example, computed in full: available after 7 evaluated runs.
4. The full pick sequence: available after 9 evaluated runs.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.
9.5 sorts your own trace beside samples from two other sizes, and 10.3 lets the decoding slot cite it.