Learn machine learning · part 9 · session 1
Tokenisation, and what a token costs
A model never sees letters. Before anything runs, the text is chopped into pieces, tokens, and each piece is swapped for a number. The pieces are not letters and not words: they are whatever chunks turned out to be worth keeping, and this page builds them.
After this session you will be able to train a byte-pair tokeniser and explain why a token count is not a letter count.
To pass: all three predicted token counts inside the stated tolerance in one submission, and the right call on which failure mode the tokeniser causes.
01 · The question
A is a fixed table and a procedure for cutting text into pieces, and on this page you build one from a short paragraph of the course corpus. Start at zero merges, where every character is its own , then raise the count and watch pairs join into fragments and fragments into common words. Each join is one . After that, tokenise your own name, an emoji, a long number and a line of Malay or Chinese, and count what each one costs. The model in the next session receives the numbers this produces, one per token.
The Wonderful Wizard of Oz, L. Frank Baum (1856-1919), 1900. Public domain.
Three things this part cannot run, and why
Three things in this part are out of reach in a browser, and I would rather say so once here than hedge on every page.
Pretraining a large language model is out of reach: it needs a cluster, weeks of compute and a corpus many orders of magnitude larger than the one on this page.
Preference training is out of reach: it needs people ranking whole answers against each other, so this part describes that stage and does not run it.
Claims that new capabilities appear suddenly at scale are out of reach as evidence, because nothing here is large enough to test one, so this part makes none.
The model bundled with these pages has about 112,000 parameters. It is a language model, not a small large language model.
Production systems are several orders of magnitude larger, and further ahead again in training tokens. I have not put a figure on that here, because a number about someone else's system belongs beside a primary source and a date, and this page has neither.
02 · Predict
Before anything runs
The word strawberry is about to be chopped into pieces. How many pieces do you expect? (One means the whole word survives as a single piece.)
Choose something first.
03 · Try it
Text gets chopped before the model sees it
trained in this browser300 merges, trained in this browserThe word strawberry as 10 separate letters.
Turn the merge count up
The merges are trained on the bundled paragraph in this browser. Moving the control truncates a list that is already trained rather than retraining from zero.
Merge count
trained in this browser3,992 charactersTraining the merges on the bundled paragraph.
Read the table in order
The merge table, most recent first
trained in this browser␣ is one space and \n is one line break. The number on the right is how many times that pair occurred.
At zero merges there is nothing in the table: every character is its own token.
Counts include overlapping occurrences. When two pairs tie, the merge takes the pair with the lower symbol ids, so base characters join before symbols an earlier merge created.
No merge has run yet: at zero merges every character is its own token.
Watch the merges happen
This figure computes nothing. It replays the merge list the control above already produced, one merge at a time, over the first stretch of the paragraph.
Watch the merges happen
illustrativeFrame 0 of 0: every character is its own token, before any merge runs.
The merges this page computed, drawn one at a time. Nothing is trained by the drawing.
Tokenise something of your own
10 of 8,192 characters.
Your text, cut into tokens
Nothing tokenised yet.
Three predictions
Predict three token counts
trained in this browserseed 7These three strings were drawn from a fixed pool of twelve by this session's seed. Predict each count at 300 merges, then check all three at once.
seed 7, the fixed number that chose these three strings, so everyone who opens this page gets the same three.
Within 1 token.
Within 1 token.
Within 1 token.
One failure mode
04 · Compare
Fill the three predictions and press Check. The verdict lands here, against the position you took above.
05 · Check your understanding
Your check: an outcome you produce
not yetIn one submission: all three predicted token counts inside their stated tolerance, and the correct failure mode. The tolerance is one token for a string under twelve tokens and two above that.
Not met yet. Keep working the session.
Where you are
- Counts inside tolerance: 0 of 0.
- The failure mode: not called yet.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question about the extremes: available after 3 evaluated runs.
2. Two of your three strings are not like the third: available after 5 evaluated runs.
3. A different example, computed in full: available after 7 evaluated runs.
4. The three counts, with their strips: available after 9 evaluated runs.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.
9.6 counts the tokens in a pasted document against a budget with this tokeniser, and 10.3 offers it as a feature for short-text classification.