Learn machine learning · deep dive D10 · optional
The tokeniser and your language
A tokeniser is not neutral about languages. It is trained on a text, and every script that text does not contain pays for its absence by the byte.
This is optional. It opens once 9.1 Tokenisation is done, and nothing in the track waits on it. The gate is two calls made before the merge source is switched: which line becomes cheapest, and which becomes dearest.
01 · The question
9.1 trained byte-pair merges on an English paragraph and watched the token count fall. The same merge list meets a sentence in a script it has never seen here, and the arithmetic that follows is not an opinion about languages, it is UTF-8.
One sentence, the same meaning three times: Malay in the Latin alphabet, the same Malay in the Jawi script, and Chinese. Two merge sources, both trained in this browser at 300 merges. The table prices all three under whichever source is selected.
Commit first: under merges trained on English prose, which of the three costs the fewest tokens per character?
02 · Predict
Before anything runs
Under merges trained on the English paragraph, which line costs the fewest tokens per character?
Choose something first.
03 · Try it
Three sentences, one price list
Tokens per character is the measure, because the three lines are not the same length in characters. A dashed token is one the fallback produced: the merges never saw that character, so each of its UTF-8 bytes becomes a token of its own.
One sentence, three scripts
computed in this browserTraining the merges in this browser.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. The answer: available after 5 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: predict results on new examples
not yetCall both before the switch: which line becomes cheapest under the Chinese merges, and which becomes dearest. Each call is locked in before the numbers are shown.
Called 0 of 0 variants so far.
The calls go in before the switch.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.