Learn machine learning · part 7 · session 2
Hidden state
This network reads text one character at a time. After each character it updates a small scratchpad of sixteen numbers, then throws the character away. The scratchpad is all it has left.
After this session you will be able to read a recurrent network's sixteen-number state and say which part of the input is driving it.
The gate: three pairs of prefixes, called before they are run, two of three right.
01 · The question
This network reads text one character at a time. After each character it updates a small scratchpad of sixteen numbers, then throws the character away. The scratchpad is all it has left, and it is the same size after two characters as after two hundred.
The scratchpad has a name: it is the network's , and a network that reads a this way, one step at a time with the same weights every step, is called .
Below you can step a small network through text one character at a time and watch those sixteen bars move. Then I give you two pairs of prefixes and ask which pair leaves the more similar state. The obvious reading, that similar-looking text leaves similar states, is wrong often enough to be worth forty minutes.
Where the numbers come from
recorded runThe alphabet is 32 symbols: 26 letters, space, full stop, comma, apostrophe, newline and one out-of-vocabulary symbol. A loss in nats per character here is not comparable with one from Parts 8 or 9, which use a different alphabet.
Text: The Wonderful Wizard of Oz, L. Frank Baum (1856-1919), 1900. Public domain.
What was recorded, and what is computed here
One run of an Elman recurrent layer, hidden 16 over a 32-symbol alphabet, 1,328 parameters, trained by scripts/ml/grids/hidden-state.mjs on the first 4096 characters of the corpus. The file carries the loss at every step, four checkpoints, and the weights the run finished with as int8. Everything after the player is computed in this browser from those weights, so editing a string changes the numbers.
02 · Predict
Before anything runs
Two prefixes end with the same eight characters but start differently. Two others start the same but end differently. After reading each one, which pair leaves the network's sixteen numbers closest together?
Choose something first.
03 · Try it
1 · play the training run
Nothing trains on your phone. The weights the next step uses are the ones this run finished with.
Loading the recorded run.
One character in, sixteen numbers out
computed in this browserthe recorded network of 7.2, stepped hereNothing read yet: the sixteen numbers start at zero.
2 · step the network
computed in this browserone forward pass per pressRead 0 of 48 characters. Next press reads "c".
The recorded weights are still loading. Stepping starts as soon as they arrive.
The hidden state, one bar per unit
Nothing has been read yet, so every unit sits at zero.
Show the 16 unit values as a table
| unit | value |
|---|---|
| 1 | 0.000 |
| 2 | 0.000 |
| 3 | 0.000 |
| 4 | 0.000 |
| 5 | 0.000 |
| 6 | 0.000 |
| 7 | 0.000 |
| 8 | 0.000 |
| 9 | 0.000 |
| 10 | 0.000 |
| 11 | 0.000 |
| 12 | 0.000 |
| 13 | 0.000 |
| 14 | 0.000 |
| 15 | 0.000 |
| 16 | 0.000 |
The state is the same size after two characters as after two hundred: 16 numbers, whatever it has read. Largest magnitude so far 0.000.
is one number between −1 and 1 saying how closely two lists of numbers point the same way. You will work one out by hand in a moment.
3 · compare two prefixes
computed in this browsercomputed from the recorded weightsCosine here is a number about two vectors: one dot product divided by two lengths. It is not a measure of how alike the two pieces of text are.
The recency effect is stronger here than a first reading suggests, and the reason is the squashing function. Each update is squashed into the range −1 to 1. After a handful of characters most of the sixteen numbers already sit near the ends of that range, so a new character shifts them more than anything left over from the start. That is why two prefixes sharing their last eight characters can agree to four decimals.
4 · one cosine by hand
The warm-up is two-dimensional so you can do it on paper: a = [3, 4] and b = [4, 3].
to two decimals, within 0.01
Attempts: 0
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on a one-dimensional toy: available after 7 evaluated runs.
4. The answer, with this run's numbers: available after 9 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: predict results on new examples
not yetThree pairs of prefixes. Call each one before you run it, and two of three passes. Each reveal runs the real network on the real strings, and the strings stay editable in the compare tool above so you can probe them.
Variant 1 of 3
Pair one is "the sea was" against "a bright sea was". Pair two is "the sea was" against "the sea had". Which pair leaves the more similar hidden state?
Call it before you run it.
Variant 2 of 3
Pair one is "mother" against "another". Pair two is "mother" against "mothers". Which pair leaves the more similar hidden state?
Call it before you run it.
Variant 3 of 3
Pair one is "the night was long" against "the light was long". Pair two is "the night was long" against "the night was calm". Which pair leaves the more similar hidden state?
Call it before you run it.
Called 0 of 3 variants so far.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.
Nothing goes on the shelf here. The recording already ships these 1,328 weights, so a shelf copy would spend bytes for nothing.