Learn machine learning · part 7 · session 4
LSTM gates
The network gets a memory tank. Instead of being rebuilt from scratch each step, the tank keeps what it holds and new value is added into it. Three valves control the tank: one on what stays, one on what goes in, one on what comes out.
After this session you will be able to set an LSTM's three gates to hold one value across many steps and say what each gate did to it.
The gate: one run in which the output at the last step matches the third input, with the forget, input and output gates set as the task requires.
01 · The question
The network gets a memory tank. Instead of being rebuilt from scratch each step, the tank keeps what it holds and new value is added into it. Three valves control the tank: one on what stays, one on what goes in, one on what comes out.
The tank has a name: it is the , and each valve is a . Below, those three valves are yours, one setting per step. The task is to carry the third input to the end untouched. Forty minutes. The gate is a behaviour: set the valves so the final output matches the third input exactly.
The tank and its three valves
computed in this browserthe passing gate program of this sessionStep 2. The input valve is shut, so 2 stays outside and the tank is still empty.
Read the update
new cell = forget × old cell + input × the new value on offer
output = output gate × new cell
The new value on offer is called the candidate, and here it is simply the value on offer this step. Real LSTMs squash the output; here it passes through unchanged so the arithmetic stays readable.
In a trained LSTM every gate value is a sigmoid of a weighted sum, which is why real gate values lie between 0 and 1 and are rarely exactly 0 or 1.
The cell is an additive accumulator. The three gates are scaling factors on it, and here you are setting them by hand.
02 · Predict
Before anything runs
To hold one value unchanged for many steps, what must the forget gate be?
03 · Try it
1 · the task
The input sequence is 5, 2, 7, 1, 9, 4. Carry the value at position 3, 7, to step 6 and let it out there.
2 and 3 · set the gates, then run
illustrativelive arithmetic, nothing is trainedThe input sequence is fixed: 5, 2, 7, 1, 9, 4. The value at position 3 is 7, and the target output at step 6 is 7.000.
Step 1, input 5
Step 2, input 2
Step 3, input 7
Step 4, input 1
Step 5, input 9
Step 6, input 4
4 · the hold test
illustrativelive arithmetic, nothing is trainedForty more steps with no further input, starting from a cell of 7.000 at step 6.
5 · the link back to 7.3
Set every forget gate to 0.6 and the cell runs 7, 4.200, 2.520, 1.512. That is the same factor 7.3 asked you to multiply thirty times, where it left about 1.5e-6 of the original value.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on another position: available after 7 evaluated runs.
4. The answer, as a table: available after 9 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: an outcome you produce
not yetOne run satisfying all four conditions. The output at step 6 equals 7.000 within 0.001; the forget gate is 1.00 at steps 4, 5 and 6; the input gate is 0.00 at those same steps; and the output gate is 0.00 at steps 1 to 5.
Not met yet. Keep working the session.
The passing settings, as a table
| step | x | forget | input | the new value on offer | cell | output gate | output |
|---|---|---|---|---|---|---|---|
| 1 | 5 | 1.00 | 0.00 | 5 | 0.000 | 0.00 | 0.000 |
| 2 | 2 | 1.00 | 0.00 | 2 | 0.000 | 0.00 | 0.000 |
| 3 | 7 | 1.00 | 1.00 | 7 | 7.000 | 0.00 | 0.000 |
| 4 | 1 | 1.00 | 0.00 | 1 | 7.000 | 0.00 | 0.000 |
| 5 | 9 | 1.00 | 0.00 | 9 | 7.000 | 0.00 | 0.000 |
| 6 | 4 | 1.00 | 0.00 | 4 | 7.000 | 1.00 | 7.000 |
Opening this is the same as taking rung four of the ladder: the session records as done with the answer shown until you re-solve it from a reset.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.
The gate schedule goes on the shelf once a run passes, so what is stored is a program that works. The hold test above is the same value carried 40 steps further.