Learn machine learning · deep dive D7 · optional

The LSTM's gradient path

The cell is added to rather than recomputed. That one design choice puts a single number between step t and step t minus one, and this dive is the derivation of what that number costs.

Deep dive D7DerivationAbout 45 minutesYour check: demonstrate the idea yourself

This is optional. It opens once 7.4 LSTM gates is done, and nothing in the track waits on it. The gate is four numbers within ten percent, one reading chosen correctly, and both gradient checks under 1e-6.

01 · The question

7.3 measured a recurrent path losing seven orders of magnitude across 30 steps. 7.4 set the gates by hand and held a value for 40. This dive is the arithmetic between those two sessions: why one path fades and the other does not, written out rather than asserted.

The cell update is new cell = forget x old cell + input x candidate. The old cell is carried, not rebuilt, so only one thing stands between the new cell and the old one. Everything else here follows from that.

Commit first, on 7.4's hold test: a forget gate of 0.98 across 40 steps.

02 · Predict

Before anything runs

What fraction of the gradient is left after 40 steps at a forget gate of 0.98?

Fraction left after 40 steps0.00 to 1.00

One line on why, if you want one

03 · Try it

One factor, not two

Differentiate the cell update with respect to the old cell and everything but the forget gate falls away: the input and candidate terms do not touch the old cell. So the factor connecting one step to the previous one on that path is the gate value itself. A plain recurrent path has no such term. Its state is recomputed every step, so its factor is the recurrent weight multiplied by the derivative of the activation, and both of those are smaller than one for most of the range.

The factor that compounds

computed in this browserforget 0.98

Forget gate0.50 to 1.00

The cell path's factor from one step to the previous one is this value, on its own.

The recurrent path is fixed at the 7.3 setting: weight 0.9 times a tanh slope of 0.7, which is a factor of 0.63 per step. The comparison is against that number, not against 0.9, because the activation derivative is part of the factor.

The running product on the cell path and on the recurrent path at three chain lengths
steps	cell path	recurrent path
10	8.1707e-1	9.8493e-3
30	5.4548e-1	9.5547e-7
50	3.6417e-1	9.2689e-11

At a forget gate of 0.98, the cell path keeps 5.4548e-1 of the gradient across 30 steps. The recurrent path keeps 9.5547e-7 over the same 30 steps. Both lines are drawn as log10 of the running product, so a straight line is a constant factor per step.

Both paths, drawn as log10 of the running product over 50 steps.

Show the products

Values shown in the figure above
steps	cell path	recurrent path
10	8.1707e-1	9.8493e-3
30	5.4548e-1	9.5547e-7
50	3.6417e-1	9.2689e-11

Four numbers

Ten percent either way is inside the tolerance, so an exponent in the right place passes. Scientific notation is accepted: 2.2e-7 reads the same as 0.00000022.

Forget gate 1.00, the product across 30 steps

ten percent of the value

Attempts: 0

Forget gate 0.60, the product across 30 steps

ten percent of the value

Attempts: 0

Forget gate 0.98, the product across 40 steps

ten percent of the value

Attempts: 0

A plain recurrent path, weight 0.9 and tanh slope 0.7, across 30 steps

ten percent of the value

Attempts: 0

7.4 held 7 units for 40 steps at a gate of 0.98 and read 3.1199 at the end. That is the same product: 7 times 0.44570. A gate below one is a leak, and a gate of 0.60 reintroduces exactly the problem the cell path solves.

The reading

One reading to choose

Why does the additive path help?

Its factor is a single gate value rather than a weight times an activation derivative, so setting it near 1 costs nothing elsewhere.The additive path removes vanishing gradients from the network.The gates make the recurrent weights larger, so the product shrinks more slowly.

Choose one before locking it in.

The check

A derivation is worth what it survives. The engine differentiates the cell update, then the same slope is measured numerically and the two are compared. The relative error is printed rather than hidden.

The derivation, checked against the engine

computed in this browsercentral differences, eps 1e-5

One cell update, then the same update ten times over with the gate reused at every step. The check compares the gradient the engine derives with the slope measured by moving each input by 1e-5 either way. The ceiling is 1e-6 relative error.

Ask for a hint

Rungs open as you run the session, not as time passes. You have run it 0 times.

1. A question: available after 3 evaluated runs.

2. The answer: available after 5 evaluated runs.

04 · Compare

05 · Check your understanding

Your check: an outcome you produce

not yet

Four numbers within ten percent, the right reading of why the additive path helps, and both gradient checks under 1e-6 relative error.

Not met yet. Keep working the session.

0 of 4 numbers checked. No reading locked in yet. The gradient checks have not run yet.

Save your takeaway

Your takeaway

Edit it into your own words, or keep the line as it is.

back to the track

7.4 LSTM gates

Open the notebook