Learn machine learning · part 7 · session 3

Vanishing gradients

To learn, the network has to trace blame backwards from the end of the sequence to the beginning. Each step it passes through multiplies that blame by one factor, and those factors are usually below one.

After this session you will be able to explain in one multiplication why a plain recurrent network stops learning from distant inputs.

grad.vanishSession 7.3About 35 minutesYour check: demonstrate the idea yourself

The gate: the hand computation accepted within ten percent, and the right consequence chosen, in one session.

01 · The question

To learn, the network has to trace blame backwards from the end of the to the beginning. Each step it passes through multiplies that blame by one factor, and those factors are usually below one. Multiply thirty numbers around 0.6 together and see what is left. That is the whole of the problem, and it is arithmetic you can do on paper in a minute.

Below there is a factor slider, a length slider, and then the same measurement on the real network from the last session. Thirty five minutes. The gate asks for one number and one consequence.

02 · Predict

Before anything runs

Thirty factors of 0.6 multiplied together. Roughly what is left?

About 0.02.About 0.001.About 1e-5.About 1e-7.

One line on why, if you want one

Choose something first.

03 · Try it

1 and 2 · the chain toy

Factor per step0.50 to 1.50

Length of the chain1 to 50 steps

Below one the product falls towards zero, and the fall is exponential in the length.

At factor 0.60 the product falls from 6.00e−1 at length 1 to 2.21e−7 at length 30, 6 orders of magnitude.

The table is the artifact here and the plot is the picture beside it. Both are the same numbers.

Show the table at lengths 1, 5, 10, 20, 30, 40 and 50

Values shown in the figure above
length	product
1	6.0000e−1
5	7.7760e−2
10	6.0466e−3
20	3.6562e−5
30	2.2107e−7
40	1.3367e−9
50	8.0828e−12

One bar per step, on the same log scale

illustrativea toy chain, nothing is trained

30 bars, one per step. The first is 6.00e−1 and the last is 2.21e−7, 6 orders of magnitude apart, drawn on a log scale.

Show the running product step by step

Values shown in the figure above
step	running product
1	6.0000e−1
2	3.6000e−1
3	2.1600e−1
4	1.2960e−1
5	7.7760e−2
6	4.6656e−2
7	2.7994e−2
8	1.6796e−2
9	1.0078e−2
10	6.0466e−3
11	3.6280e−3
12	2.1768e−3
13	1.3061e−3
14	7.8364e−4
15	4.7018e−4
16	2.8211e−4
17	1.6927e−4
18	1.0156e−4
19	6.0936e−5
20	3.6562e−5
21	2.1937e−5
22	1.3162e−5
23	7.8973e−6
24	4.7384e−6
25	2.8430e−6
26	1.7058e−6
27	1.0235e−6
28	6.1409e−7
29	3.6846e−7
30	2.2107e−7

3 · the real measurement

The message that does not arrive

illustrativea chain of factor 0.6, the toy of this session

Step 1 of 5

Step 30 has a learning signal to send back.

The learning signal fades before it reaches the start of the sequence.

The same measurement on the recurrent network of 7.2, at lengths 5, 10, 20, 30, 50: the size of the learning signal that still reaches the very first character.

computed in this browsercomputed from the recorded 1.3k network of 7.2

Gradient clipping is off here. Clipping would cap the number and the raw size is the point.

The recorded weights are still loading.

Text the network was trained on: The Wonderful Wizard of Oz, L. Frank Baum (1856-1919), 1900. Public domain.

4 · the hand computation

0.6 to the power 30

within ten percent. Scientific notation is accepted: 2.2e-7 reads the same as 0.00000022.

Attempts: 0

5 · choose the consequence

A gradient of that size reaches the weights at step 1. What follows? The weights at step 1 get an update about ten million times smaller than those at step 30, so they stop changing.The network cannot represent a long sequence at all, so the early characters are lost from the state as well as from the update.The loss cannot fall below 1e-7, because a gradient that size puts a floor under how well the network can ever score.The network needs more hidden units before step 1 can be learned at all, because sixteen numbers cannot hold thirty characters.

Ask for a hint

Rungs open as you run the session, not as time passes. You have run it 0 times.

1. A question: available after 3 evaluated runs.

2. A narrowing: available after 5 evaluated runs.

3. A worked fragment, on another base: available after 7 evaluated runs.

4. The answer: available after 9 evaluated runs.

04 · Compare

05 · Check your understanding

Your check: an outcome you produce

not yet

Both parts in one session: the hand computation accepted within ten percent relative tolerance, and consequence (a) chosen. A value accepted on the third attempt still passes, and the session is then marked done with the answer shown.

Not met yet. Keep working the session.

Hand computation: not yet. Consequence: not chosen yet.

Save your takeaway

Your takeaway

Edit it into your own words, or keep the line as it is.

Nothing goes on the shelf here. The chain is arithmetic and the network is already in the 7.2 recording.

next up

7.4 LSTM gates Play now

Open the notebook