Learn machine learning · deep dive D4 · optional
Momentum and Adam, hand-stepped
Two modifications to the update loop, derived from their own rules and stepped by hand on the surface 4.4 used, so nothing about either one has to be taken on trust.
Optional, and it blocks nothing. It opens because 4.4 Gradient descent by hand is done. The gate is six hand computations plus one configuration that reaches a loss of 0.01 in 8 steps or fewer.
01 · brief
The surface is the one from 4.4: L(w1, w2) = (w1 - 3)² + 4(w2 + 1)², starting at (0, 0). Curvature 2 in one direction and 8 in the other, which is the whole reason one learning rate does not suit both.
Momentum keeps a running sum of gradients: v = beta x v + g, then w = w - lr x v. Adam keeps two running averages, one of the gradient and one of its square, corrects both for starting at zero, then divides one by the square root of the other.
The question you cannot answer yet: what makes Adam's first step the same size whether the gradient is 1 or 100.
02 · commit
Before anything runs
Before any stepping: momentum at lr 0.1 with beta 0.9, against plain descent at its best constant rate of 0.20. How long does momentum take to reach a loss of 0.01?
Choose something first.
03 · work
Momentum at lr 0.1, beta 0.9, beside plain descent at the same rate
computed in this browserv is updated first, then the weights: v = beta x v + g, then w = w - lr x v. The surface is the one 4.4 uses, L(w1, w2) = (w1 - 3)² + 4(w2 + 1)², from (0, 0).
Step 0 of 8
| step | v1 | v2 | w1 | w2 | loss | plain loss | difference |
|---|---|---|---|---|---|---|---|
| 0 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 13.0000 | 13.0000 | +0.0000 |
Both optimisers start at w1 0, w2 0 with a loss of 13.0000.
Show the steps taken so far as a table
| step | v1 | v2 | w1 | w2 | loss | plain loss |
|---|---|---|---|---|---|---|
| 0 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 13.0000 | 13.0000 |
To four decimals, within 0.0001.
Attempts: 0
To four decimals, within 0.0001.
Attempts: 0
To four decimals, within 0.0001.
Attempts: 0
To eight decimals or better, within 1e-8, which is Adam's eps. The update is negative.
Attempts: 0
To eight decimals or better, within 1e-8, which is Adam's eps. The update is negative.
Attempts: 0
To eight decimals or better, within 1e-8, which is Adam's eps. The update is negative.
Attempts: 0
0 of 6 computations accepted.
Run a configuration to a loss of 0.01
Every run starts at (0, 0) on the same surface with a budget of 60 steps. The gate asks for one configuration that reaches the target in 8 steps or fewer.
The stability limit on this surface is 0.25, because the curvature in w2 is 8.
A longer memory raises the effective step to about lr / (1 - beta) in a consistent direction.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. The answer: available after 5 evaluated runs.
04 · verdict
Run a configuration in the race above and the verdict states what each one took against what you committed to.
05 · gate
Your check: a step count on the stated surface
not yetAll six hand computations within their tolerances, plus one configuration of momentum or plain descent that reaches a loss of 0.01 in 8 steps or fewer.
Not met yet. Keep working the session.
Two configurations fit: momentum at lr 0.13 with beta 0.3, and plain descent at lr 0.20. Adam at its usual rate does not, on this surface, and a rate that made it fit here would not be its usual rate. Matching plain descent's speed as well as its stability needs a shorter memory, not a smaller step.
carry
Your takeaway
Edit it into your own words, or keep the line as it is.