Learn machine learning · part 4 · session 4
Gradient descent by hand
Measure the slope, take a step against it, measure again. The only choice in the loop is how big a step, and this surface is lopsided on purpose.
After this session you will be able to run gradient descent by hand, read a walk off a contour map, and say what makes a learning rate too big.
To pass: reach a loss at or below 0.01 within 40 steps from (0, 0), and match all four traces to diverged, oscillating, crawling and converged.
01 · The question
Training is not a solve. It is a loop: measure the slope, take a step against it, measure again. That loop is . The only choice in this loop is how big a step. Too small takes longer; too big can send successive steps further from the minimum, with a loss that keeps growing.
The surface here is written out so you can check every step by hand, and it is deliberately lopsided: one direction has four times the curvature of the other. Take three steps with a pen first. Then let it run, and find the at which it breaks.
The surface, in full
L(w1, w2) = (w1 - 3) ** 2 + 4 * (w2 + 1) ** 2
dL/dw1 = 2 * (w1 - 3), dL/dw2 = 8 * (w2 + 1)
Minimum at (3, -1) where L = 0. Start at (0, 0), where L = 13.0000. The two directions have curvature 2 and 8: one direction has four times the curvature of the other, which is why a step size that suits one overshoots the other.
02 · Predict
Before anything runs
With a learning rate of 0.1, how many steps does this take to reach a loss below 0.01?
Name a rate first.
03 · Try it
One step by hand
You measured slopes by hand in 4.3 and you could do it again here. For a curve this simple the textbook shortcut is exact, so I am giving it to you: dL/dw1 means the slope in the w1 direction, and for this surface it equals 2 * (w1 - 3). Check one of them with the probe method if you want to.
The two gradient components at (0, 0), then w1 after one step at a learning rate of 0.1.
To four decimals.
Attempts: 0
To four decimals.
Attempts: 0
To four decimals.
Attempts: 0
The three steps the engine takes at a rate of 0.1, for checking
From (0, 0) at a learning rate of 0.1 the loss goes 13.0000, 5.9200, 3.6928, 2.3596 over three steps.
Show the three rows
| step | w before | gradient | w after | L after |
|---|---|---|---|---|
| 1 | (0.000, 0.000) | (-6.000, 8.000) | (0.600, -0.800) | 5.9200 |
| 2 | (0.600, -0.800) | (-6.000, 8.000) | (1.080, -0.960) | 3.6928 |
| 3 | (1.080, -0.960) | (-4.800, 1.600) | (1.464, -0.992) | 2.3596 |
Walk it on the map
Choose the rate before each step, or let twenty run. You can change the rate between steps. The 3D surface, contour map and loss curve show the same calculation.
Record your prediction in step 02 before taking a step. A wrong guess costs nothing.
Nothing taken yet. The point is at (0, 0) and the loss is 13.0000.
Steps taken: 0. Current loss: 13.0000.
How high is the loss at this step?
The floor is the two weights, w1 and w2. Height is loss: lower is better. The bright point is the selected step; the path connects the actual updates.
Loading the 3D surface. The contour map and numbers below remain available.
Bounds: w1 −2 to 6; w2 −5 to 3. Height is linear, with 12 loss units per weight-axis unit. Camera controls change only the view. Use the shared playback controls below to select a step.
Step 0 of 0 at a learning rate of 0.10. The point is at (0.000, 0.000) and the loss is 13.0000, from 13.0000 at the start.
Each ring is one height on the hill, the way a walking map draws them: every point on a ring costs the same. The rings are squashed because the hill is steeper one way than the other. The cross marks the minimum at (3, -1). w1 runs -2 to 6 left to right and w2 runs -5 to 3 bottom to top.
Show the last twelve steps as numbers
| step | w1 | w2 | dL/dw1 | dL/dw2 | loss |
|---|---|---|---|---|---|
| 0 | 0.000 | 0.000 | -6.000 | 8.000 | 13.0000 |
Find the rate that breaks it
Within 0.005.
Attempts: 0
Four traces, four names
Four fixed runs from (0, 0), drawn together. Read the evidence, then put the four names in trace order: the first name goes with trace 1.
Three step sizes on the same hill
illustrativedescend on the 4.4 surfaceThree walks, the same hill, the same start.
the axis is log base ten of the loss
Trace 1, at a rate of 0.30: loss 13.00 to 9.28 to 15.60 to 30.16 to 59.04 to 115.70, roughly doubling every step after the second; it passes one million at step 19. Trace 2, at a rate of 0.25: w2 alternates 0, -2, 0, -2, 0 and keeps doing so; the loss falls to 4.00 and settles there, never lower. Trace 3, at a rate of 0.01: loss 12.03 after 1 step, 6.76 after 10, 0.1583 after 100; it needs 169 steps to reach 0.01. Trace 4, at a rate of 0.12: loss 0.01241 at step 12 and 0.00717 at step 13, the first below 0.01.
The first 25 steps of each run. A value of 0 on this axis is a loss of 1, and 2 is a loss of 100.
Show the four runs as numbers
| trace | learning rate | loss at step 1 | loss at step 10 | first step at or below 0.01 |
|---|---|---|---|---|
| trace 1 | 0.30 | 9.2800 | 3346.7302 | never inside its run |
| trace 2 | 0.25 | 6.2500 | 4.0000 | never inside its run |
| trace 3 | 0.01 | 12.0292 | 6.7632 | 169 |
| trace 4 | 0.12 | 5.2048 | 0.0372 | 13 |
The four names, in trace order
Focus a name, press Enter to lift it, move it with the arrow keys, then press Enter again.
04 · Compare
Walk the surface and name the four traces, and the verdict lands here.
05 · Check your understanding
Your check: a loss of at most 0.01 on this surface
not yetBoth: a loss at or below 0.01 within 40 steps from (0, 0), and all four traces matched to diverged, oscillating, crawling and converged.
Not met yet. Keep working the session.
Where you are
- First step at or below 0.01: not yet, against a budget of 40.
- Traces matched: 0 of 4.
A constant rate of 0.20 lands in 8 steps and 0.12 in 13. A rate of 0.01 needs 169 and misses. Anything above 0.25 runs off.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on a one-variable surface: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.