Learn machine learning · part 5 · session 4
Backprop by hand
Nine numbers, small enough that every intermediate value fits on the wires. Blame for the loss travels backwards along the same wires the numbers came forward on, and at every junction it is multiplied by one local slope.
After this session you will be able to compute a backpropagation gradient by hand and check it against a numeric slope.
The gate: ∂L/∂v1 accepted on attempt one or two, both sign predictions right, and the five forward values accepted with at most one assisted.
01 · The question
In 4.4 I handed you the slope and you took the step. This session is where the slope comes from. The network is nine numbers, small enough that every intermediate value fits on the wires.
Blame for the loss travels backwards along the same wires the numbers came forward on. At every junction the blame is multiplied by one local slope, how much that junction's output moves when its input moves by a hair. Multiply those together along the path and you have the . Mathematicians call this the , and it is the whole of : multiplication along a path, in reverse. It is the hardest session in the track and it pays for the rest of it. Commit first: will the two hidden weights get gradients of the same sign.
The network we will work on: two inputs, two hidden units, one output (N221)
A 2-2-1 network with sigmoid hidden units and one sigmoid output, x = (1.00, 0.50), y = 1.00 and the loss L = ½(y − ŷ)². Hidden: w11 = 0.50, w12 = −0.40, b1 = 0.10, w21 = 0.30, w22 = 0.80, b2 = −0.20. Output: v1 = 0.60, v2 = −0.50, c = 0.20. No dataset, and nothing here is trained.
02 · Predict
One hidden unit feeds the output through a positive weight (v1 = 0.60), the other through a negative one (v2 = −0.50).
Before anything runs
Will the two hidden weights w11 and w21 be pushed in the same direction or opposite directions?
Choose something first.
03 · Try it
Read ∂L/∂v1 as “how much the loss changes when v1 changes”. It is the same slope you took steps with in 4.4, written the way everyone writes it.
1 · the forward pass, five values in sequence
Backprop reuses every number the forward pass produced, so compute them first. Each value you get right is written onto its wire below and stays there. z2 and h2 are already on the diagram, because you need them for u.
to three decimals, within 0.001
Attempts: 0
to three decimals, within 0.002
Attempts: 0
to three decimals, within 0.001
Attempts: 0
to three decimals, within 0.002
Attempts: 0
to three decimals, within 0.001
Attempts: 0
2 · follow the path
Blame walks back
computed in this browserfixture N221, computed in this browserThe forward pass, left to right: h1 is 0.5987, u is 0.2480, the output is 0.5617 and the loss is 0.0961.
Fixture N221, every value on its wire
computed in this browsermicrograd.tsinputs
x1 = 1.00
x2 = 0.50
y = 1.00
hidden unit 1
z1 = ?
h1 = ?
hidden unit 2
z2 = ?
h2 = ?
output unit
u = ?
ŷ = ?
L = ?
Every parameter here is a button. Choosing one sets the path below to that parameter. The values with a question mark open as you accept them.
The path from the loss back to v1
illustrativethe order of the multiplications3 local slopes on this path. One press, one hop. The chain stays on screen and is never collapsed into its answer.
3 and 4 · the two gradients by hand
within 0.01
Attempts: 0
within 0.002, because 0.01 would accept zero
Attempts: 0
5 · predict two signs
6 · the numeric gradient check
This is the check that the whole chain was right: nudge w11 by a hair, see how much the loss actually moved, and compare that to the number you got by multiplying. If they agree, the path was correct. The nudge is ±1e-5 and the two-point slope is the one from 4.3.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment on a different network, fully computed: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: an outcome you produce
not yetAll three of: ∂L/∂v1 accepted within 0.01 on attempt one or two, both sign predictions correct, and the five forward values accepted with at most one assisted. ∂L/∂w11 is worked but does not gate.
Not met yet. Keep working the session.
- Forward values accepted: 0 of 5, with 0 assisted
- ∂L/∂v1: not accepted yet
- Sign calls: 0 of 2
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.