Learn machine learning · deep dive D5 · optional
Every weight of the 2-2-1 network, then the matrix form
Nine parameters, nine routes from the loss, and one arithmetic you already know. Finish all of them, then write the same thing in four lines of matrix algebra and watch the numbers come out the same.
Optional, and it blocks nothing. It opens because 5.4 Backprop by hand is done. The gate is 7 of the nine gradients by hand plus all four matrix forms.
01 · brief
Session 5.4 walked two of these nine routes and asserted the rest. This finishes them. The fixture is unchanged: a 2-2-1 network with sigmoid everywhere and the squared-error loss, at one input and one target.
The routes overlap heavily. Once the error term at the output is computed, the three output-layer gradients differ only in what multiplies it at the final hop; once the error term at each hidden unit is computed, the same is true there. That overlap is what the matrix form is: not a new derivation, the same one with the shared work written once.
The question you cannot answer yet: what shape each of those four expressions has to have for the arithmetic to be legal at all.
02 · commit
Before anything runs
Before computing anything: which of the nine gradients is the largest in magnitude?
Choose something first.
03 · work
Fixture N221, forward pass
computed in this browser2-2-1, sigmoid hidden, sigmoid outputInputs x1 1, x2 0.5, target y 1, loss L = ½(yhat - y)². The nine parameters start at w11 0.5, w12 -0.4, b1 0.1, w21 0.3, w22 0.8, b2 -0.2, v1 0.6, v2 -0.5, c 0.2.
| z1 | h1 | z2 | h2 | u | yhat | L |
|---|---|---|---|---|---|---|
| 0.400000 | 0.598688 | 0.500000 | 0.622459 | 0.247983 | 0.561680 | 0.096062 |
0 of 9 accepted.
Within 0.01. Signs matter.
Attempts: 0
Within 0.01. Signs matter.
Attempts: 0
Within 0.01. Signs matter.
Attempts: 0
Within 0.002. Signs matter.
Attempts: 0
Within 0.002. Signs matter.
Attempts: 0
Within 0.002. Signs matter.
Attempts: 0
Within 0.002. Signs matter.
Attempts: 0
Within 0.002. Signs matter.
Attempts: 0
Within 0.002. Signs matter.
Attempts: 0
The same nine numbers as four matrix expressions
x is 2 x 1, h is 2 x 1, W1 is 2 x 2, V is 1 x 2, yhat is 1 x 1. Pick the expression whose shape is legal in each slot. The symbol ⊙ is elementwise multiplication.
Pick one expression in each slot first.
One gradient check over all nine parameters
Each parameter is nudged up and down by 0.00001 and the loss is measured either side. The slope that gives is compared with the one the backward pass produced.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. The answer: available after 5 evaluated runs.
04 · verdict
Accept a gradient or check the four shapes, and the verdict states where you are.
05 · gate
Your check: an outcome you produce
not yet7 of the nine gradients accepted by hand, and all four matrix expressions assembled with a legal shape.
Not met yet. Keep working the session.
The gradient check is the third leg and it is not part of the gate: a check that passes proves the tape and the algebra agree, not that you can do either. It should report a largest relative error under 1e-6.
carry
Your takeaway
Edit it into your own words, or keep the line as it is.