Learn machine learning · part 5 · session 2
Hidden layers build features
In 4.6 the line failed on XOR until you added the product feature by hand. Here the hidden units draw their own lines, and the output unit adds and subtracts what they report.
After this session you will be able to solve XOR with two hidden units and say what each unit contributes.
The gate: held-out accuracy of at least 0.95 with the control at exactly two hidden units, and, for each of the two hidden units, picking the description of which corners it is positive at.
01 · The question
In 4.6 the line failed on XOR until you added the product feature by hand. The question left open was where that feature comes from when nobody hands it to you. Here you add hidden units and the training finds them.
Each hidden unit in a is a line of its own drawn in the same space, and the output unit is a weighted sum of what those lines report. Start with one hidden unit and watch it fail, then add the second. Before you run anything, commit to the smallest number of hidden units that reaches 0.95 on held-out points.
The data
syntheticxor(600, seed 7, sigma 0.18)Four gaussian clusters at (±1, ±1). The two clusters where the coordinates have opposite signs are class 1. 400 rows to train on, 200 held out, and the class balance is exactly even, so guessing one class scores 0.50.
02 · Predict
A hidden unit is one line drawn in the same picture as the data. Ask yourself: how many straight lines does it take to fence off the two diagonal corners?
Before anything runs
The smallest hidden-unit count that reaches 0.95 held out is:
03 · Try it
1 to 4 · run it, read the units, watch the boundary form
The run
trained in this browserxor(600, seed 7, sigma 0.18) · init seed 7Each hidden unit squashes its total with tanh, an S-curve like the sigmoid from 4.2 but running from −1 to +1 instead of 0 to 1. One sigmoid output. Adam at 0.05, 400 steps, every training row in each step.
400 training rows, 200 held out, both from the same 600 points. The seed is printed above and never hidden.
Two lines make a band
illustrativethe hand-set pair from this session's answerFour clusters. No straight line puts both class 1 clusters on one side by itself.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment on the band construction: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: reach the stated target
not yetHeld-out accuracy of at least 0.95 with the control at exactly two hidden units, and, for each of the two hidden units, the description of which corners it is positive at picked correctly.
Not met yet. Keep working the session.
- Hidden units on the last run: 0
- Held-out accuracy: no run yet
- Descriptions: not checked yet
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.
The shelf takes this network once a two-unit run has finished.