Learn machine learning · part 3 · session 2
Bootstrap and feature bagging
A forest has to make its trees disagree on purpose, and it has two independent ways of doing it. Both make each individual tree worse. Two switches, four corners, one of which is fifty identical trees.
After this session you will be able to name a forest's two sources of randomness, say which way each one fails, and read agreement beside accuracy to tell those two failures apart.
The gate is three forests with their settings hidden. Name the misconfigured control from the diagnostics, and two of three passes.
01 · The question
Fifty trees that vote is a . You already know from the last session that voting is worth nothing if the voters copy each other, and trees grown on the same rows with the same rules are exact copies. So a forest has to make its trees disagree on purpose, and it has two independent ways of doing it.
The two ways are: give each tree a different random handful of the rows, and stop each split from looking at every column. The first is called sampling, the second is called feature bagging.
Both of them make each individual tree worse. That is the trade this session is about. Two switches, four corners, one of which is fifty identical trees. Commit to which switch you think does more of the work before you touch either.
Fifty trees, four corners
illustrativeshape stands for varietyThe data on this page
syntheticseed 7, 2,000 rowsGenerated in this browser from seed 7. Not calibrated to any real lending data. 2,000 rows, nine columns, split 60/20/20 into 1,200 training rows, the 400 validation rows every number in this part is measured on, and 400 test rows nothing in this part touches. Every forest here is 50 trees at depth 8 with a minimum of 5 rows per leaf, grown on the main thread a few trees per animation frame.
02 · Predict
Before anything runs
Which control does more for held-out accuracy on this data? (held-out means the 400 rows no tree was trained on)
Choose something first.
03 · Try it
One tree, for reference
syntheticseed 72,000 synthetic loan applications, generated in this browser from seed 7 and not calibrated to any real lending data. 1,200 rows to train on, 400 held out. This is the same tree the forest is built from: depth 8, minimum 5 rows per leaf.
No reference tree yet.
The two randomness controls
computed in this browser50 trees, seed 7How many of the 9 columns a split may look at. At 9 it looks at all of them.
No forest grown yet.
The four corners
computed in this browserAgreement is the mean fraction of held-out rows on which two trees give the same label. It is a fraction of matching predictions, not a correlation coefficient.
| corner | bootstrap | max_features | held-out | agreement | mean single tree | distinct trees |
|---|---|---|---|---|---|---|
| neither control | off | 9 | not run yet | not run yet | not run yet | not run yet |
| bootstrap only | on | 9 | not run yet | not run yet | not run yet | not run yet |
| max_features only | off | 3 | not run yet | not run yet | not run yet | not run yet |
| both controls | on | 3 | not run yet | not run yet | not run yet | not run yet |
No forest grown yet.
The four settings, each grown in this browser on the same 1,200 training rows.
The vote filling up
computed in this browser50 trees, seed 7The trees are already grown and the arithmetic is already done. This paces it: each tree joins the vote in turn, and the three readouts move as it does.
Grow a forest to watch the vote fill up.
Drawing with replacement
illustrativemulberry32, seed 7Ten rows in the jar, and an empty tray.
To three decimals, within 0.005.
Attempts: 0
That is 0.3678, and it converges to 1/e for any large n. Write it as about 37 percent of rows left out of each sample: it is a limit, not a measurement. With 50 trees, about 18.4 of them never saw a given row. Deep dive D3 turns that into free validation.
max_features, 1 to 9
computed in this browserbootstrap on, 50 treesRun the sweep to draw the two lines.
No sweep yet.
The solid line is the forest's vote. The dashed line is the mean accuracy of the trees inside it.
04 · Compare
The verdict
Run all four corners, then read the verdict against what you committed to.
Waiting on a recorded position and all four corners.
05 · Check your understanding
The gate: three forests with hidden settings
computed in this browserEach one is a real 50-tree forest on the same rows, with one of the two controls set wrong. You see the diagnostics and call the fault before the settings open.
0 of 3 ready.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment on a different example: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.
Run the corner with both controls on to have something to put on the shelf. 3.3 needs it.