Learn machine learning · deep dive D3 · optional
Out-of-bag error as free validation
Every bootstrap sample leaves rows behind. Those rows are a validation set the forest gets for nothing, as long as there are enough trees to make it worth reading.
Optional, and it blocks nothing. It opens because 3.2 Bootstrap and feature bagging is done. The gate asks for an out-of-bag accuracy at 50 trees within 0.02 of the held-out figure on the same data, with the five-tree row reported beside it.
01 · brief
Session 3.2 fitted each tree on a bootstrap sample: the training rows drawn again, with replacement, so some rows arrive twice and some never arrive at all. The rows that never arrived are out of bag for that tree, and that tree has never seen them.
So every training row has a private jury: the subset of trees that were fitted without it. Score each row with only those trees and you have an estimate on rows the voting trees never saw, without holding a single row back.
The question you cannot answer yet: how close that free estimate lands to the 400 rows Part 3 actually held back, and how many trees it takes before the answer is worth reading.
02 · commit
Before anything runs
Before anything runs: what will the out-of-bag accuracy be at fifty trees? The held-out figure in 3.2 is around 0.84.
03 · work
To four decimals or better, within 0.0001.
Attempts: 0
To four decimals or better, within 0.0005.
Attempts: 0
0 of 2 computations accepted. Fifty trees times about 0.3678 is 18.4 trees per row.
One hundred trees on the Part 3 rows
syntheticseed 7The same synthetic loan table and the same three-way split Part 3 uses: 0 training rows, the 400 test rows held back, and the forest settings 3.2 settled on.
0 of 100 grown.
Step one: how often is one row left out
Each tree is fitted on a bootstrap sample of 0 rows drawn with replacement, so some rows are missing from it. Pick a training row and count the trees that never saw it.
Drag or type. Every position from 0 to the last training row is available.
Grow the forest first and this reads a real count.
Step two: score every row with only its out-of-bag trees
Each training row is voted on by the trees that never saw it, and the vote is compared with the row's label. No rows are held back for it.
Available once the hundred trees are grown.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. The answer: available after 5 evaluated runs.
04 · verdict
Grow the forest and score the rows, and the verdict states both numbers against what you predicted.
05 · gate
Your check: reach the stated target
not yetAn out-of-bag accuracy at 50 trees within 0.02 of the held-out accuracy on the same data, plus both hand computations.
Not met yet. Keep working the session.
carry
Your takeaway
Edit it into your own words, or keep the line as it is.