Learn machine learning · part 2 · session 4
Pruning and instability
Same data, same settings, a different draw of the training rows. This session measures how much the tree moves when that draw changes.
After this session you will be able to measure how much a tree moves when its rows are resampled, and hold it steadier with rows per leaf.
The gate is three pairs of configurations: call the more unstable one before it is measured, and two of three passes. Two configurations count as the same when their disagreement differs by less than 0.02.
01 · The question
Same data, same settings, different draw of the training rows. Press reshuffle and the tree is rebuilt from a fresh resample of the same 1,200 rows. Sometimes the root split changes column entirely.
Each resample draws 1,200 rows from the same 1,200, one at a time, putting each back before the next draw, so some applications appear twice and some not at all. It is the closest thing to "if we had collected the data again", and it is the Part 3 builds a forest out of.
That is worth sitting with, because the tree you shipped in the last session looked like a finding. It is one of many trees the data would support, and the deeper it goes the more the picture moves. This session measures that movement: how often two runs disagree about the same held-out application, and how much the held-out score wanders.
Data: synthetic loan applications, seed 7, 1,200 / 400 / 400. Twenty resamples per measurement, drawn with replacement from the training rows; the run seeds are 7,001 to 7,020 and each run prints its own.
syntheticseed 7
02 · Predict
Two trees built from two slightly different draws of the same data will not always give the same answer for the same application. The share of held-out applications they answer differently is what this session measures.
Before anything runs
Two trees, depth 3 and depth 12, fitted on twenty resamples each. Which one's held-out predictions move more between runs?
Choose something first.
03 · Try it
Step 1 · reshuffle the root
Each press draws a fresh resample of the 1,200 training rows and refits at depth 12.
Resamples: 0 of 20
The figure of the two trees fills in once the first two resamples are fitted.
Step 2 · minimum samples per leaf
This dial forbids any leaf smaller than the number you set, so the tree cannot build a rule on two applications. Watch what it does to the leaf count and to the gap, and then, in step 3, to how much the tree moves. Cutting a grown tree back like this is .
The tree is refitted at depth 12 on the full 1,200 training rows as you move this.
Step 3 · measure the two trees you committed to
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on a different example: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
04 · Compare
Measure the two configurations in step 3 and the verdict states what they did.
05 · Check your understanding
Prepare the three pairs
Six configurations, twenty resamples each, measured before you call them. No number from a pair is shown until you have called that pair.
Your check: predict results on new examples
not yetThree pairs of configurations. Call the more unstable one in each before it is revealed; two of three passes. A disagreement difference below 0.02 counts as about the same, and that band is what the third option means.
Called 0 of 0 variants so far.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.