Learn machine learning · part 3 · session 4
Forest versus tree
Every number in this part so far came from seed 7. This session runs all three of your models across twenty seeds and asks one narrow question: is the forest's margin over the pruned tree larger than the wobble between seeds, or is it the wobble.
After this session you will be able to tell a real margin from seed-to-seed noise by reading a mean difference against its own standard error.
The gate is three unseen comparisons. Call each one larger than noise or inside it before the ratio opens, and two of three passes.
01 · The question
Every number in this part so far came from 7. That is one draw of the data, and it is not enough to say anything. This session runs all three of your models across twenty seeds and shows you twenty results each instead of one.
Noise here means how much the answer moves just from re-shuffling the data, before any model is any better than another.
The question is narrow and it is the question underneath most model comparisons you will read: is the forest's margin over the pruned tree larger than the wobble between seeds, or is it the wobble. You will answer it with a mean, a spread and a division, and nothing more.
Larger than noise, or inside it
illustrativea seeded scatter, not measured seedsThe three models
recorded runrecord-runs.mjs- The hand rule from 1.1, at the author's default: refuse if has_collateral = no and debt_ratio > 0.45, then refuse if income_band = low and years_employed < 3.
- The pruned tree from 2.4, at the author's default: depth 5, minimum 20 rows per leaf.
- The forest from 3.2, at the configuration that session settled on.
The recording used those defaults. If your own shelf holds different ones, the numbers below are still the author's; the rerun control under the plot fits the same comparison in this browser so you can watch it happen, and the gate never depends on it.
02 · Predict
Before anything runs
Across twenty seeds, how much held-out accuracy does the forest gain over the pruned tree?
A margin, then a call about it.
Make the second call before you record.
03 · Try it
Loading the recorded run.
Run the same comparison in this browser
computed in this browserseeds 1 to 20synthetic2,000 rows per seedOptional. About nine seconds of fitting, one seed per animation frame, and you can stop it at any seed. It never changes the gate.
Not run in this browser yet.
To four decimals, within 0.0005.
Attempts: 0
Spread that out: the standard error is the spread of the five differences divided by the square root of how many there are. It is how much the average itself would wobble if you drew five more seeds.
To four decimals, within 0.0005.
Attempts: 0
0.020 / 0.0071 = 2.83. The rule for the rest of this session: call a margin larger than noise when the mean difference is more than 2 standard errors from zero. It is a rough reading of twenty paired runs, not a hypothesis test, and it says nothing about probability or confidence.
04 · Compare
The verdict
Reveal all twenty seeds, then read the verdict against what you committed to.
Waiting on a recorded position and all twenty seeds.
05 · Check your understanding
recorded runrecord-runs.mjs · seeds 21 to 40synthetic2,000 rows per seed
Loading the recorded comparisons.
Check a comparison yourself
Each reveal above is a recorded run. This refits the same comparison here, seed by seed, so you can open it rather than take it.
The same comparison, fitted in this browser
computed in this browserseeds 21 to 40synthetic2,000 rows per seedAbout nine seconds, one seed per animation frame, stoppable at any seed.
Not run in this browser yet.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment on a different example: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.
Reveal all twenty seeds to have a baseline to put on the shelf. 5.6 needs it.
Across twenty seeds on this synthetic data the forest's held-out mean sits above the pruned tree's. That is a sentence about these twenty runs and this generator, and it extends to no real dataset and no named benchmark.