Learn machine learning · part 6 · session 3
Cross validation
A single validation number is one draw. Five folds give five numbers, and the useful thing is not their average, it is their spread.
After this session you will be able to compare two models fold by fold and say when a difference is inside the noise.
The gate is three calls: for two of the three model pairs, say whether the difference is real or inside the noise.
01 · The question
A single validation number is one draw. Split the same rows a different way and it moves, sometimes by more than the improvement you were celebrating. Five-fold gives five numbers instead of one, and the useful thing is not their average, it is their spread. Noise here means the wobble that comes from which rows happened to land in which slice, not from the models being different.
I trained the five fold models once and shipped the runs, so here you play the folds one at a time and watch the distribution build. Then you compare two models fold by fold, because the same fold is hard for both of them and pairing removes that. Commit first: how much do you expect one model's fold scores to vary from lowest to highest?
02 · Predict
Before anything runs
Range from the lowest to the highest fold score for one model on 320-row folds.
03 · Try it
Play one fold at a time
kFold deals the 1,600 training and validation rows of the 1.3 split into five disjoint folds of 320. The 400 test rows are not touched by any of this.
Rotating which slice is held out
recorded runrecord-runs.mjs · one model per fold · seed 7Fold 1 is the held-out slice; the other four train the model.
Five folds, five models, one score each
syntheticloan seed 7 · 320 rows per foldEach model is trained on the other four folds and scored on its own, so no row is ever in both sides of its own comparison.
The range check opens once all five folds are played.
Pair the folds
Two models, five folds each. Separately, their spreads overlap and nothing is clear. Fold 3 happens to hold harder rows, so both models score low on it. Comparing them fold by fold cancels that out: what is left is the difference between the models, not the difference between the slices. This example is a fixture, so the arithmetic is checkable by hand.
Pair 1, as two strips or as one
illustrativefixture cvPairsModel A, five folds
Model A scores 0.812, 0.836, 0.799, 0.845, 0.823, mean 0.8230.
Show the values
| fold | value |
|---|---|
| fold 1 | 0.812 |
| fold 2 | 0.836 |
| fold 3 | 0.799 |
| fold 4 | 0.845 |
| fold 5 | 0.823 |
Model B, the same five folds
Model B scores 0.841, 0.859, 0.828, 0.866, 0.851, mean 0.8490.
Show the values
| fold | value |
|---|---|
| fold 1 | 0.841 |
| fold 2 | 0.859 |
| fold 3 | 0.828 |
| fold 4 | 0.866 |
| fold 5 | 0.851 |
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on three folds: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: predict results on new examples
not yetThree pairs of five fold scores. For two of the three, say whether the difference is real or inside the noise. Each call is locked in before the paired numbers are revealed.
Variant 1 of 3
Pair 1. Model A scores 0.812, 0.836, 0.799, 0.845, 0.823. Model B scores 0.841, 0.859, 0.828, 0.866, 0.851.
Call it before you run it.
Variant 2 of 3
Pair 2. Model A scores 0.823, 0.847, 0.801, 0.859, 0.818. Model B scores 0.831, 0.839, 0.812, 0.845, 0.829.
Call it before you run it.
Variant 3 of 3
Pair 3. Model A scores 0.804, 0.851, 0.796, 0.862, 0.817. Model B scores 0.829, 0.833, 0.822, 0.841, 0.830.
Call it before you run it.
Called 0 of 3 variants so far.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.