Learn machine learning · part 1 · session 3
Train, validate, test
Three splits, three jobs, and a button on this page that shows you the test score whenever you want it. Every press is counted.
After this session you will be able to keep a test set clean, and say exactly where a protocol stopped being honest.
Four protocols, four calls, all four correct. Each one tests a different rule.
01 · The question
Three splits, not two. The fit the model. The choose between models you have already trained. The are spent once, at the end, to report a number you did not tune towards. There is a button on this page that shows you the test score whenever you want it. It works, it is honest, and every press is counted and carried in your notebook. I left it there because the temptation is the lesson. A test set stops being a test set the moment it starts choosing anything.
A protocol is contaminated if the test number changed any decision: a choice of depth, of columns, of anything. Look for the moment the test score was allowed to influence something.
02 · Predict
Before anything runs
You will tune a depth on the validation rows, then look at test once. Which will be higher?
Choose something first.
03 · Try it
Three piles, three jobs
computed in this browserthe split you set aboveOne stack: 2,000 rows, all alike.
Move the split
computed in this browserseed 7Everything not held back trains the model.
The rest becomes the test set, spent once at the end.
| Split | Rows | Standard error near 0.80 | Its job |
|---|---|---|---|
| Training | 1,200 | 0.012 | Fits the model. |
| Validation | 400 | 0.020 | Chooses between models you have already trained. |
| Test | 400 | 0.020 | Spent once, at the end, on a number you did not tune towards. |
Synthetic, seed 7. Row counts update live with the sliders.
Training 1,200 rows, validation 400, test 400. At an accuracy near 0.80, 400 rows give a standard error of 0.020.
Because these are only 400 rows, the score wobbles. The usual measure of that wobble is the standard error: take the accuracy p, multiply by (1 minus p), divide by the number of rows n, take the square root. At p = 0.80 and n = 400 that is the square root of 0.8 times 0.2 over 400.
To three decimals, within 0.002.
Attempts: 0
Depth is a dial on the model, the way an oven has a temperature dial. You build the model behind it in Part 2; today it is just a dial with ten settings, and the only question is which rows you are allowed to use to pick a setting.
Tune a depth on the validation rows
computed in this browserseed 7Nothing breaks when you press it. The cost is that the number stops being an honest estimate of new rows, because you now know it while you are still choosing.
Peeks so far: 0. Shown on your notebook page and on the lab report.
Nothing fitted yet. Choose a depth and press "Fit and score on validation" to start the log.
Four protocols, one line each
For each one, ask a single question: did the test number change any decision? Write the answer in your own words. These lines are stored with your notebook and are not marked. The calls are made at the gate below.
P1
Split once. Choose depth on validation. Report test once at the end.
P2
Fit forty depths. Report the best test score.
P3
Split, train, look at test, change the feature set, retrain, report test.
P4
Split, tune on validation, find test is 0.004 lower, report both numbers.
0 of 4 lines written.
04 · Compare
The verdict appears once you have fitted at least one depth on the validation rows.
05 · Check your understanding
Your check: predict results on new examples
not yetCall all four protocols, clean or contaminated, and get all four right. Each one tests a distinct rule, which is why this gate does not settle for two of three.
Variant 1 of 4
P1. Split once. Choose depth on validation. Report test once at the end.
Call it before you run it.
Variant 2 of 4
P2. Fit forty depths. Report the best test score.
Call it before you run it.
Variant 3 of 4
P3. Split, train, look at test, change the feature set, retrain, report test.
Call it before you run it.
Variant 4 of 4
P4. Split, tune on validation, find test is 0.004 lower, report both numbers.
Call it before you run it.
Called 0 of 4 variants so far.
Save your takeaway
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on a different example: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
Your takeaway
Edit it into your own words, or keep the line as it is.