Learn machine learning · part 2 · session 3
Depth and overfitting
One control, two curves. The training curve keeps climbing; the validation curve rises, turns, and falls. The session is about reading the turn.
After this session you will be able to pick a depth from a validation curve, and say why the training curve cannot be used for it.
The gate is the depth you lock: its validation accuracy within 0.005 of the best validation accuracy in the sweep.
01 · The question
One slider this time, from depth 1 to depth 20, and two curves. The training curve goes up and keeps going up; at depth 20 it reaches 1.000, because a deep enough tree can isolate almost every training row. The validation curve rises, turns, and falls. Your job is to read the turn. Scoring well on the rows a tree was fitted to and worse on new ones is , and the gap between the two curves is how you see it.
The test rows stay hidden until you have chosen, and then all three curves are drawn together. Watch the regions while you drag: they get smaller and more numerous, and each one is supported by fewer rows.
Data: synthetic loan applications, seed 7, 1,200 training and 400 validation rows from the 1.3 split plan, all nine non-leaky columns. Every tree here is fitted in your browser while you watch.
syntheticseed 7
02 · Predict
Before anything runs
At which depth will validation accuracy be highest?
03 · Try it
Step 1 · drag the depth
The sweep and the picture follow this control. The scrubber under the figure moves the same value.
The first trees are still being fitted.
Scoring the remaining depths.
No depth has been scored yet. Move the depth control to start.
No depth has been scored yet.
Synthetic, seed 7. 1,200 training rows and 400 validation rows, all nine non-leaky columns.
The figure of the boundary fills in once the twenty trees above are fitted.
Step 2 · read the regions
The regions at depth 1
computed in this browsertwo columnsThis picture is drawn in your browser from the seed, so it appears once the page is running.
Hollow circle: repaid. Filled square: did not repay. A shaded region is one the tree scores as did not repay.
The picture fills in once the first tree is fitted.
The curves above use all nine columns, which cannot be drawn. This picture is the same recipe run on just the two columns you can see, so you can watch the shape of the change even though the numbers differ. Synthetic, seed 7.
Half the leaves hold more rows than the median, half fewer. Watch it fall as you increase depth: a leaf holding two rows is a rule built on two applications.
Step 3 · lock a depth
The test rows are not scored on screen until you lock. Depths scored so far: 0.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on a different example: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
04 · Compare
Lock a depth and the three curves are drawn together.
05 · Check your understanding
Your check: an outcome you produce
not yetThe locked depth's validation accuracy within 0.005 of the best validation accuracy in the sweep, which is printed here once you have locked. 400 validation rows give a standard error near 0.020, so this tolerance is deliberately narrower than the noise and admits only the genuine plateau.
Not met yet. Keep working the session.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.