Learn machine learning · part 6 · session 1

The overfitting machine

One network, six training-set sizes, and nothing else moving. You make the gap between training and validation open on purpose, then watch it close again, and you read a stopping point off a validation curve.

After this session you will be able to cause overfitting on purpose and read the stopping point off a validation curve.

fit.overSession 6.1About 30 minutesYour check: demonstrate the idea yourself

The gate is a behaviour: open a cell at 200 rows or fewer whose gap is above 0.20, open the 1,200-row cell with a gap below 0.05, and put the early-stopping marker where validation turns.

01 · The question

In 2.3 you watched a tree fit the training rows perfectly and get worse on rows it had not seen. That is . The same thing happens to the network you trained last session, and here you cause it deliberately by starving it. Nothing about the model changes. Only the number of training rows moves, from fifty up to the twelve hundred the training split holds, with the validation rows held fixed so the comparison stays fair.

I trained the six sizes once and shipped the runs, so the control moves between them and the picture builds as it goes. Watch the two curves separate and then close again. Commit first: at fifty training rows, how far apart will training and validation accuracy end up?

02 · Predict

For reference, 0.00 means the model does exactly as well on new rows as on the rows it studied, and 0.40 would mean it is 40 points worse on new rows.

Before anything runs

Train minus validation accuracy at 50 training rows.

The gap you expect0.00 to 0.40 of accuracy

One line on why, if you want one

03 · Try it

A boundary that hugs the noise

computed in this browsermoons(1600, seed 7, 0.30), the same 2-16-8-1 network at both sizes

Step 1 of 4

With too few rows the boundary memorises where the points fell, not where the pattern is.

Shrink the training set

The control below moves the player between six recorded runs. Each one is the same 9-16-8-1 network, the same seed, the same thousand steps: the only difference is how many rows it was fitted on. Commit to the gap before you open a cell.

Training rows

syntheticloan seed 7

Six recorded runs, one per row count. The 400 validation rows and the 400 held-out rows are the same rows in all six.

At 50 training rows, how far apart will training and validation accuracy end up?

At 50 training rows, how far apart will training and validation accuracy end up?0.00 to 0.40 of accuracy

The sweep so far

Play a cell and its two accuracies appear here.

No cell has been played yet, so the chart is empty.

Training rows on a log axis. The shaded band is the gap; the short rule is what you predicted.

The gap check opens once you have opened the 1,200-row cell.

Find the early-stopping epoch

This curve is a fixture rather than a training run, and the two formulas that generate it are behind a disclosure on it. The axis is in epochs: one is one pass over every training row, which is many steps when the batch is small. Move the marker to the epoch you would stop at.

illustrativeearlystop-20-epoch-curve

These two curves come from formulas, not a training run; open the numbers below to see them.

Show the two formulas

val(e) = 0.280 + 0.600 e^(-0.25 e) + 0.00031117 e squared

train(e) = 0.760 e^(-0.115 e) + 0.055

Marker epoch1 to 20

Arrow keys move the marker one epoch at a time.

Epoch 9: validation loss 0.368, training loss 0.325.

Marker at epoch 9. Validation loss 0.368, training loss 0.325. Training loss falls at every epoch of this fixture; validation falls, turns and rises.

An illustrative curve, not a training run. Both series come from the two formulas behind the disclosure above.

Show all twenty epochs

Values shown in the figure above
epoch	training loss	validation loss
1	0.732	0.748
2	0.659	0.645
3	0.593	0.566
4	0.535	0.506
5	0.483	0.460
6	0.436	0.425
7	0.395	0.400
8	0.358	0.381
9	0.325	0.368
10	0.296	0.360
11	0.270	0.356
12	0.246	0.355
13	0.225	0.356
14	0.207	0.359
15	0.190	0.364
16	0.176	0.371
17	0.163	0.378
18	0.151	0.387
19	0.140	0.398
20	0.131	0.409

Name what early stopping costs

One of these four is supported by the curve you just read. The other three are not.

What stopping early costs youIt saves compute and nothing else: the model you keep is the same one you would have had.The epochs after the minimum still lower the training loss, so stopping early leaves training loss on the table. That is the trade.It costs nothing at all: an earlier epoch is better by every measure.It replaces the validation set, because the stopping rule is doing that job now.

Ask for a hint

Rungs open as you run the session, not as time passes. You have run it 0 times.

1. A question: available after 3 evaluated runs.

2. A narrowing: available after 5 evaluated runs.

3. A worked fragment, on a different curve: available after 7 evaluated runs.

4. The answer: available after 9 evaluated runs.

04 · Compare

05 · Check your understanding

Your check: an outcome you produce

not yet

Open a cell at 200 training rows or fewer whose gap is above 0.20, open the 1,200-row cell with a gap below 0.05, and set the early-stopping marker to the epoch where validation turns, within one epoch.

Not met yet. Keep working the session.

Still needed: a played run at 200 rows or fewer with a gap above 0.20
Still needed: the played run at 1,200 rows with a gap below 0.05
Still needed: the early-stopping marker on the epoch where validation turns

Save your takeaway

Your takeaway

Edit it into your own words, or keep the line as it is.

next up

6.2 Three regularisers Play now

Open the notebook