Learn machine learning · part 6 · session 4
Distribution shift
A model that passed every check can stop working after release, because the process that made the data did not hold still. Here you find the column that moved.
After this session you will be able to show that a validated model can fail on later data and name what would have caught it.
The gate is two of three calls and a monitoring plan that re-evaluates against fresh labels on a stated interval.
01 · The question
Every number you have produced so far assumed the held-out rows came from the same process as the training rows. When they do not, the track calls it . The synthetic generator behind this loan data does not hold still: at some point in its nine years of applications, one column starts to matter more than it did.
The model was fitted on the early years and comes to you finished. Scoring it is a forward pass over a few hundred rows, so every number here is computed on your own device as you press the controls. You score it on the late years and find the column. This is the ordinary way a model that passed every check stops working after release. Commit first: which single feature has changed its relationship to the label?
02 · Predict
The model reads nine columns: which income band the applicant is in, whether they put up collateral, how many years they have been employed, how much they asked for, what share of their income their debts already take (debt_ratio), how old they are, which region they are in, which year they applied, and how many loans they have had before.
Before anything runs
Which feature changed its relationship to the label?
Choose something first.
03 · Try it
Loading the recorded model.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on a two-feature model: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
04 · Compare
05 · Check your understanding
The gate: calls on unseen variants, and a plan
not yetTwo of the three calls, and a monitoring plan that re-scores the shipped model against freshly labelled rows on a stated recurring interval. Both parts are required: three correct calls with a plan that never looks at a label does not pass, and the right plan with one correct call does not either.
Call 1 of 3
Which block does the model fitted on 2016 to 2020 score worst on?
Call it before you see the note.
Call 2 of 3
Which feature changed its relationship to the label?
Call it before you see the note.
Call 3 of 3
For debt_ratio, what moved between the blocks?
Call it before you see the note.
Your plan: none selected yet. The plans are in the work beat above.
All three calls first.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.