Learn machine learning · part 1 · session 2
Features: the model only sees columns
The box is fixed. The only thing you change is which columns it may read, and one of them will push held-out accuracy to a number you should not want.
After this session you will be able to tell a leaky column from a strong one by asking when its value exists.
Three variants on three seeds. Call the leaky column in at least two of them before the reveal.
01 · The question
The in this session is fixed. The only thing you change is which columns it is allowed to read. Toggle a column off and it is gone; the model has no other source of information about the applicant. One of the columns on this page pushes accuracy to about 0.99, which is far above anything the others reach together. Before you celebrate a number like that, ask when its value becomes available. A column recorded after the outcome is not a feature. It is the answer, wearing a different name.
The column you are trying to predict is called the : here, repaid or not. A is just a column the model is allowed to read. A column that quietly contains the label is said to leak. Correlation, below, is a single number for how closely a column tracks the label: 0 means no relation, 1 means it moves exactly with it.
About the box
The fixed model is a depth-3 decision tree. You build one of these yourself in Part 2. Today it is a box with one input you control: the column list.
02 · Predict
Before anything runs
Held-out accuracy with every column switched on will be:
Choose something first.
03 · Try it
The columns the box is allowed to read
computed in this browserdepth 3 · seed 7Switch a column off and it is gone. The box has no other source of information about the applicant.
Held-out accuracy on 600 rows
0.827
9 columns on. Held-out accuracy 0.827, unchanged. Adding days_since_default_flag would be the next one.
When does each column exist?
illustrativethe columns of this generator, placed in timeThe timing test
For each column, say the earliest moment its value exists. The application is scored at the moment it arrives, so any value that does not exist by then cannot be read.
| Column | Earliest moment this value exists | Moment the application is scored |
|---|---|---|
| debt_ratio | At application | |
| days_since_default_flag | At application | |
| prior_loans | At application |
Fill the middle column for each row. Nothing here counts towards the gate.
The three variants
Each variant is a fresh 2,000 rows on its own seed, split 1,400 and 600. For every candidate column you get the held-out accuracy the box reaches when that column alone is added to a fixed base set, and the column's correlation with the label. The calls are made at the gate.
Variant 1 of 3
syntheticseed 7 · 2,000 rowsBase set of 7 columns, held-out 0.790.
Seed 7. Base set alone 0.790. debt_ratio, held-out 0.827, correlation 0.44. days_since_default_flag, held-out 0.993, correlation 0.94. prior_loans, held-out 0.790, correlation 0.13.
Synthetic. Seed shown above. The leaky column in this exercise is constructed from the outcome on purpose.
Show the candidate numbers
| what was measured | accuracy |
|---|---|
| debt_ratio | 0.827 |
| days_since_default_flag | 0.993 |
| prior_loans | 0.790 |
| the base set on its own | 0.790 |
Show the candidates as a table
| Column | Held-out accuracy when added alone | Correlation with the label |
|---|---|---|
| debt_ratio | 0.827 | 0.44 |
| days_since_default_flag | 0.993 | 0.94 |
| prior_loans | 0.790 | 0.13 |
Variant 2 of 3
syntheticseed 21 · 2,000 rowsBase set of 7 columns, held-out 0.790.
Seed 21. Base set alone 0.790. loan_amount, held-out 0.790, correlation -0.25. collection_letters_sent, held-out 0.987, correlation 0.93. age, held-out 0.790, correlation -0.12.
Synthetic. Seed shown above. The leaky column in this exercise is constructed from the outcome on purpose.
Show the candidate numbers
| what was measured | accuracy |
|---|---|
| loan_amount | 0.790 |
| collection_letters_sent | 0.987 |
| age | 0.790 |
| the base set on its own | 0.790 |
Show the candidates as a table
| Column | Held-out accuracy when added alone | Correlation with the label |
|---|---|---|
| loan_amount | 0.790 | -0.25 |
| collection_letters_sent | 0.987 | 0.93 |
| age | 0.790 | -0.12 |
Variant 3 of 3
syntheticseed 33 · 2,000 rowsBase set of 7 columns, held-out 0.763.
Seed 33. Base set alone 0.763. years_employed, held-out 0.763, correlation -0.17. reviewed_by_recovery_team, held-out 0.927, correlation 0.83. income_band, held-out 0.763, correlation -0.34.
Synthetic. Seed shown above. The leaky column in this exercise is constructed from the outcome on purpose.
Show the candidate numbers
| what was measured | accuracy |
|---|---|
| years_employed | 0.763 |
| reviewed_by_recovery_team | 0.927 |
| income_band | 0.763 |
| the base set on its own | 0.763 |
Show the candidates as a table
| Column | Held-out accuracy when added alone | Correlation with the label |
|---|---|---|
| years_employed | 0.763 | -0.17 |
| reviewed_by_recovery_team | 0.927 | 0.83 |
| income_band | 0.763 | -0.34 |
Correlation figures are properties of this synthetic generator only.
04 · Compare
The verdict appears once you have recorded a position in beat two. It opens with the number beat two asks you for.
05 · Check your understanding
Your check: predict results on new examples
not yetThree variants on three seeds. Name the leaky column in each one before you run it. Two of three passes. Each reveal refits the real tree on the real variant, so you can open it and check.
Variant 1 of 3
Variant 1, seed 7. Three candidate columns, with the held-out accuracy each one reaches when it alone is added to the base set and its correlation with the label. Which one leaks?
Call it before you run it.
Variant 2 of 3
Variant 2, seed 21. Three candidate columns, with the held-out accuracy each one reaches when it alone is added to the base set and its correlation with the label. Which one leaks?
Call it before you run it.
Variant 3 of 3
Variant 3, seed 33. Three candidate columns, with the held-out accuracy each one reaches when it alone is added to the base set and its correlation with the label. Which one leaks?
Call it before you run it.
Called 0 of 3 variants so far.
Save your takeaway
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on a different example: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
Your takeaway
Edit it into your own words, or keep the line as it is.