Learn machine learning · part 3 · session 3
Feature importance
Nine columns go in and one number comes out, and there is no rule path to read. Shuffle one column so its values no longer line up, and measure how much accuracy falls. That drop is the whole definition.
After this session you will be able to measure how much a model leaned on one column, and say why that measurement is about the model and not about borrowers.
The gate is a behaviour: duplicate a strong column, show what happens to the two bars, and pick the reading that explains it.
01 · The question
The is now better than the tree, and less legible. Nine columns go in and one number comes out, and there is no rule path to read. Permutation importance is the blunt way back in: take the , shuffle one column so its values no longer line up with the rows, and measure how much accuracy falls. That drop is the whole definition.
It is a statement about this model, not about lending. Before the chart appears, commit to which column you expect to matter most, and then I will duplicate that column and ask you to predict what happens to its bar.
The data and the model
syntheticseed 7The same synthetic loan rows as 3.2, generated in this browser from seed 7 and not calibrated to any real lending data, scored on the same 400 validation rows, and the forest your shelf carries as forest-config. Every shuffle uses seed 11, so the same bar comes back on a reload.
02 · Predict
Before anything runs
Which column will lose the forest the most held-out accuracy when it is shuffled?
Choose something first.
03 · Try it
Shuffling one column
illustrativea stand-in for the held-out blockNine columns of held-out rows, every value beside its own row.
This session needs the forest from 3.2
3.2 built the forest this session measures. Go back, run the corner with both controls on, and press Save. It takes about a minute. There is no forest-config on your shelf yet, and this session measures what that forest leaned on rather than building a different one.
04 · Compare
The verdict
Measure every column, then read the verdict against the column you committed to.
Waiting on a recorded position and at least one measured column.
05 · Check your understanding
Your check: an outcome you produce
not yetDuplicate a column whose bar is at least 0.02, show that neither copy keeps more than about half of it and that the two together are no larger than the one, and select the reading that explains why.
Not met yet. Keep working the session.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment on a different example: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.
The table on the shelf is this model's reliance on these columns at seed 11, across 5 shuffles each. This tells you what the model leaned on overall. It does not tell you why any one application was refused: a single tree could show you the path it took, and a forest has fifty paths.