Learn machine learning · part 1 · session 5
Accuracy lies
Ninety-five repaid against five not, a price on each of the two mistakes, and one slider. The threshold with the best accuracy is about to be the most expensive one on the page.
After this session you will be able to price two kinds of mistake and pick a threshold that minimises cost instead of error count.
Total cost at or below 410 on the audit set, and the follow-up reason chosen correctly.
01 · The question
I have deliberately rebuilt this held-out set so that only 5 in 100 are defaults, because that lopsidedness is where stops being informative. Approving every application scores 0.95 and misses every default. So put a price on the two mistakes instead. In this exercise, approving an application that is not repaid costs ten synthetic cost units; refusing one that would have been repaid costs one. Those numbers are invented for the exercise and describe no real lender. Move the , watch the four cells of the table, and find the cheapest place to stand. The four metrics you computed by hand in 1.4 sit beside the matrix and move with it, so you can see which of them the cheapest threshold sacrifices.
A model you have not built yet has given every application a repayment score between 0 and 1; higher means more likely to be repaid. Here the 1,000 applications are sorted by that score and cut into ten equal groups of 100, riskiest group first.
| Score decile, riskiest first | Rows | Not repaid | Local not-repaid rate | Upper score boundary |
|---|---|---|---|---|
| 1 | 100 | 18 | 18 per cent | 0.41 |
| 2 | 100 | 11 | 11 per cent | 0.53 |
| 3 | 100 | 7 | 7 per cent | 0.61 |
| 4 | 100 | 5 | 5 per cent | 0.68 |
| 5 | 100 | 4 | 4 per cent | 0.74 |
| 6 | 100 | 2 | 2 per cent | 0.79 |
| 7 | 100 | 1 | 1 per cent | 0.84 |
| 8 | 100 | 1 | 1 per cent | 0.89 |
| 9 | 100 | 1 | 1 per cent | 0.94 |
| 10 | 100 | 0 | 0 per cent | 1.00 |
02 · Predict
Before anything runs
The threshold with the highest accuracy will also have:
Choose something first.
03 · Try it
Sliding the line across 1,000 applications
syntheticseed 7 · the audit set aboveThreshold 0.00: 0 refused, cost 500, accuracy 0.950. Approve everything. This is also the most accurate the sweep gets.
Move the threshold
computed in this browserseed 7 · 1,000 audit rowsAn application scoring at or below the threshold is refused. Everything above it is approved.
0.00 approves every application. 1.00 refuses every application.
A positive prediction here means: refuse this application, the exercise's name for this one will not be repaid
| Actually positive | Actually negative | |
|---|---|---|
| Predicted positive | refused and not repaid18no cost: the refusal was right | refused and repaid821 unit each |
| Predicted negative | approved and not repaid3210 units each | approved and repaid868no cost: the approval was right |
Predicted positive, actually positive: 18. Predicted positive, actually negative: 82. Predicted negative, actually positive: 32. Predicted negative, actually negative: 868.
- Accuracy
- 0.886of everything, how much was right
- Precision
- 0.180of the rows you flagged, how many were real
- Recall
- 0.360of the rows that were real, how many did you flag
- F1
- 0.240the balance of the last two, in one number
Total cost, in synthetic cost units invented for this exercise
done402
10 × 32 approved and not repaid, plus 1 × 82 refused and repaid. The target for this session is 410.
These prices are invented for the exercise and describe no real lender.
Threshold 0.50. Approved 900, of which 868 repaid and 32 not repaid. Refused 100, of which 82 repaid and 18 not repaid. Accuracy 0.886, precision 0.180, recall 0.360, F1 0.240. Cost 402.
A whole number, matched exactly.
Attempts: 0
A whole number, matched exactly.
Attempts: 0
Why the accuracy-best threshold was the worst by cost
04 · Compare
The verdict appears once you have recorded a position in beat two. It names what the accuracy-best threshold costs, which is the thing beat two asks you to call.
05 · Check your understanding
Your check: reach the stated target
not yetTotal cost on the audit set at or below 410, and the follow-up reason selected correctly. Approving everything costs 500 and refusing everything costs 950, so neither of those passes.
Not met yet. Keep working the session.
- done Cost at or below 410: you are at 402, and the cheapest you have reached is 402.
- not yet The follow-up reason chosen correctly.
Save your takeaway
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on a different example: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
Your takeaway
Edit it into your own words, or keep the line as it is.