Learn machine learning · part 1 · session 4
Accuracy, precision, recall, F1
Twenty predictions, four cells, four numbers. Each of the four answers a different question, and one of them is about to come in a good deal lower than the others.
After this session you will be able to compute accuracy, precision, recall and F1 from four counts, and say which question each one answers.
Three engine-generated matrices in a row, all four numbers each time, within 0.01.
01 · The question
answers one question, and it is not the only question worth asking. Here are twenty predictions from the validation split you set up in 1.3. Sort each row into one of four cells: predicted and actual, in the two ways they can agree and the two ways they can disagree. From those four counts you can compute four numbers, and each one answers something different. asks about the rows you flagged. asks about the rows that were real. is the number you report when you want one figure that neither precision nor recall can flatter on its own.
The four cells have names. Flagged and it happened: true positive. Flagged and it did not: false positive. Not flagged and it happened: false negative. Not flagged and it did not: true negative.
F1 is built from the other two, and this is the whole of it: F1 = 2 × precision × recall ÷ (precision + recall).
Positive does not mean good. It means the thing you are looking for, here an application that will not be repaid, the way a positive test result means the illness was found.
Synthetic loan applications, seed 7, twenty rows from the validation split. No real lending data is used anywhere in this track. The positive class throughout this session is "did not repay".
02 · Predict
Four numbers, and what each one counts:
- accuracy: of every row, how many were right.
- precision: of the rows you flagged, how many really were positive.
- recall: of the rows that really were positive, how many you flagged.
- F1: the two above folded into one number that stays low when either is low.
Before anything runs
Twenty rows, only a few of them defaults. Which of the four do you expect to come out lowest?
Choose something first.
03 · Try it
The twenty rows
syntheticseed 7 · validation splitThe positive class throughout this session is "did not repay". Move through the rows with the arrow keys, press Enter to select one, then choose the cell it belongs in.
Select a row first. The four buttons place the selected row.
All twenty rows have to be placed before the sort can be checked.
0 of 20 rows placed.
The matrix filling up
illustrativeThis view trains nothing. It moves counts the fixture has already set, one row at a time, and names which of the four numbers changed on each move.
A positive prediction here means: this application was not repaid
| Actually positive | Actually negative | |
|---|---|---|
| Predicted positive | true positive0 | false positive0 |
| Predicted negative | false negative0 | true negative0 |
Predicted positive, actually positive: 0. Predicted positive, actually negative: 0. Predicted negative, actually positive: 0. Predicted negative, actually negative: 0.
Nothing placed yet. All four counts are zero, so only accuracy has a value and it is 0.000.
The same four cells, two different denominators
syntheticseed 7 · the twenty authored rowsFour counts, one table.
Four hand computations, each within 0.01
To two decimals, within 0.01.
Attempts: 0
To two decimals, within 0.01.
Attempts: 0
To two decimals, within 0.01.
Attempts: 0
To two decimals, within 0.01.
Attempts: 0
What question does each one answer
Select a metric, then select the question it answers. Re-matching costs nothing.
No metric selected.
of the rows that were real, how many did you flag
of everything, how much was right
the balance of the last two, in one number
of the rows you flagged, how many were real
0 of 4 questions matched.
Why the harmonic mean
Take a deliberately lopsided pair: a prediction that flags a single row and is right about it has precision 1.00 and recall 0.02. The arithmetic mean of that pair is (1.00 + 0.02) / 2 = 0.51, which reads like a passable result for a prediction that missed 98 of every 100 positives.
To two decimals, within 0.01.
Attempts: 0
The harmonic mean sits near the smaller of the two numbers, so one of them being near zero cannot be averaged away.
04 · Compare
The verdict appears once you have submitted the sort.
05 · Check your understanding
Three in a row, all four numbers each time
computed in this browserseed 101Three in a row, all four numbers each time. A wrong entry sends the count back to zero, and the attempts are written to your notebook either way.
A positive prediction here means: this row is positive
| Actually positive | Actually negative | |
|---|---|---|
| Predicted positive | true positive7 | false positive23 |
| Predicted negative | false negative31 | true negative27 |
Predicted positive, actually positive: 7. Predicted positive, actually negative: 23. Predicted negative, actually positive: 31. Predicted negative, actually negative: 27.
All four boxes need a number before submitting.
Streak: 0 of 3.
Predicted positive, actually positive: 7. Predicted positive, actually negative: 23. Predicted negative, actually positive: 31. Predicted negative, actually negative: 27.
Your check: an outcome you produce
not yetThree consecutive engine-generated matrices with all four metrics entered correctly within 0.01. The four inputs for one matrix are submitted together, and a single wrong entry restarts the streak at zero.
Not met yet. Keep working the session.
Save your takeaway
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on a different example: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
Your takeaway
Edit it into your own words, or keep the line as it is.