Learn machine learning · part 4 · session 2

Sigmoid and log-loss

A probability is a price. Set ten of them on held-out applications, settle the book, and read what confidence costs when it is wrong.

After this session you will be able to turn a score into a probability, and say what a confident wrong answer costs against a hedged one.

loss.betSession 4.2About 35 minutesYour check: reach the stated target

To pass: a mean log-loss at or below 0.40 on the ten held-out rows, and the right reason for why 0.5 everywhere beats 0.99 everywhere here.

01 · The question

A score is a number on an open scale and it is not much use for deciding anything. The function that squashes it into the range zero to one, so it can be read as a , is the . Then the model has to pay for that probability, and the payment is minus the logarithm of the probability it put on whatever actually happened. That payment is .

The logarithm is doing one job here: it turns a small probability into a large bill. It is a betting book, and the price of being confident and wrong is very high while the price of being confident and right is almost nothing. Ten held-out applications are below. Set your ten numbers before you see any outcome.

The bill

illustrativeminus the natural logarithm

Being right costs almost nothing. Being confident and wrong costs a great deal, and the cost rises without limit as your probability approaches zero.

02 · Predict

Before anything runs

Someone bets 0.99 on every one of these ten applications. Three of them did not repay. How does their mean log-loss compare with betting 0.5 on everything?

Lower, because seven of ten are right.About the same.Higher, by roughly a factor of two.

One line on why, if you want one

Choose something first.

03 · Try it

The sigmoid, by hand

sigmoid(z) = 1 / (1 + exp(-z)). Two values, to three decimals.

Squashing a score

illustrativethe sigmoid from linear.ts

Step 1 of 4

A score axis, with a marker at 2.

Any score, however large, lands somewhere between 0 and 1; a score of 0 lands exactly on 0.5.

sigmoid(0) = ?

Within 0.005.

Attempts: 0

sigmoid(2) = ?

Within 0.005.

Attempts: 0

A score of 0 is a probability of 0.5, which is the boundary line from 4.1 restated.

Train the model

Two columns, debt ratio and collateral, fitted on the training rows of the loan generator at seed 7. Everywhere else in this track the model predicts a refusal. Here it predicts repayment instead, because you are betting on the loan being repaid.

trained in this browserfitLogistic, 400 steps, seed 7

Nothing is fitted yet. The run takes 400 steps over 1,400 rows and finishes in front of you.

Place ten bets

Each slider is your probability that the application repaid. The outcomes stay hidden until you settle.

Ten held-out applications

syntheticseed 7

Generated in this browser. Not calibrated to any real lending data.

Set your ten first. Then compare.

A1
debt ratio 0.20, collateral 1
Your probability that A1 repaid0.01 to 0.99
Your probability that A1 repaid, as a number
A2
debt ratio 0.35, collateral 1
Your probability that A2 repaid0.01 to 0.99
Your probability that A2 repaid, as a number
A3
debt ratio 0.72, collateral 0
Your probability that A3 repaid0.01 to 0.99
Your probability that A3 repaid, as a number
A4
debt ratio 0.44, collateral 1
Your probability that A4 repaid0.01 to 0.99
Your probability that A4 repaid, as a number
A5
debt ratio 0.28, collateral 0
Your probability that A5 repaid0.01 to 0.99
Your probability that A5 repaid, as a number
A6
debt ratio 0.55, collateral 1
Your probability that A6 repaid0.01 to 0.99
Your probability that A6 repaid, as a number
A7
debt ratio 0.88, collateral 0
Your probability that A7 repaid0.01 to 0.99
Your probability that A7 repaid, as a number
A8
debt ratio 0.31, collateral 1
Your probability that A8 repaid0.01 to 0.99
Your probability that A8 repaid, as a number
A9
debt ratio 0.95, collateral 1
Your probability that A9 repaid0.01 to 0.99
Your probability that A9 repaid, as a number
A10
debt ratio 0.40, collateral 0
Your probability that A10 repaid0.01 to 0.99
Your probability that A10 repaid, as a number

No running log-loss is shown here, because the outcomes are hidden. The mean of your ten bets is 0.700 and the base rate on the training rows is 0.70.

Three reference strategies

The same function, run on three flat strategies and on the model's own column, with the arithmetic beside each.

Strategy	Arithmetic	Mean log-loss
0.5 on every row	every row costs -ln(0.5) = 0.693147; 10 x 0.693147 = 6.93147; divided by 10	0.693147
0.99 on every row	7 right rows at -ln(0.99) = 0.010050 give 0.070352; 3 wrong rows at -ln(0.01) = 4.605170 give 13.815511; sum 13.885863; divided by 10	1.388586
0.70 on every row, the base rate	7 x -ln(0.70) = 7 x 0.356675 = 2.496725; 3 x -ln(0.30) = 3 x 1.203973 = 3.611918; sum 6.108643; divided by 10	0.610864
the model's own p	per row from the table above	0.285006

One more by hand, then the reason

Betting 0.99 on all ten, what is the total cost of the three rows that did not repay?

Within 0.001.

Attempts: 0

Why does 0.5 on every row cost less than 0.99 on every row here?

0.5 is right more often.Log-loss counts errors and 0.5 makes fewer.The logarithm of a small number is large in magnitude, so three rows at -ln(0.01) = 4.61 outweigh seven rows at -ln(0.99) = 0.01.0.99 is outside the allowed range.

04 · Compare

Settle the book and the verdict lands here, against the position you took above.

05 · Check your understanding

Your check: reach the stated target

not yet

Both: a mean log-loss at or below 0.40 on the ten held-out rows, and the reason that names the logarithm rather than the error count.

Not met yet. Keep working the session.

Where you are

Your mean log-loss: nothing settled yet against a target of 0.40. For scale: betting 0.5 on everything settles at 0.69, and the model's own numbers settle at 0.29.
The reason: not yet.

For reference, betting the base rate of 0.70 on all ten settles at 0.6109, which misses.

Ask for a hint

Rungs open as you run the session, not as time passes. You have run it 0 times.

1. A question: available after 3 evaluated runs.

2. A narrowing: available after 5 evaluated runs.

3. A worked fragment, on four other rows: available after 7 evaluated runs.

4. The answer: available after 9 evaluated runs.

Save your takeaway

Your takeaway

Edit it into your own words, or keep the line as it is.

next up

4.3 Slope without calculus Play now

Open the notebook