Learn machine learning · part 4 · session 5
Softmax and temperature
Several scores into one distribution. Adding a constant changes nothing, multiplying by one changes a great deal, and the multiplier has a name.
After this session you will be able to turn a row of scores into probabilities by hand, and say what temperature does to the shape of them.
To pass: the three-way softmax entered by hand within 0.01 and summing to 1, with the exponentials given and the division yours, and the right call on what doubling every score does, made before the doubled bars are drawn.
01 · The question
The sigmoid turned one score into one probability. The operation that does the same job for several scores at once, exponentiating each and then dividing by the total so they sum to one, is .
Exponentiate means raise e to the power of each score. e is about 2.718, and the point of it is that a score one point higher comes out about 2.7 times bigger, so small gaps in score become large gaps in probability.
Two things follow that are worth knowing before you meet them again in 8.2, 9.2 and 10.1. Adding the same number to every score changes nothing, which is what makes the operation safe to compute. Multiplying every score by the same number changes a great deal, and the control that does the multiplying is called . Compute one by hand first.
The three class names below sit on an example score vector and attach to no real credit outcome.
02 · Predict
Before anything runs
Double every score, from (2.0, 1.0, 0.1) to (4.0, 2.0, 0.2). What happens?
Choose something first.
03 · Try it
The three-way softmax, by hand
Scores (2.0, 1.0, 0.1) over repaid, late, default. Three values, each within 0.01, and the three together must sum to 1.00 within 0.02.
e^2.0 = 7.389, e^1.0 = 2.718, e^0.1 = 1.105, total 11.212. Now divide each by the total.
To two decimals.
Attempts: 0
To two decimals.
Attempts: 0
To two decimals.
Attempts: 0
Within 0.02. A distribution has to sum to one.
Attempts: 0
Four scores, live
The fourth class opens at a very low score so the three-class exercise above stays readable in the bars.
Scores 2.0, 1.0, 0.1, -4.0. Probabilities 0.6579, 0.2420, 0.0984, 0.0016. Highest is repaid at 0.6579.
Temperature 1.0. Probabilities 0.6579, 0.2420, 0.0984, 0.0016. The gap between the top two is 0.4159.
Your four scores
computed in this browsertemperature 1.0Bars are drawn against the largest of them, which is the filled one. Scores 2.0, 1.0, 0.1, -4.0 at temperature 1.0.
Scores 2.0, 1.0, 0.1, -4.0. Probabilities 0.6579, 0.2420, 0.0984, 0.0016. Highest is repaid at 0.6579.
How the engine keeps the exponential from overflowing
This is a computing detail, not a modelling one: the engine takes the row maximum off before the exponential. Here that maximum is 2.0000 and the shifted scores are 0.00, -1.00, -1.90, -6.00.
Temperature, in a table
Turning the temperature
illustrativesoftmax on the 4.5 scoresT = 1.0. 0.66, 0.24, 0.10, 0.00.
At the example scores, T = 0.5 gives 0.8638, 0.1169, 0.0193 and T = 2 gives 0.5017, 0.3043, 0.1940. Halving the temperature is the same operation as doubling every score.
Show the four temperatures
| T | probabilities for (2.0, 1.0, 0.1) | reading |
|---|---|---|
| 0.5 | 0.8638, 0.1169, 0.0193 | the same as doubling every score |
| 1.0 | 0.6590, 0.2424, 0.0986 | the scores as given |
| 2.0 | 0.5017, 0.3043, 0.1940 | flatter |
| 0.2 | 0.9932, 0.0067, 0.0001 | close to argmax |
Double the scores
Your call is recorded above. Press the button and the doubled bars are drawn beside the original ones.
Record your position above first.
State the limit
04 · Compare
Draw the doubled scores and the verdict lands here, against the position you took above.
05 · Check your understanding
Your check: an outcome you produce
not yetBoth: the three probabilities for (2.0, 1.0, 0.1) entered by hand within 0.01 without the working shown and summing to 1.00 within 0.02, with the three exponentials given and the division yours, and the correct call on doubling, selected before the doubled bars were drawn.
Not met yet. Keep working the session.
Where you are
- Probabilities entered without the working shown: 0 of 3.
- The sum check: not yet.
- The call on doubling: not drawn yet.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on two classes: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
Save your takeaway
The temperature you are taking with you
7.5 and 9.2 open at whatever you set here. The default offered is 0.8, sharper than 1 without being argmax.
Your takeaway
Edit it into your own words, or keep the line as it is.