Learn machine learning · deep dive D1 · optional

Entropy versus Gini

Fourteen rows, four columns, and two ways to measure a mess. Compute the gains by hand under one measure, switch to the other, and say exactly what moved.

Deep dive D1DerivationAbout 40 minutesThe gate is a behaviour you produce

Optional, and it blocks nothing. It opens because 2.2 Impurity and splits is done. The gate is four computations within 0.002 plus one statement about what the criterion toggle changes.

01 · brief

Session 2.2 measured impurity with Gini and never said why that one. The usual alternative is entropy, and the usual claim is that the choice rarely matters. This is the table that claim is normally made on, so it is the table to check it on.

Entropy of a node is the sum over classes of minus p log2 p, measured in bits. Gini is one minus the sum of the squared proportions, which reads as the chance that two draws from the node disagree. Both are zero on a pure node and largest on an even mix. Both are used the same way: score every branch, weight by rows, subtract from the parent.

The question you cannot answer yet: does swapping the measure change which column the tree splits on first, and if not, what does it change?

02 · commit

Before anything runs

Before any arithmetic: switching from entropy to Gini on these fourteen rows does what to the ranking of the four columns?

Nothing. The same column comes first and the order holds.It reorders them, because the two measures reward different splits.The top column holds but the bottom two swap.

One line on why, if you want one

Choose something first.

03 · work

Parent entropy for nine plays and five does not

To three decimals or better, within 0.002.

Attempts: 0

Information gain for outlook

To three decimals or better, within 0.002.

Attempts: 0

Information gain for humidity

To three decimals or better, within 0.002.

Attempts: 0

Information gain for wind

To three decimals or better, within 0.002.

Attempts: 0

0 of 4 computations accepted.

The fourteen rows

computed in this browserpublic-domain teaching table

The fourteen-row weather table: four categorical columns and a play column.
row	outlook	temperature	humidity	wind	play
1	sunny	hot	high	weak	no
2	sunny	hot	high	strong	no
3	overcast	hot	high	weak	yes
4	rain	mild	high	weak	yes
5	rain	cool	normal	weak	yes
6	rain	cool	normal	strong	no
7	overcast	cool	normal	strong	yes
8	sunny	mild	high	weak	no
9	sunny	cool	normal	weak	yes
10	rain	mild	normal	weak	yes
11	sunny	mild	normal	strong	yes
12	overcast	mild	high	strong	yes
13	overcast	hot	normal	weak	yes
14	rain	mild	high	strong	no

Nine plays and five does not, so the parent counts are 9 and 5. This table is a public-domain teaching fixture. It is not drawn from the loan generator.

Which criterion the table below is measured with

Entropy, measured in bitsGini, a probability of disagreement

Parent entropy: 0.940286.

Weighted impurity and gain for each column under entropy.
column	branches	weighted	gain
outlook	3	0.693536	0.246750
temperature	3	0.911063	0.029223
humidity	2	0.788450	0.151836
wind	2	0.892159	0.048127

Under entropy the parent scores 0.940286 and the four columns rank outlook 0.246750, humidity 0.151836, wind 0.048127, temperature 0.029223.

Show the same four rows as a table

Values shown in the figure above
column	branches	weighted	gain
outlook	3	0.693536	0.246750
temperature	3	0.911063	0.029223
humidity	2	0.788450	0.151836
wind	2	0.892159	0.048127

Show every branch

Per-branch counts, impurity and weight under entropy.
column	branch	plays and does not	impurity	weight
outlook	sunny	2 and 3	0.970951	0.3571
outlook	overcast	4 and 0	0.000000	0.2857
outlook	rain	3 and 2	0.970951	0.3571
temperature	hot	2 and 2	1.000000	0.2857
temperature	mild	4 and 2	0.918296	0.4286
temperature	cool	3 and 1	0.811278	0.2857
humidity	high	3 and 4	0.985228	0.5000
humidity	normal	6 and 1	0.591673	0.5000
wind	weak	6 and 2	0.811278	0.5714
wind	strong	3 and 3	1.000000	0.4286

The two rankings, side by side

Entropy: outlook, then humidity, then wind, then temperature
Gini: outlook, then humidity, then wind, then temperature

Same order, different units. Neither criterion is the more correct one, and this table says nothing about which performs better in general.

Ask for a hint

Rungs open as you run the session, not as time passes. You have run it 0 times.

1. A question: available after 3 evaluated runs.

2. The answer: available after 5 evaluated runs.

04 · verdict

Finish the four computations and switch the criterion at least once, and the verdict states both rankings against what you committed to.

05 · gate

Your check: an outcome you produce

not yet

All four entropy computations within 0.002, plus the correct statement of what the Gini toggle changes.

Not met yet. Keep working the session.

What the toggle changes on this table

The ranking is unchanged and every number roughly halves, because entropy counts bits and Gini counts a probability of disagreement.Gini puts humidity above outlook, so the root split changes.Gini is the more correct measure, so its ranking is the one to trust.Gini is not an impurity measure, so there is no gain to compute.

Choose one first.

carry

Your takeaway

Edit it into your own words, or keep the line as it is.

back to the track

2.2 Impurity and splits

A deep dive blocks nothing. Open the notebook