Learn machine learning · deep dive D6 · optional
The dataset where dropout hurts
Dropout is sold as a free win. Here is a small synthetic task where it costs five points of held-out accuracy across five seeds, and the condition that makes it do so.
Optional, and it blocks nothing. It opens because 6.2 Three regularisers is done. The gate asks you to store a five-seed mean at dropout 0.5 that is at least 0.04 below your five-seed mean at dropout 0.0, and to pick the explanation the numbers support.
01 · brief
The task is 200 points from the two-moons generator at noise 0.28, synthetic, split 140 for training and 60 held out. The network is 2-8-8-1 with tanh, trained by Adam at 0.02 for 800 full-batch steps.
At this size a single seed proves nothing: the held-out block is 60 rows, so one row is 1.7 points of accuracy. Every number on this page is therefore a mean over five seeds, 11 to 15, with the spread of those five drawn behind it.
The question you cannot answer yet: whether dropout can lower held-out accuracy at all, and if it can, what has to be true of the network for it to happen.
02 · commit
Before anything runs
Before anything is read: what will the five-seed mean held-out accuracy be at dropout 0.5 on this task? Without dropout it is a little over 0.86.
03 · work
Loading the recorded runs.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. The answer: available after 5 evaluated runs.
04 · verdict
Store dropout 0.0 and dropout 0.5 above, and the verdict states all three sweeps against what you predicted.
05 · gate
Your check: an outcome you produce
not yetA stored five-seed mean at dropout 0.5 at least 0.04 below your stored five-seed mean at dropout 0.0, plus the explanation the numbers support.
Not met yet. Keep working the session.
Choose one first.
carry
Your takeaway
Edit it into your own words, or keep the line as it is.