Learn machine learning · part 10 · session 4
The map
Every model in this track on two axes: how much structure the method assumes before it sees any data, and how much data it needs before it is useful.
After this session you will be able to place any model you meet on two axes and say what problem its architecture was solving.
The gate: at least 8 of 9 placements inside their rectangle, the seven lineage steps in order, at least 6 of 7 statements matched, and one sentence on what is still open.
01 · The question
Everything in this track is now on one page. Two properties separate the models more usefully than the order you met them in: how much structure the method imposes before it sees any data, and how many examples it needs before it is useful.
A hand-written rule imposes everything and needs almost nothing. A transformer block imposes almost nothing about the problem and pays for that in data. Place each model on those two axes, then put the lineage in order and name the problem each step solved. The last one is not solved.
Nothing runs here. Each placed model links to the session it was built in.
02 · Predict
Before anything runs
Which of these sits furthest towards few assumptions and high data hunger?
Choose something first.
03 · Try it
The horizontal axis is built-in assumptions, from few on the left to many on the right, where an assumption is something the method imposes before seeing data: axis-aligned cuts, a straight boundary, a fixed similarity, a recurrence. The vertical axis is data hunger, from low at the bottom to high at the top, meaning how many examples the method needs before it is useful on the tasks in this track. Both are five-step scales, so a placement is two selections and never a drag.
One is placed for you, to set the scale: the hand-written rule sits at the bottom right, because it imposes everything and needs almost no examples. Take it off and move it if you disagree.
1 · choose a model, then press the cell you would put it in
1 of 9 placed. Placing one moves the selection on to the next model that has no cell yet.
Data hunger up the side, built-in assumptions across the bottom
- Hand-written rulevery many assumptions, very low data hunger
- Linear modelnot placed yet
- Decision treenot placed yet
- Random forestnot placed yet
- Embedding retrievalnot placed yet
- Multi-layer perceptronnot placed yet
- Long short-term memory networknot placed yet
- Transformer blocknot placed yet
- Mixture-of-experts layernot placed yet
2 · read the map
Every placement, with the session each model was built in
- 1 · Hand-written rulesession 1.1 assumptions 0.9, data hunger 0.1
- 2 · Linear modelsession 4.1not placed yet
- 3 · Decision treesession 2.2not placed yet
- 4 · Random forestsession 3.2not placed yet
- 5 · Embedding retrievalsession 8.1 and 9.6not placed yet
- 6 · Multi-layer perceptronsession 5.2not placed yet
- 7 · Long short-term memory networksession 7.5not placed yet
- 8 · Transformer blocksession 8.5not placed yet
- 9 · Mixture-of-experts layersession 10.1not placed yet
These two axes are my judgement of the methods in this track, and the rectangles are wide because of it.
3 · put the seven steps in the order they arrived
The lineage, earliest first
Focus a step, press Enter to lift it, move it with the arrow keys, then press Enter again.
4 · match the problem each step solved
0 of 7 matched. Each statement belongs to one step only.
5 · one problem that is still open
Recorded, not scored. The only check on it looks for the mental-state verbs this track refuses for models, and it cannot judge your writing.
1 of 9 placed, 0 of 7 matched.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on a model that is not on the map: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: an outcome you produce
not yetIn one submission: at least 8 of 9 placements inside their rectangle, or within 0.05 of it; the seven lineage steps in the order they arrived; at least 6 of 7 statements matched to the right step; and one non-empty sentence on what is still open, with no banned verb in it.
Not met yet. Keep working the session.
Save your takeaway
What you can now say
A language model is a machine that, given the text so far, produces odds over the next small piece of text, picks one, and repeats. It has no plan for the sentence, no memory outside its window, and no marker inside it separating right from wrong.
What a language model does
computed in this browserone forward pass, weights trained offline by the course scriptThe text so far sits in a slot.
Your takeaway
Edit it into your own words, or keep the line as it is.
Step seven names the problem the mixture-of-experts layer opened rather than closed, so the map ends on a routing problem rather than on an arrival.