Learn machine learning · part 10 · session 3
The capstone
Three briefs, three seeds you have not seen, and nothing new to learn. Every model here already exists; this page scores it.
After this session you will be able to assemble a pipeline from parts you built, beat a named baseline on data you have not seen, and say what each part contributed.
The gate: all three briefs beat their named baseline on the unseen seed, every slot carries a justification that passes both checks, and both removal tests have a prediction recorded before they are scored.
01 · The question
A pipeline is four choices in a row: which data, how the text or rows are turned into numbers, which model, and where the cut-off sits. Every brief below is those same four slots.
Three briefs, three seeds you have not seen, and nothing new to learn. Each brief has a fixed set of slots and a named baseline, and only artifacts on your shelf may go in a slot. Nothing is trained here. Every model in these lists already exists, trained in its own session or recorded by the course script, and what this page does is score them on data none of them has seen, which is one pass through a few hundred rows and is over before you lift your finger. The baselines are scored on the same rows.
The checks refuse a pipeline whose shapes do not line up and tell you why rather than failing quietly. For each slot you write one line saying why it is there, then you predict the held-out number before the scoring pass. Then take two pieces back out, one at a time, and say first what you think each removal will cost. The first brief has a right answer that is not the newest thing on your shelf.
The three briefs
| Brief | Seed | Held out on | Baseline |
|---|---|---|---|
| 1 · Tabular risk | 913 | cost in synthetic units per 100 rows | base-pruned-tree |
| 2 · Short-text classification | 2287 | accuracy | base-bow-41 |
| 3 · Text completion | 4471 | cross-entropy in nats per character | base-lstm-char-6k |
Synthetic, all three. Brief 2 is scored with the precision, recall and F1 of the same two-by-two table beside its accuracy, from the metric card of 1.4.
Not on your shelf yet
17 artifacts are missing. Each one is a session, and each session is still there.
feature-setsession 1.2
Would fill brief 1, the features slot. Go back to it
metric-cardsession 1.4
Would fill the four metrics printed beside every accuracy. Go back to it
cost-modelsession 1.5
Would fill brief 1, the threshold slot and the two prices. Go back to it
pruned-treesession 2.4
Would fill brief 1, the model slot, and brief 1's baseline. Go back to it
forest-configsession 3.2
Would fill the model slot of briefs 1, 2 and 3. Go back to it
logistic-loansession 4.2
Would fill brief 1, the model slot. Go back to it
temperature-settingsession 4.5
Would fill brief 3, the decoding slot. Go back to it
xor-mlp-2hsession 5.2
Would fill brief 2, the model slot. Go back to it
loan-mlp-v1session 5.6
Would fill brief 1, the model slot. Go back to it
bow-classifier-41session 7.1
Would fill brief 2's baseline. Go back to it
lstm-char-6ksession 7.5
Would fill brief 3, the model slot, and brief 3's baseline. Go back to it
embed-subset-16session 8.1
Would fill brief 2, the representation slot. Go back to it
copy-attn-2ksession 8.6
Would fill brief 3, the model slot. Go back to it
tok-bpe300session 9.1
Would fill the representation slot of briefs 2 and 3. Go back to it
head-text20session 9.4
Would fill brief 2, the model slot. Go back to it
chk-bannedsession 9.5
Would fill the check on every justification line. Go back to it
moe-k2session 10.1
Would fill brief 1, the model slot. Go back to it
02 · Predict
Before anything runs
Which model will win brief 1 on held-out cost?
Choose something first.
Before anything runs
What held-out accuracy will your brief 2 pipeline reach on the forty scored strings?
Before anything runs
What held-out cross-entropy will your brief 3 pipeline reach on the thirty passages?
Uniform over thirty-two symbols costs 3.47 nats a character, which is the number a model that has acquired nothing would produce.
03 · Try it
Loading the word table and the three seeds.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on a different problem: available after 7 evaluated runs.
4. A passing pipeline for each brief: available after 9 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: three baselines on three unseen seeds, with every slot justified and both removal tests predicted first
not yetBrief 1's held-out cost strictly below base-pruned-tree on the same 400 rows. Brief 2's held-out accuracy at least 0.05 above base-bow-41 on the same 40 strings. Brief 3's cross-entropy strictly below base-lstm-char-6k on the same 30 passages, and its word-shape rate at or above the baseline's on the same seeds. In every brief: four justifications that pass both checks, and two removal tests whose prediction was recorded before the pass.
Not met yet. Keep working the session.
- Brief 1: the pipeline is refused; nothing is scored yet; a slot has no justification that passes both checks; two removal tests with a prediction recorded before the pass are needed
- Brief 2: the pipeline is refused; nothing is scored yet; a slot has no justification that passes both checks; two removal tests with a prediction recorded before the pass are needed
- Brief 3: nothing is scored yet; a slot has no justification that passes both checks; two removal tests with a prediction recorded before the pass are needed
Your lab report
The report is a page drawn from a code in its own address bar. It carries the three pipelines, the numbers, your justifications and your "What you can do" list, and nothing else from your notebook.
Available when all three briefs pass.
Generated from teaching models trained in a browser on small synthetic data. The numbers are real results from those models. They are not benchmarks and do not transfer to production systems.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.
The report travels as a link. Anyone you send it to sees these three briefs and the lines you wrote, and none of the rest of your notebook.