Learn machine learning · part 9 · session 6
Context windows and retrieval
Attention compares every position with every earlier position, so the work grows with the square of the length, and that cost is paid again for every token produced.
After this session you will be able to compute what a longer context costs and say when retrieval is the cheaper answer.
To pass: two of three scenarios called correctly, each with a one-line reason recorded beside it.
01 · The question
The context window is a fixed slot
illustrativea drawing, not a measurementOne slot, one fixed width.
Everything a model can see at one moment sits in one fixed-width slot: your question, the documents you pasted, its own answer so far. That slot is the . Nothing outside it exists for the model.
is the alternative to sending everything: keep the documents outside, and each time you ask, send only the two or three passages that look closest to the question. On this page a length control prints the operation count for the bundled model's shape, using the same counter that drives the honest estimates elsewhere in this track. Then you paste a few documents, the page embeds them with the vectors from session 8.1, and a query pulls the three nearest passages instead of sending everything.
Production context lengths are not quoted here. A number about someone else's system belongs beside a primary source and a date, and the arithmetic on this page does not need one.
02 · Predict
Before anything runs
Raising the context from 1,024 to 4,096 multiplies the attention operation count by:
Choose something first.
Attention compares every position with every earlier position, so four times the context is sixteen times the attention cost. The control below counts it for this shape.
03 · Try it
The length control
What a longer context costs
computed in this browser2 blocks · d 64 · feed-forward 256- Attention, full pass
- 536.9M operations
- Attention, per token produced
- 524.3k operations
- Feed-forward, full pass
- 134.2M operations
- Ratio
- 4.00 to 1
- Crossover
- 256
The attention count is the score-and-mix part: every query position scores every position over 64 columns and then mixes them, four operations per column per pair, over 2 blocks. The projections are linear in the length and sit outside this count. The crossover is where the attention total overtakes the feed-forward total, computed rather than stated.
Context 1,024: attention 536.9M operations for a full pass, feed-forward 134.2M, ratio 4.0 to 1.
Show every length on the control
| context | attention | feed-forward | ratio |
|---|---|---|---|
| 64 | 2.1M | 8.4M | 0.25 |
| 128 | 8.4M | 16.8M | 0.50 |
| 256 | 33.6M | 33.6M | 1.00 |
| 512 | 134.2M | 67.1M | 2.00 |
| 1,024 | 536.9M | 134.2M | 4.00 |
| 2,048 | 2.1G | 268.4M | 8.00 |
| 4,096 | 8.6G | 536.9M | 16.00 |
| 8,192 | 34.4G | 1.1G | 32.00 |
Retrieve instead
Paste, index, query
computed in this browser2,000 word vectors, 32 dimensionsLoading the word vectors.
Against a budget
Send everything, or retrieve three
computed in this browsercounting characters for now- Send everything
- 0 tokens, 0 attention operations for a full pass. Inside the budget.
- Retrieve three passages
- 0 tokens, 0 attention operations for a full pass. Inside the budget.
At a budget of 1,200 tokens, send everything and retrieve three passages fit.
Show the two rows
| option | tokens | attention operations | inside the budget |
|---|---|---|---|
| Send everything | 0 | 0 | yes |
| Retrieve three passages | 0 | 0 | yes |
0 passages indexed, 0 returned by the last query.
Counting characters for now; finish 9.1, put tok-bpe300 on the shelf, and this becomes a token count.
04 · Compare
Call the three scenarios below and the verdict lands here, against the position you took above.
05 · Check your understanding
Your check: predict results on new examples
not yetTwo of three. Each scenario takes a call and a one-line reason; the reason is recorded with the call and is not scored.
Variant 1 of 3
A 400-page manual. Forty questions a day, each touching one section. The budget is the lowest cost per question.
Call it before you run it.
Variant 2 of 3
One contract of about 3,000 tokens. One question that requires comparing clause 4 with clause 39. Asked once.
Call it before you run it.
Variant 3 of 3
An assistant that must use the last two messages of the current conversation and nothing else, with a tight latency budget.
Call it before you run it.
Called 0 of 3 variants so far.
One line on why, recorded with each call
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question about each scenario: available after 3 evaluated runs.
2. What settles each one: available after 5 evaluated runs.
3. A fourth scenario, worked in full: available after 7 evaluated runs.
4. The three calls with their notes: available after 9 evaluated runs.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.
10.1 reuses this operation counter as the dense reference row beside the mixture-of-experts count.