Learn machine learning · part 8 · session 2
Attention by hand
Compare queries and keys, turn scores into weights, then blend values. Work through each stage with four keys and four values you can check on paper.
After this session you will be able to compute attention weights and use them to blend value vectors into an output.
The gate is two things you produce: the four weights within 0.02, and an edited query for which key 3 takes the largest weight.
01 · The question
has three stages here. Compare the query with each key using a dot product. Scale those scores and apply softmax to get four weights that sum to one. Then blend the four value vectors in those proportions to produce the output.
Every position produces three things: a , which is what it is looking for; a key, which is what it advertises; and a value, which is what it hands over if chosen. Scores compare a query with each key; the weights decide how much of each value to take.
The words attends to, which I use from here on, are the name of that weighted average and nothing more. When I write that a position attends to another, I mean its weighted average drew some fraction of its value from that row. It is not a claim about attention in the ordinary sense.
Below is one query, four keys and four values, all integers, all editable. Forty five minutes, and you will be doing the arithmetic.
02 · Predict
Query [2, −1] points right and down. Key 1 is [1, 0] and key 3 is [1, 1]. Which of the two lines up better with the query?
Before anything runs
Query [2, −1] against keys [1, 0], [0, 1], [1, 1] and [−1, 1]. Which key will take the largest weight, and roughly how large?
Choose something first.
03 · Try it
Looking, then blending
illustrativethe fixture of this session, computed hereOne query, and four keys each carrying a value.
The fixture
| i | key | value |
|---|---|---|
| 1 | [1, 0] | [1, 0] |
| 2 | [0, 1] | [0, 2] |
| 3 | [1, 1] | [2, 2] |
| 4 | [-1, 1] | [-2, 1] |
Query q = [2, -1]. The dimension is 2, so the scale is 1 divided by the square root of 2, which is 0.707107.
illustrative Written for the lesson rather than measured from anything, and every number on this page is one you can check on paper.
One, the four dot products
within 0.001
Attempts: 0
within 0.001
Attempts: 0
within 0.001
Attempts: 0
within 0.001
Attempts: 0
Two, the scale
With longer vectors the scores get bigger, and big scores make the softmax put almost everything on one key. Dividing by the square root of the length keeps the scores in a useful range. The gradient through a saturated softmax is the chain you multiplied out in 7.3.
within 0.001
Attempts: 0
Three, the four weights
Each weight is the exponential of its scaled score divided by the sum of all four exponentials. Tolerance is 0.02 per weight, and the four have to sum to 1 within 0.02.
within 0.02
Attempts: 0
within 0.02
Attempts: 0
within 0.02
Attempts: 0
within 0.02
Attempts: 0
Check all four weights to compare them with the calculation.
Four, the output
within 0.02
Attempts: 0
within 0.02
Attempts: 0
Five, edit the query
Now change the numbers. The gate wants a query for which key 3 takes a strictly larger weight than every other key, and the engine judges the weights it computed rather than your arithmetic.
The fixture, editable
illustrativedimension 2Every number below is an input. Change one and the four dot products, the four weights and the output all follow, without any transition, so you can hold the two states side by side.
| row | key, first entry | key, second entry | value, first entry | value, second entry |
|---|---|---|---|---|
| 1 | ||||
| 2 | ||||
| 3 | ||||
| 4 |
The query
The four weights
computed in this browserKey 1 takes the largest weight, 0.6090, then key 3 at 0.3003.
Bars are drawn against the largest of the four, which is the filled one.
The same four weights as a table
| key | dot product | scaled score | weight |
|---|---|---|---|
| 1 | 2.0000 | 1.414214 | 0.6090 |
| 2 | -1.0000 | -0.707107 | 0.0730 |
| 3 | 1.0000 | 0.707107 | 0.3003 |
| 4 | -3.0000 | -2.121320 | 0.0177 |
Output: [1.1740, 0.7643]. Key 1 takes the largest weight, 0.6090.
Key 1 takes the largest weight, 0.6090, then key 3 at 0.3003. The output is 1.1740 and 0.7643.
Key 1 takes the largest weight, 0.6090. Key 3 is at 0.3003.
Ask for a hint
Rungs open as you run the session, not as time passes. You have run it 0 times.
1. A question: available after 3 evaluated runs.
2. A narrowing: available after 5 evaluated runs.
3. A worked fragment, on a different set: available after 7 evaluated runs.
4. The answer: available after 9 evaluated runs.
04 · Compare
05 · Check your understanding
Your check: an outcome you produce
not yetTwo things, in one session. The four softmax weights on the fixture within 0.02 each, summing to 1 within 0.02; and an edited query for which key 3 carries a strictly larger weight than every other key.
Not met yet. Keep working the session.
Where the two halves stand
Weights: not yet. Edited query: not yet.
Save your takeaway
Your takeaway
Edit it into your own words, or keep the line as it is.