Learn machine learning · deep dive D8 · optional

Two heads of dimension one

Multi-head attention is the same operation run on disjoint slices of the same vectors. Split the 8.2 fixture down the middle, compute both halves, and the claim is either true on these numbers or it is not.

Deep dive D8DerivationAbout 45 minutesYour check: demonstrate the idea yourself

This is optional. It opens once 8.2 Attention by hand is done, and nothing in the track waits on it. The gate is five of six numbers within 0.02 and the right reading of why the two results differ.

01 · The question

8.2 computed one head over two-dimensional vectors. Here the same fixture is cut into its first and second coordinates, and each half is run as its own head of dimension one. The scale is one over the square root of the head's dimension, which here is one over the square root of one. It is printed as 1 so that the rule is visible rather than mistaken for a constant somebody dropped.

Commit before you compute. The two head outputs, side by side, against the single head's two-dimensional output from 8.2.

02 · Predict

Before anything runs

How will the two one-dimensional heads compare with the single two-dimensional head from 8.2?

The same two numbers.Within about 0.01 of each other.Clearly different, by more than 0.1.

One line on why, if you want one

Choose something first.

03 · Try it

The fixture, split

Head A takes the first coordinate of the query, of every key and of every value. Head B takes the second. Nothing else changes: same operation, same softmax, same weighted average.

head A, coordinate 1

computed in this browserscale 1 / sqrt(1) = 1

Query 2, keys [1, 0, 1, -1], values [1, 0, 2, -2]. The scale is one over the square root of one, which is 1, so the scores and the scaled scores are the same numbers here. That is the rule doing its job, not the rule being skipped.

key 10.4643

key 20.0628

key 30.4643

key 40.0085

Scores, exponentials and weights for head A
step	value
scores, query times key	2, 0, 2, -2
scaled by 1 over the square root of 1	2, 0, 2, -2
exponentials	7.3891, 1.0000, 7.3891, 0.1353
their sum	15.9134
weights	0.4643, 0.0628, 0.4643, 0.0085
output, weights against values	1.3760

head A reads coordinate 1 only: query 2, keys 1, 0, 1, -1, values 1, 0, 2, -2. The largest weight is 0.4643 on key 1. The output is 1.3760.

head A computed on its own coordinate, live in this browser.

Show this head's numbers

Values shown in the figure above
key	key value	value	score	weight
1	1	1	2	0.4643
2	0	0	0	0.0628
3	1	2	2	0.4643
4	-1	-2	-2	0.0085

head B, coordinate 2

computed in this browserscale 1 / sqrt(1) = 1

Query -1, keys [0, 1, 1, 1], values [0, 2, 2, 1]. The scale is one over the square root of one, which is 1, so the scores and the scaled scores are the same numbers here. That is the rule doing its job, not the rule being skipped.

key 10.4754

key 20.1749

key 30.1749

key 40.1749

Scores, exponentials and weights for head B
step	value
scores, query times key	0, -1, -1, -1
scaled by 1 over the square root of 1	0, -1, -1, -1
exponentials	1.0000, 0.3679, 0.3679, 0.3679
their sum	2.1036
weights	0.4754, 0.1749, 0.1749, 0.1749
output, weights against values	0.8744

head B reads coordinate 2 only: query -1, keys 0, 1, 1, 1, values 0, 2, 2, 1. The largest weight is 0.4754 on key 1. The output is 0.8744.

head B computed on its own coordinate, live in this browser.

Show this head's numbers

Values shown in the figure above
key	key value	value	score	weight
1	0	0	0	0.4754
2	1	2	-1	0.1749
3	1	2	-1	0.1749
4	1	1	-1	0.1749

Six numbers

Tolerance is 0.02. Five of the six have to land.

Head A, the score for key 1

0.02

Attempts: 0

Head A, the weight on key 1

0.02

Attempts: 0

Head A, the output

0.02

Attempts: 0

Head B, the score for key 2

0.02

Attempts: 0

Head B, the weight on key 1

0.02

Attempts: 0

Head B, the output

0.02

Attempts: 0

Against the single head

The concatenated heads against the single two-dimensional head
run	first entry	second entry
two heads, concatenated	1.3760	0.8744
one head of dimension two, from 8.2	1.1740	0.7643
difference	0.2019	0.1101

Both are computed here, in this browser, from the same fixture.

One reading to choose

Why do the two results differ?

Each head normalises over its own scores, so a key that is strong in one coordinate cannot be outvoted by the other coordinate inside that head.The two runs use different scale factors.The difference is rounding.They should be identical and one of the two runs is wrong.

Choose one before locking it in.

After the concatenation

The output projection

computed in this browseridentity

The two head outputs are joined into one vector and then multiplied by a matrix before they leave the layer. Set the four numbers and watch what arrives at the next layer. At the identity nothing moves, which is why the identity is where it starts.

Output row 1

head Ahead B

Output row 2

head Ahead B

The concatenated head outputs before and after the output projection
vector	first entry	second entry
heads, concatenated	1.3760	0.8744
through the projection	1.3760	0.8744

The concatenated heads are 1.3760 and 0.8744. Through this projection they come out as 1.3760 and 0.8744.

Concatenation is one step; the projection after it is part of the same operation.

Ask for a hint

Rungs open as you run the session, not as time passes. You have run it 0 times.

1. A question: available after 3 evaluated runs.

2. The answer: available after 5 evaluated runs.

04 · Compare

05 · Check your understanding

Your check: an outcome you produce

not yet

Five of the six numbers within 0.02, and the right reading of why the concatenated heads are not the single head's answer.

Not met yet. Keep working the session.

0 of 6 numbers within tolerance, 5 needed. No reading locked in yet.

Save your takeaway

Your takeaway

Edit it into your own words, or keep the line as it is.

back to the track

8.2 Attention by hand

Open the notebook