each token looks back at the others
to figure out what a word means here, the model checks which earlier tokens matter for it.
who does “it” look at?
in the cat sat because it…, “it” pays most attention to “cat”: the 4%, cat 71%, sat 15%, because 10%. (illustrative numbers. the idea is real.)
how it works: q, k, v
| part | question it answers |
|---|---|
| query | what am i looking for? |
| key | what do i contain? |
| score | q · k → softmax |
| value | blend by score |
long text costs a lot more
every token checks every other token. double the text and the attention work roughly quadruples.
happybot’s note: no peeking at the future
a mask hides the tokens that come later. that’s what keeps next-token prediction honest.
next lesson: the transformer block, assembled.
in my own words · taught by b (@bbybarbarella)