every token gets a vector
a token id is just a row number. the model looks up that row in a big table and gets back a long list of numbers.
one token, 768 numbers
cat → id 9246 → [0.21, -1.30, 0.08, 0.77, -0.45, …]. that list is the token’s embedding. (the numbers here are made up. the shape is real.)
the table learns too
| step | what happens |
|---|---|
| start | random numbers |
| train | nudge the rows that help |
| predict | the next token |
| end | similar words cluster |
meaning turns into distance
words used in similar ways end up close together, so what the model learns about one helps with the others.
happybot’s note: the table alone is huge
50,257 tokens × 768 numbers ≈ 38 million weights, before the model has done any thinking at all.
in my own words · taught by b (@bbybarbarella)
next: how tokens pay attention →