what is a token?

models don’t read words

chatgpt has never read a single word you typed. it reads tokens: small chunks of text, often just part of a word.

take “unhappiness”. a tokenizer might split it into un · happi · ness.

text ⇄ ids

each chunk has an id in a fixed vocabulary. text → ids is encoding. ids → text is decoding, which is how the model’s answer becomes words again.

how it works: byte-pair encoding

1. split all your training text into single characters.
2. count every pair of neighbors. the most common pair merges into one new token.
3. in low, lower, lowest, “l o” is the most common pair → lo. then lo + w → low.
4. repeat until the vocabulary hits a target size, like 50,000 tokens.

key insight: nothing is ever unknown

common words become one token. rare words become pieces. any text can be written with the vocabulary.

why it matters

you pay per token, and the context limit counts tokens, not words.

sources

• stanford cs336, language modeling from scratch (cs336.stanford.edu)
• sennrich, haddow & birch (2016), neural machine translation of rare words with subword units
• gage (1994), a new algorithm for data compression
• karpathy (2024), let’s build the gpt tokenizer
• hugging face llm course, byte-pair encoding tokenization

in my own words · taught by b (@bbybarbarella)