Tokens in one sentence
Language models don’t read letters or whole words. They read tokens: frequently occurring chunks of text chosen when the model was built. Common words like “the” or “cat” are usually a single token. Longer or rarer words are split into pieces — “tokenization” might become “token” and “ization”. Spaces and punctuation are often folded into tokens too.
How text is split into tokens
Each model family has a tokenizer, a fixed vocabulary of tens of thousands of tokens plus rules for breaking any text into them. Most tokenizers are built with a method called byte-pair encoding, which starts from single characters and repeatedly merges the pairs that appear most often in a large sample of text. The result: everyday English words get their own tokens, while rare words, names and typos are assembled from several smaller pieces.
Because each company trains its own tokenizer, the same sentence can be a different number of tokens in different models. Most AI providers publish a free tokenizer tool so you can paste text and see exactly how it is split.
How many words is a token?
| Tokens | Approximate English words | Roughly equivalent to |
|---|---|---|
| 1 | ¾ of a word | A short word or part of a long one |
| 100 | 75 words | A short paragraph |
| 1,000 | 750 words | A one-and-a-half page essay |
| 100,000 | 75,000 words | A full-length novel |
These are averages for plain English prose. Text in languages such as Arabic, Hindi or Japanese commonly needs more tokens for the same meaning, because tokenizers were trained mostly on English. Computer code, long numbers, URLs and emoji can also be token-hungry.
Why tokens explain some strange AI mistakes
Ask a chatbot how many times the letter “r” appears in “strawberry” and older models often got it wrong. The model never saw individual letters — it saw two or three tokens — so counting characters is genuinely hard for it. The same goes for reversing words, rhyming tricky words or doing arithmetic on long numbers that are split awkwardly. Newer models handle many of these cases better, but the underlying reason hasn’t gone away.
Tokens and pricing
If you use AI through an API — for example, to power a chatbot on your website — you pay per token, typically quoted per million tokens. Two things are worth knowing:
- Input and output are priced separately. The tokens you send (your prompt, instructions and any documents) are input; the tokens the model writes back are output. Output usually costs several times more.
- The whole conversation is resent each turn. Chat apps send the full history with every new message, so a long conversation costs more per reply than a fresh one.
What is a context window?
The context window is the maximum number of tokens a model can consider at one time. It includes everything: system instructions, the conversation so far, any files you attached and the reply being written. If the total would exceed the limit, the app has to drop or summarise older material.
Context windows have grown fast. Early chat models handled a few thousand tokens — a handful of pages. Many current models accept hundreds of thousands of tokens, and some advertise a million or more, enough for several books at once.
Bigger isn’t automatically better
A large window doesn’t guarantee the model uses all of it well. Research published in 2023 under the title “Lost in the Middle” found that models were noticeably better at using information placed at the beginning or end of a long input than information buried in the middle. Long inputs also cost more and respond more slowly.
Practical tips for working with tokens
- Put your most important instructions at the start or end of a long prompt, not in the middle.
- Start a new chat when you change subject — it keeps the context focused and cheaper.
- Paste only the relevant part of a document rather than the whole thing.
- Ask for a summary of a long conversation, then continue in a fresh chat with that summary.
- If you’re building an app, use retrieval (see RAG vs. Fine-Tuning) to send only the passages that matter.
Frequently asked questions
Is a token the same as a word?
No. Short common words are often one token, but longer words, names and numbers are usually several. On average, English works out to about 1.3 tokens per word.
Do images count as tokens?
For models that accept images, yes — pictures are converted into a number of tokens based on their size and detail, and they count toward the context window and the bill.
Why did the AI forget what I said earlier?
Either the conversation grew past the context window and early messages were trimmed, or the detail was buried in a long context where the model paid it less attention. Restating key facts usually fixes it. For the bigger picture of how models process text, read How Do LLMs Work?.
