Area 01 · LLM · Concept 01 of 20

Tokenization

Text is cut into tokens (word pieces) before the model reads it. Limits and prices are counted in tokens.

01–10 · The basePhase 1 · Weeks 3–6≈ 45 min

The idea

A model never sees letters or words. Before your prompt reaches it, a tokenizer splits the text into tokens: common words become one token, rare words are split into several pieces, and spaces and punctuation are usually attached to the piece that follows. Each token is then swapped for an integer ID from a fixed vocabulary of roughly 50,000 to 200,000 entries.

The rule of thumb for English is about 4 characters or 0.75 words per token. Code, numbers, non-English text and unusual spellings use noticeably more tokens per word, so the same "page" of text can cost twice as much in one language as another.

Tokens are the unit for everything that matters operationally: the context window limit, the max output setting, the price per call and the speed (models generate a fixed-ish number of tokens per second). When something feels odd, such as the model miscounting letters in a word, tokenization is often the reason: it saw "strawberry" as two or three chunks, not ten letters.

Why it matters

You cannot estimate cost, stay under a context limit or chunk documents sensibly without counting tokens. Every budget, limit and dashboard you build later is in tokens.

Picture it

Words to know

  • TokenA chunk of text (often a word piece) mapped to an integer ID.
  • BPEByte Pair Encoding: the common algorithm that builds the vocabulary by merging frequent character pairs.
  • VocabularyThe fixed list of tokens a model knows.

Try it without code

See how text becomes tokens

  1. Open the OpenAI tokenizer page (platform.openai.com/tokenizer) in a browser.
  2. Paste a sentence in English, then the same sentence in another language, then a line of Python.
  3. Compare token counts and notice where words are split.

Use it in code

Count tokens locally with tiktoken, then ask the API how many input tokens it billed.

python
# pip install tiktoken anthropic
import tiktoken
import anthropic

text = "Tokenization decides your cost, your limits and your speed."

enc = tiktoken.get_encoding("cl100k_base")
ids = enc.encode(text)
print(len(ids), "tokens (cl100k):", [enc.decode([i]) for i in ids])

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY
count = client.messages.count_tokens(
    model="claude-sonnet-5-5",
    messages=[{"role": "user", "content": text}],
)
print("Billed input tokens:", count.input_tokens)
$ python tokens.py
PRACTICE · TICK IT AFTER THIS

Write a function that takes a file and prints its token count and estimated input cost at your provider’s price. Run it on a README, a Python file and a non-English text and compare.

Common mistakes

  • Using word count or character count as if it were token count.
  • Assuming every model uses the same tokenizer. Counts differ between providers.
  • Forgetting that the system prompt and chat history are tokens too, on every call.

Go deeper