Tokenization: The Part of LLMs Nobody Explains Well

Why Tokenization Exists

LLMs cannot process raw text directly. They need a fixed-size discrete vocabulary to predict over. Tokenization converts text into a sequence of integers from a vocabulary of fixed size.

The choice of tokenization scheme has significant implications for what LLMs can and cannot do well.

Byte Pair Encoding (BPE): The Standard Approach

Modern LLMs use BPE (or variants like WordPiece or SentencePiece). BPE starts with individual characters as tokens, then iteratively merges the most common adjacent pairs until reaching the desired vocabulary size (typically 32,000–100,000 tokens).

The result: common words become single tokens. Rare words are split into subword tokens. Unknown words are split into characters.

pythonCopy
import tiktoken

enc = tiktoken.encoding_for_model("gpt-4")

text = "The transformer architecture revolutionized NLP"
tokens = enc.encode(text)
print(tokens)
# Output: [976, 47503, 10886, 91310, 291, 452, 47, 6484]

token_strings = [enc.decode([t]) for t in tokens]
print(token_strings)
# Output: ['The', ' transformer', ' architecture', ' revolution', 'ized', ' NLP']
# Note: "revolutionized" splits into "revolution" + "ized"

Why This Causes Surprising Failures

Arithmetic

Numbers are tokenized inconsistently. 123 might be one token. 1234 might be two tokens (12 and 34). 12345 might tokenize differently still. The model never sees digits - it sees token IDs that happen to correspond to strings of digits.

This is why LLMs struggle with arithmetic: they are pattern-matching over token sequences, not computing over numbers.

Letter Counting

"How many r's in 'strawberry'?" seems trivially easy. But the model sees tokens, not characters. "strawberry" might tokenize as ["st", "raw", "berry"]. The model cannot introspect individual characters without special handling.

Non-English Text

BPE tokenizes based on frequency in the training corpus. English text tokenizes efficiently (most common words = 1 token). Less-represented languages tokenize into many small subword tokens, consuming more context window space and sometimes degrading performance.

pythonCopy
# Compare tokenization efficiency across languages
texts = {
    "English": "The quick brown fox",
    "Swahili": "Mbweha wa kahawia anakimbia haraka",
    "Chinese": "那只快速的棕色狐狸",
}
for lang, text in texts.items():
    tokens = enc.encode(text)
    print(f"{lang}: {len(tokens)} tokens for '{text}'")

Context Window Is Measured in Tokens, Not Words

When a model has a 200,000-token context window, that does not translate to a fixed number of words. English text averages roughly 0.75 tokens per word (common words tokenize efficiently). Code tends to use more tokens per "word." Languages with rich morphology may use many tokens per word.

This has practical implications: a document that is 50,000 words in English might be 35,000 tokens, well within a 100K context window. The same document translated into a less-commonly-tokenized language might consume significantly more tokens.

Estimating Token Count Before Sending

API costs are per token. Estimate before calling:

pythonCopy
import tiktoken

def estimate_tokens(text: str, model: str = "gpt-4") -> int:
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))

# For Anthropic models, token counts are similar
# Anthropic's client has a count_tokens method:
# client.count_tokens(text)

document = open("long_document.txt").read()
token_count = estimate_tokens(document)
# Token pricing varies by model and changes frequently.
# Check your provider's current pricing page for accurate rates.
print(f"Estimated tokens: {token_count:,}")

Practical Implications for Prompt Engineering

• Instructions before content: The model gives slightly more attention to tokens near the start and end of the context. Put critical instructions at the beginning of the system prompt.
• Avoid unnecessary text: Every token costs money and consumes context window. Trim system prompts of redundant phrasing.
• For multi-language systems: Test tokenization efficiency for your target languages. The context window is smaller in effective capacity for less-common languages.
• For arithmetic: Use code interpreter or a tool call for any computation, not the LLM's raw generation.

What to Practice Next

• Install tiktoken and tokenize five sentences that contain technical jargon, code snippets, and non-English words - count the tokens and explain why token counts differ from word counts for each example.
• Find a prompt where tokenization causes a surprising failure (e.g., a model that counts letters incorrectly or mishandles a word with unusual BPE boundaries) and explain the root cause in terms of how BPE merges were learned.
• Estimate the cost and context window usage for a real document you want to process using an LLM - use tiktoken to get the exact token count and calculate the API cost at current pricing.