Tokenization.

Tokenization is the preprocessing step that converts raw text into a sequence of tokens—the smallest units an AI model works with. Tokens can be individual words, subwords, characters, or byte-pair encodings. The phrase 'SEO strategy' might become three tokens: 'SE', 'O', 'strategy', depending on the tokenizer.

Why Tokenization Matters

The size of a model's context window is measured in tokens. Understanding tokenization helps explain why models sometimes struggle with very long documents, why costs scale with input length, and why some languages are more expensive to process (languages with fewer training examples have less efficient tokenizers).

Practical Implications

For SEO and content purposes, tokenization explains why semantic relationships matter more than exact keyword strings. Two sentences using different words can have nearly identical token-level representations if they express the same idea, which is why synonyms and related concepts influence rankings alongside exact match terms.