See how LLMs break text into tokens
Byte Pair Encoding (BPE) starts with individual characters and iteratively merges the most frequent pair of adjacent tokens. This builds a vocabulary of subword units that balances between character-level and word-level tokenization.
WordPiece (used by BERT and similar models) is similar to BPE but differs in how it selects which pair to merge.
Instead of picking the most frequent pair, WordPiece picks the pair that maximises the likelihood of the training data. Specifically, it scores each pair (A, B) as:
score(A, B) = freq(AB) / (freq(A) × freq(B))
This means it prefers merging pairs where the combination is common relative to how common the individual pieces are. A rare pair of rare tokens may score higher than a frequent pair of frequent tokens.
WordPiece also uses a ## prefix to indicate that a token is a continuation of a previous token (i.e. not the start of a word). For example, "tokenizer" might become ["token", "##izer"].
Character-level tokenization splits text into individual characters. It has a tiny vocabulary (just the characters in the language) but produces very long sequences.
Type in the input above to see character-level tokenization below:
Whitespace tokenization simply splits on whitespace boundaries. It is the simplest approach but cannot handle unknown words or morphological variations.
Type in the input above to see whitespace tokenization below: