2025
Language Models over Canonical Byte-Pair Encodings
ICML 2025poster
Modern language models represent probability distributions over character strings as distributions over (shorter) token strings derived via a deterministic tokenizer, such as byte-pair encoding. While this approach is highly effective at scaling up language models to large corpora, its current incar…