2026
Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
ICML 2026poster
Byte-level tokenization enables language models to handle any Unicode input, but models can generate invalid UTF-8 sequences when encountering rare or unseen characters. We investigate the relationship between training scale and UTF-8 generation reliability with a 355M parameter model trained on 80B…