Kevin Slagle

SpaceByte: Towards Deleting Tokenization from Large Language Modeling

Tokenization is widely used in large language models because it significantly improves performance. However, tokenization imposes several disadvantages, such as performance biases, increased adversarial vulnerability, decreased character-level modeling performance, and increased modeling complexity.…