2024
BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training
EMNLP 2024main
Language models can greatly benefit from efficient tokenization. However, they still mostly utilize the classical Byte-Pair Encoding (BPE) algorithm, a simple and reliable method. BPE has been shown to cause such issues as under-trained tokens and sub-optimal compression that may affect the downstre…