Anton Schäfer

On the Effect of (Near) Duplicate Subwords in Language Modelling

Tokenisation is a core part of language models (LMs). It involves splitting a character sequence into subwords which are assigned random indices before being served to the LM. However, this process—while typically lossless—may lead to less efficient LM training, because it removes character-level in…