Kengatharaiyer Sarveswaran

Egalitarian Language Representation in Language Models: It All Begins with Tokenizers

Tokenizers act as a bridge between human language and the latent space of language models, influencing how language is represented in these models. Despite the dominance of English-Centric (EC) Large Language Models (LLMs), tokenization methods often fail to fairly represent complex scripts like Tam…