Craig W. Schmidt — accepted AI-conference papers

2024

Greed is All You Need: An Evaluation of Tokenizer Inference Methods

Omri Uzan, Craig W. Schmidt, Chris Tanner, Yuval Pinter

ACL 2024short

While subword tokenizers such as BPE and WordPiece are typically used to build vocabularies for NLP models, the method of decoding text into a sequence of tokens from these vocabularies is often left unspecified, or ill-suited to the method in which they were constructed. We provide a controlled ana…

Cited by 12 Source ↗PDF ↗code ↗Scholar ↗

2024

Tokenization Is More Than Compression

Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter +1

EMNLP 2024main

Tokenization is a foundational step in natural language processing (NLP) tasks, bridging raw text and language models. Existing tokenization approaches like Byte-Pair Encoding (BPE) originate from the field of data compression, and it has been suggested that the effectiveness of BPE stems from its a…

Cited by 34 Source ↗PDF ↗code ↗Scholar ↗