Marco Cognetta

2 accepted papers

2024

Distributional Properties of Subword Regularization

Marco Cognetta, Vilém Zouhar, Naoaki Okazaki

EMNLP 2024main

Subword regularization, used widely in NLP, improves model performance by reducing the dependency on exact tokenizations, augmenting the training corpus, and exposing the model to more unique contexts during training. BPE and MaxMatch, two popular subword tokenization schemes, have stochastic dropou…

Cited by 0 Source ↗PDF ↗Scholar ↗

2024

Two Counterexamples to Tokenization and the Noiseless Channel

Marco Cognetta, Vilém Zouhar, Sangwhan Moon, Naoaki Okazaki

COLING 2024main

In Tokenization and the Noiseless Channel (Zouhar et al., 2023), Rényi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer which leads to the highest Rényi efficiency of the unigram distribution should be chosen. The Rényi efficiency is thus tre…

Cited by 4 Source ↗PDF ↗code ↗Scholar ↗