COLING 2025main2 citations
Beyond Surprisal: A Dual Metric Framework for Lexical Skill Acquisition in LLMs
Nazanin Shafiabadi, Guillaume Wisniewski
Abstract
Many studies have explored when and how LLMs learn to use specific words, primarily by examining their learning curves. While these curves capture a model’s capacity to use words correctly in context, they often neglect the equally important skill of avoiding incorrect usage. In this paper, we introduce a new metric, anti-surprisal, which measures a model’s capacity to refrain from using words in inappropriate or unexpected contexts. By examining both correct usage and error avoidance, we offer a more comprehensive perspective on the learning dynamics of LLMs.
BibTeX
@inproceedings{shafiabadi-wisniewski-2025-beyond,
title = "Beyond Surprisal: A Dual Metric Framework for Lexical Skill Acquisition in {LLM}s",
author = "Shafiabadi, Nazanin and
Wisniewski, Guillaume",
editor = "Rambow, Owen and
Wanner, Leo and
Apidianaki, Marianna and
Al-Khalifa, Hend and
Eugenio, Barbara Di and
Schockaert, Steven",
booktitle = "Proceedings of the 31st International Conference on Computational Linguistics",
month = jan,
year = "2025",
address = "Abu Dhabi, UAE",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.coling-main.443/",
pages = "6636--6641"
}