TEXT-CAKE: Challenging Language Models on Local Text Coherence
Luca Dini, Dominique Brunato, Felice Dell’Orletta, Tommaso Caselli
Abstract
We present a deep investigation of encoder-based Language Models (LMs) on their abilities to detect text coherence across four languages and four text genres using a new evaluation benchmark, TEXT-CAKE. We analyze both multilingual and monolingual LMs with varying architectures and parameters in different finetuning settings. Our findings demonstrate that identifying subtle perturbations that disrupt local coherence is still a challenging task. Furthermore, our results underline the importance of using diverse text genres during pre-training and of an optimal pre-traning objective and large vocabulary size. When controlling for other parameters, deep LMs (i.e., higher number of layers) have an advantage over shallow ones, even when the total number of parameters is smaller.
BibTeX
@inproceedings{dini-etal-2025-text,
title = "{TEXT}-{CAKE}: Challenging Language Models on Local Text Coherence",
author = "Dini, Luca and
Brunato, Dominique and
Dell{'}Orletta, Felice and
Caselli, Tommaso",
editor = "Rambow, Owen and
Wanner, Leo and
Apidianaki, Marianna and
Al-Khalifa, Hend and
Eugenio, Barbara Di and
Schockaert, Steven",
booktitle = "Proceedings of the 31st International Conference on Computational Linguistics",
month = jan,
year = "2025",
address = "Abu Dhabi, UAE",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.coling-main.296/",
pages = "4384--4398"
}