Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metrics
Théo Gigant, Camille Guinaudeau, Marc Decombas, Frederic Dufaux
Abstract
Automatic metrics are used as proxies to evaluate abstractive summarization systems when human annotations are too expensive. To be useful, these metrics should be fine-grained, show a high correlation with human annotations, and ideally be independant of reference quality; however, most standard evaluation metrics for summarization are reference-based, and existing reference-free metrics correlates poorly with relevance, especially on summaries of longer documents. In this paper, we introduce a reference-free metric that correlates well with human evaluated relevance, while being very cheap to compute. We show that this metric can also be used along reference-based metrics to improve their robustness in low quality reference settings.
BibTeX
@inproceedings{gigant-etal-2024-mitigating,
title = "Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metrics",
author = "Gigant, Th{\'e}o and
Guinaudeau, Camille and
Decombas, Marc and
Dufaux, Frederic",
editor = "Al-Onaizan, Yaser and
Bansal, Mohit and
Chen, Yun-Nung",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-main.1078/",
doi = "10.18653/v1/2024.emnlp-main.1078",
pages = "19355--19368"
}