COLING 2022main1 citations
PrefScore: Pairwise Preference Learning for Reference-free Summarization Quality Assessment
Ge Luo, Hebi Li, Youbiao He, Forrest Sheng Bao
Abstract
Evaluating machine-generated summaries without a human-written reference summary has been a need for a long time. Inspired by preference labeling in existing work of summarization evaluation, we propose to judge summary quality by learning the preference rank of summaries using the Bradley-Terry power ranking model from inferior summaries generated by corrupting base summaries. Extensive experiments on several datasets show that our weakly supervised scheme can produce scores highly correlated with human ratings.
BibTeX
@inproceedings{luo-etal-2022-prefscore,
title = "{P}ref{S}core: Pairwise Preference Learning for Reference-free Summarization Quality Assessment",
author = "Luo, Ge and
Li, Hebi and
He, Youbiao and
Bao, Forrest Sheng",
editor = "Calzolari, Nicoletta and
Huang, Chu-Ren and
Kim, Hansaem and
Pustejovsky, James and
Wanner, Leo and
Choi, Key-Sun and
Ryu, Pum-Mo and
Chen, Hsin-Hsi and
Donatelli, Lucia and
Ji, Heng and
Kurohashi, Sadao and
Paggio, Patrizia and
Xue, Nianwen and
Kim, Seokhwan and
Hahm, Younggyun and
He, Zhong and
Lee, Tony Kyungil and
Santus, Enrico and
Bond, Francis and
Na, Seung-Hoon",
booktitle = "Proceedings of the 29th International Conference on Computational Linguistics",
month = oct,
year = "2022",
address = "Gyeongju, Republic of Korea",
publisher = "International Committee on Computational Linguistics",
url = "https://aclanthology.org/2022.coling-1.515/",
pages = "5896--5903"
}