REFeREE: A REference-FREE Model-Based Metric for Text Simplification

Yichen Huang; Ekaterina Kochmar

COLING 2024main1 citations

REFeREE: A REference-FREE Model-Based Metric for Text Simplification

Abstract

Text simplification lacks a universal standard of quality, and annotated reference simplifications are scarce and costly. We propose to alleviate such limitations by introducing REFeREE, a reference-free model-based metric with a 3-stage curriculum. REFeREE leverages an arbitrarily scalable pretraining stage and can be applied to any quality standard as long as a small number of human annotations are available. Our experiments show that our metric outperforms existing reference-based metrics in predicting overall ratings and reaches competitive and consistent performance in predicting specific ratings while requiring no reference simplifications at inference time.

BibTeX

@inproceedings{huang-kochmar-2024-referee,
    title = "{REF}e{REE}: A {RE}ference-{FREE} Model-Based Metric for Text Simplification",
    author = "Huang, Yichen  and
      Kochmar, Ekaterina",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1200/",
    pages = "13740--13753"
}