Evaluating the Morphosyntactic Well-formedness of Generated Texts
Adithya Pratapa, Antonios Anastasopoulos, Shruti Rijhwani, Aditi Chaudhary, David R. Mortensen, Graham Neubig, Yulia Tsvetkov
Abstract
Text generation systems are ubiquitous in natural language processing applications. However, evaluation of these systems remains a challenge, especially in multilingual settings. In this paper, we propose L’AMBRE – a metric to evaluate the morphosyntactic well-formedness of text using its dependency parse and morphosyntactic rules of the language. We present a way to automatically extract various rules governing morphosyntax directly from dependency treebanks. To tackle the noisy outputs from text generation systems, we propose a simple methodology to train robust parsers. We show the effectiveness of our metric on the task of machine translation through a diachronic study of systems translating into morphologically-rich languages.
BibTeX
@inproceedings{pratapa-etal-2021-evaluating,
title = "Evaluating the Morphosyntactic Well-formedness of Generated Texts",
author = "Pratapa, Adithya and
Anastasopoulos, Antonios and
Rijhwani, Shruti and
Chaudhary, Aditi and
Mortensen, David R. and
Neubig, Graham and
Tsvetkov, Yulia",
editor = "Moens, Marie-Francine and
Huang, Xuanjing and
Specia, Lucia and
Yih, Scott Wen-tau",
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2021",
address = "Online and Punta Cana, Dominican Republic",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.emnlp-main.570/",
doi = "10.18653/v1/2021.emnlp-main.570",
pages = "7131--7150"
}