Towards Human-aligned Evaluation for Linear Programming Word Problems
Linzi Xing, Xinglu Wang, Yuxi Feng, Zhenan Fan, Jing Xiong, Zhijiang Guo, Xiaojin Fu, Rindra Ramamonjison
Abstract
Math Word Problem (MWP) is a crucial NLP task aimed at providing solutions for given mathematical descriptions. A notable sub-category of MWP is the Linear Programming Word Problem (LPWP), which holds significant relevance in real-world decision-making and operations research. While the recent rise of generative large language models (LLMs) has brought more advanced solutions to LPWPs, existing evaluation methodologies for this task still diverge from human judgment and face challenges in recognizing mathematically equivalent answers. In this paper, we introduce a novel evaluation metric rooted in graph edit distance, featuring benefits such as permutation invariance and more accurate program equivalence identification. Human evaluations empirically validate the superior efficacy of our proposed metric when particularly assessing LLM-based solutions for LPWP.
BibTeX
@inproceedings{xing-etal-2024-towards,
title = "Towards Human-aligned Evaluation for Linear Programming Word Problems",
author = "Xing, Linzi and
Wang, Xinglu and
Feng, Yuxi and
Fan, Zhenan and
Xiong, Jing and
Guo, Zhijiang and
Fu, Xiaojin and
Ramamonjison, Rindra and
Mostajabdaveh, Mahdi and
Han, Xiongwei and
Zhou, Zirui and
Zhang, Yong",
editor = "Calzolari, Nicoletta and
Kan, Min-Yen and
Hoste, Veronique and
Lenci, Alessandro and
Sakti, Sakriani and
Xue, Nianwen",
booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
month = may,
year = "2024",
address = "Torino, Italia",
publisher = "ELRA and ICCL",
url = "https://aclanthology.org/2024.lrec-main.1438/",
pages = "16550--16556"
}