LEGOBench: Scientific Leaderboard Generation Benchmark
Shruti Singh, Shoaib Alam, Husain Malwat, Mayank Singh
Abstract
The ever-increasing volume of paper submissions makes it difficult to stay informed about the latest state-of-the-art research. To address this challenge, we introduce LEGOBench, a benchmark for evaluating systems that generate scientific leaderboards. LEGOBench is curated from 22 years of preprint submission data on arXiv and more than 11k machine learning leaderboards on the PapersWithCode portal. We present a language model-based and four graph-based leaderboard generation task configuration. We evaluate popular encoder-only scientific language models as well as decoder-only large language models across these task configurations. State-of-the-art models showcase significant performance gaps in automatic leaderboard generation on LEGOBench. The code is available on GitHub and the dataset is hosted on OSF.
BibTeX
@inproceedings{singh-etal-2024-legobench,
title = "{LEGOB}ench: Scientific Leaderboard Generation Benchmark",
author = "Singh, Shruti and
Alam, Shoaib and
Malwat, Husain and
Singh, Mayank",
editor = "Al-Onaizan, Yaser and
Bansal, Mohit and
Chen, Yun-Nung",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-emnlp.855/",
doi = "10.18653/v1/2024.findings-emnlp.855",
pages = "14598--14613"
}