FOLIO: Natural Language Reasoning with First-Order Logic
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng
Abstract
Large language models (LLMs) have achieved remarkable performance on a variety of natural language understanding tasks. However, existing benchmarks are inadequate in measuring the complex logical reasoning capabilities of a model. We present FOLIO, a human-annotated, logically complex and diverse dataset for reasoning in natural language (NL), equipped with first-order logic (FOL) annotations. FOLIO consists of 1,430 examples (unique conclusions), each paired with one of 487 sets of premises used to deductively reason for the validity of each conclusion. The logical correctness of the premises and conclusions is ensured by their FOL annotations, which are automatically verified by an FOL inference engine. In addition to the main NL reasoning task, NL-FOL pairs in FOLIO constitute a new NL-FOL translation dataset. Our experiments on FOLIO systematically evaluate the FOL reasoning ability of supervised fine-tuning on medium-sized language models. For both NL reasoning and NL-FOL translation, we benchmark multiple state-of-the-art language models. Our results show that a subset of FOLIO remains a challenge for one of the most capable Large Language Model (LLM) publicly available, GPT-4.
BibTeX
@inproceedings{han-etal-2024-folio,
title = "{FOLIO}: Natural Language Reasoning with First-Order Logic",
author = "Han, Simeng and
Schoelkopf, Hailey and
Zhao, Yilun and
Qi, Zhenting and
Riddell, Martin and
Zhou, Wenfei and
Coady, James and
Peng, David and
Qiao, Yujie and
Benson, Luke and
Sun, Lucy and
Wardle-Solano, Alexander and
Szab{\'o}, Hannah and
Zubova, Ekaterina and
Burtell, Matthew and
Fan, Jonathan and
Liu, Yixin and
Wong, Brian and
Sailor, Malcolm and
Ni, Ansong and
Nan, Linyong and
Kasai, Jungo and
Yu, Tao and
Zhang, Rui and
Fabbri, Alexander and
Kryscinski, Wojciech Maciej and
Yavuz, Semih and
Liu, Ye and
Lin, Xi Victoria and
Joty, Shafiq and
Zhou, Yingbo and
Xiong, Caiming and
Ying, Rex and
Cohan, Arman and
Radev, Dragomir",
editor = "Al-Onaizan, Yaser and
Bansal, Mohit and
Chen, Yun-Nung",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-main.1229/",
doi = "10.18653/v1/2024.emnlp-main.1229",
pages = "22017--22031"
}