Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers
Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan
Abstract
The increased use of large language models (LLMs) across a variety of real-world applications calls for mechanisms to verify the factual accuracy of their outputs. In this work, we present Factcheck-Bench, a holistic end-to-end framework for annotating and evaluating the factuality of LLM-generated responses, which encompasses a multi-stage annotation scheme designed to yield detailed labels for fact-checking and correcting not just the final prediction, but also the intermediate steps that a fact-checking system might need to take. Based on this framework, we construct an open-domain factuality benchmark in three-levels of granularity: claim, sentence, and document. We further propose a system, Factcheck-GPT, which follows our framework, and we show that it outperforms several popular LLM fact-checkers. We make our annotation tool, annotated data, benchmark, and code available at https://github.com/yuxiaw/Factcheck-GPT.
BibTeX
@inproceedings{wang-etal-2024-factcheck,
title = "Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers",
author = "Wang, Yuxia and
Gangi Reddy, Revanth and
Mujahid, Zain Muhammad and
Arora, Arnav and
Rubashevskii, Aleksandr and
Geng, Jiahui and
Mohammed Afzal, Osama and
Pan, Liangming and
Borenstein, Nadav and
Pillai, Aditya and
Augenstein, Isabelle and
Gurevych, Iryna and
Nakov, Preslav",
editor = "Al-Onaizan, Yaser and
Bansal, Mohit and
Chen, Yun-Nung",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-emnlp.830/",
doi = "10.18653/v1/2024.findings-emnlp.830",
pages = "14199--14230"
}