Bootstrapping LLM-based Fact-checking via Iterative Rationalization Finetuning

Abstract

Fact-checking, the task of reasoning about a claim’s truthfulness based on evidence, has become increasingly crucial with the rapid spread of misinformation. In real-world scenarios, fact-checking often involves checking complex claims necessitating multi-step reasoning, thus imposing a high requirement for a model’s autonomous ability. LLM-based fact-checking performs multi-step reasoning through generating natural language rationales. However, it is susceptible to error propagation problems, which means any error occurring inside rationales will result in an incorrect label. To this end, we propose an iterative rationalization finetuning approach to address this issue, enhancing the LLM’s ability by finetuning it with high-quality rationales. Specifically, we guide the generation of high-quality rationales using golden labels and, inversely, utilize these to finetune the LLM itself, thus constructing a self-improvement cycle. We demonstrate the effectiveness of the proposed method on HoVer and FEVEROUS-S benchmarks, where it achieves state-of-the-art performance, particularly in multi-step reasoning scenarios.