Inference Retrieval-Augmented Multi-Modal Chain-of-Thoughts Reasoning for Language Models

Abstract

Recent advancements in Large Language Models (LLMs) have catalyzed the exploration of Chain of Thought (CoT) approaches, particularly in extending their application to multimodal tasks to enhance reasoning capabilities. However, current studies often fail to fully leverage the inferential capabilities of these models, as they primarily focus on selecting similar questions or images rather than exploring identical inferences, thereby limiting the potential for analogical learning. In this paper, we propose an inference retrieval-augmented method that incorporates two strategies to identify training set examples with inferential processes similar to those of the target problem. Initially, we fine-tune a vanilla model to generate the pseudo-inference relevant to the question, which is then utilized to retrieve analogous examples. Next, we hypothesize that questions sharing similar contexts may share inferences, thus retrieving examples based on similarity in contextual questions. Ultimately, we construct multiple reasoning pathways from the retrieved examples and employ a voting mechanism to determine the most frequent answer. Our method surpasses all few-shot approaches and most supervised methods on the ScienceQA dataset, achieving an accuracy of 87.37% with ChatGPT and exceeding the human benchmark in several categories.