Transitive Inference in Large Language Models and Prompting Intervention

Abstract

Transitive inference (TI) is a critical form of deductive reasoning, essential to both human and animal cognition. This study explores whether state-of-the-art large language models (LLMs) possess TI capabilities and examines the impact of two emerging prompting methods on model performance. Four LLMs—GPT-3.5-Turbo, GPT-4, Llama3-8B, and Qwen—are evaluated using a TI task involving a 10-item hierarchy. Results indicate that these models demonstrate solid performance, along with human-like behavioral effects such as the symbolic distance effect, terminal item effect, and context effect. The sequence of input premises significantly affects model accuracy, with all models showing a preference for the chain condition over the jump condition. Two prompting methods — Model Confidence prompts (likelihood tests) and Chain-of-Thought prompts—are applied in order to further enhance TI performance. While GPT-4 benefits the most from these prompts, other models experience a decline in performance. Additionally, behavioral biases like the terminal item effect persist and are even amplified following prompt adjustments. Therefore, specific prompting methods may not be universally effective across different models, and in some cases, may cause adverse effects. Further research is needed to better understand the nature of TI in LLMs.