Hop-level Direct Preference Optimization for Knowledge Graph Reasoning with Trees

Abstract

Recent advancements in knowledge graph question answering (KGQA) have shown promise, yet existing methods often fail to align with human reasoning patterns. This study proposes HD-PORT (hop-level direct preference optimization for knowledge graph reasoning with trees), a novel approach that combines Monte Carlo Tree Search (MCTS) with hop-level direct preference optimization (HDPO) for KGQA tasks. MCTS simulates human-like reasoning by exploring multiple inference paths in knowledge graphs, generating interpretable reasoning chains and rich hop-level preferences. HDPO then leverages these preferences for model optimization, addressing the limitations of traditional supervised learning and existing DPO methods that focus on overall solution preferences. By optimizing preferences at each reasoning step, HD-PORT more closely mirrors human problem-solving strategies. Experimental results on benchmark datasets demonstrate that HD-PORT significantly outperforms state-of-the-art methods in both accuracy and interpretability, particularly for complex, multi-hop reasoning tasks.