Déjà Vu: Unlocking Transparent Action Reasoning for Object-Goal Navigation Via Large Language Models

Abstract

The remarkable interaction and reasoning capabilities of Large Language Models (LLMs) make them promising in collaborative Embodied AI tasks, particularly for Object-goal Navigation (ObjNav) tasks that require both decision-making and transparent explanation. However, existing work mainly uses LLMs as proxy target indicators, leaving the role of direct action decision-making to other components. This separation causes non-transparent action decisions and extra adaptation requirements. This observation prompts us to reconsider their role: Can LLMs be transformed into the central ``brain'' of agents, directly outputting action choices and explaining their reasoning? In pursuit of this inquiry, we decouple perception from action reasoning to focus specifically on the feasibility of deploying LLMs as navigation policies. We introduce the Experience-aware Action Cogitator (proposedmethod) that integrates two kinds of experience, ie, expert-informed experience and trial & error experience, into prompts. Inspired by David Hume's philosophical principles that knowledge is acquired through reflective experience, these experiences are designed for two critical questions: (i) ``What action should be selected as the best option?'' and (ii) ``What actions have been tried but proven suboptimal?'' By analyzing and reflecting on these two types of experience, we show that LLMs can reason navigation actions in unseen environments effectively without costly fine-tuning. Experiments on the widely-adopted iTHOR yield significant improvements in ObjNav performance. These compelling results validate the feasibility of our proposedmethod. Compared to vanilla LLMs, proposedmethod nearly doubles both the Success Rate and the Success weighted by Path Length, reaching peak values of 73.93% and 48.35% in unseen scenes, respectively.