Recommendation Navigation Based on User Information Using VLM

Abstract

In this paper, we propose a novel recommendation-based path planning system that leverages VLM and LLM to interpret user intentions. The system infers user preferences through both conversational and behavioral data, thereby delivering personalized navigation and guidance services within complex consumer environments. The LLM component is designed to deduce user intent even in the absence of direct item references by utilizing higher-level conceptual cues, while the VLM component analyzes images of user behavior to extract contextual information. A virtual museum simulation was implemented using Isaac Sim, and a metadata dataset for exhibits was constructed to validate the system’s performance. Experimental results demonstrate that the proposed system effectively interprets user intent and generates optimized pathways. Future work will focus on extending the system to consumer spaces such as department stores and supermarkets—areas where conventional 2D semantic maps are inadequate—by exploring topology-based mapping solutions. Ultimately, this research aims to revolutionize user experience by enabling personalized robotic services in consumer environments.