Vision-Language Navigation with Continual Learning for Unseen Environments

Abstract

Vision-language navigation (VLN) is a pivotal area within embodied intelligence, where agents must navigate based on natural language instructions. While traditional VLN research has focused on enhancing environmental comprehension and decision-making policy, these methods often reveal substantial performance gaps when agents are deployed in novel environments. This issue primarily arises from the lack of diverse training data. Expanding datasets to encompass a broader range of environments is impractical and costly. To address this challenge, we propose Vision-Language Navigation with Continuous Learning (VLNCL), a framework that allows agents to learn from new environments while preserving previous knowledge incrementally. We introduce a novel dual-loop scenario replay method (Dual-SR) inspired by brain memory mechanisms integrated with VLN agents. This approach helps consolidate past experiences and improves generalization across novel tasks. As a result, the agent exhibits enhanced adaptability to new environments and mitigates catastrophic forgetting. Our experiment demonstrates that VLN agents with Dual-SR effectively resist forgetting and adapt to unfamiliar environments. Combining VLN with continual learning significantly boosts the performance of otherwise average models, achieving SOTA results.