VecNav: Vector Goal Robot Navigation from In-the-wild Videos

Abstract

We propose VecNav, a novel approach that trains a monocular navigation model through self-supervision using uncalibrated, human-captured videos. These videos, characterized by unknown camera intrinsics and extrinsics, are readily available from video-sharing platforms (e.g. YouTube) and are referred to as "in-the-wild" videos due to their unregulated capture conditions. Our approach involves estimating ground truth trajectories from these videos using monocular visual odometry. We then train a transformer-based diffusion policy that takes a goal specified by a vector and RGB images as input and generates action predictions. Our method leverages a significantly larger and more diverse dataset compared to existing monocular visual navigation approaches. This diversity holds the potential to develop a generalist navigation model capable of guiding various types of robots in unfamiliar environments. We evaluated our method on a differential drive robot, demonstrating its capability to effectively navigate using solely "in-the-wild" videos for training. Our experiments demonstrate that VecNav successfully learned to act based on visual affordances, relying solely on uncalibrated "in-the-wild" data.