Beyond Waypoints: Semantic-Centric Autonomy with Unreliable Maps through Learned Abstractions

Abstract

Autonomous navigation that relies on precise metric maps is inherently fragile to environmental changes and mapping inaccuracies. These discrepancies often lead to failures in localization and path planning, as the robot's internal representation of the world no longer matches reality. We propose an alternative navigation approach that instead focuses on how a robot interacts with its surroundings rather than its precise metric position. Our core contribution is a learned behavioral vocabulary conditioned on raw sensor data that can be used to compose plans for navigation. Our system transforms LiDAR data into low-dimensional learned embeddings which are clustered to create a set of abstract, human-interpretable behaviors (e.g., along wall, exiting intersection, on bridge). This representation allows the robot to control its behavior with respect to the embedding rather than controlling its state with respect to a specific metric cost function or waypoint, thereby minimizing the impact of map and position inaccuracies. We define the mission as a topological sequence of behavioral clusters on the overhead map, enabling high-level navigation.This approach provides a robust way to decompose the environment into recognizable and actionable states that can reliably compose a plan, even on stale maps with environmental deformations and world changes. Our method achieves higher navigation success under intentional map distortions, with average mission success rates 53 and 55 percentage points higher for short and long term plans respectively when compared to baselines which rely on accurate metric maps.