TOPO-Bench: An Open-Source Topological Mapping Evaluation Framework with Quantifiable Perceptual Aliasing

Abstract

Topological mapping offers a compact and robust representation for navigation, but progress in the field is hindered by the lack of standardized evaluation metrics, datasets, and protocols. Existing systems are evaluated in different environments under different criteria, preventing fair and reproducible comparison. Moreover, a key challenge---perceptual aliasing---remains under-quantified despite its strong influence on system performance. We address these gaps by (i) formalizing emph{topological consistency} as the fundamental property of topological maps and showing that, under mild assumptions, localization accuracy provides an efficient and interpretable surrogate metric, and (ii) introducing, to our knowledge, the first quantitative measure of dataset ambiguity for fair comparison across environments. To support this protocol, we curate a diverse benchmark dataset with calibrated ambiguity levels, implement and release deep learning-based baseline systems, and evaluate them alongside classical methods. Our experiments provide new insights into the limitations of current approaches under perceptual aliasing. All datasets, baselines, and evaluation tools are publicly released to foster consistent and reproducible research in topological mapping.