Topological Planning With Transformers for Vision-and-Language Navigation

Kevin Chen; Junshen K. Chen; Jo Chuang; Marynel Vazquez; Silvio Savarese

CVPR 2021poster129 citations

Topological Planning With Transformers for Vision-and-Language Navigation

Kevin Chen, Junshen K. Chen, Jo Chuang, Marynel Vazquez, Silvio Savarese

Abstract

Conventional approaches to vision-and-language navigation (VLN) are trained end-to-end but struggle to perform well in freely traversable environments. Inspired by the robotics community, we propose a modular approach to VLN using topological maps. Given a natural language instruction and topological map, our approach leverages attention mechanisms to predict a navigation plan in the map. The plan is then executed with low-level actions (e.g. forward, rotate) using a robust controller. Experiments show that our method outperforms previous end-to-end approaches, generates interpretable navigation plans, and exhibits intelligent behaviors such as backtracking.

BibTeX

@inproceedings{cvpr2021_topologicalplann,
  title = {Topological Planning With Transformers for Vision-and-Language Navigation},
  author = {Kevin Chen and Junshen K. Chen and Jo Chuang and Marynel Vazquez and Silvio Savarese},
  booktitle = {CVPR 2021},
  year = {2021}
}