Exploring Graph-aware Reasoning and Bidirectional Selection for Vision-Language Navigation

Abstract

Inspired by structured state space models and graph neural network modeling, we proposes a novel graph-aware reasoning (GAR) model to effectively solve the problem between memory utilization efficiency and reasoning navigation. First, we introduce graph networks into the navigation framework to enhance the model modeling ability for long sequence dependencies. It is integration graph neural networks into the state space to ensures that the agent can accurately find topological paths based on instructions and environmental interactions. Then, we design a bidirectional selective state space model to enhance the agent-aware of spatial information. When without relying on the attention network, we capture the context information in the image through position embedding. Finally, we fuison the features in the bidirectional state space and effectively compresses and improves the fine-grained image features by a residual connection methods. Our experimental results on R2R and REVERIE datasets show that GAR reduces GPU memory usage by 4.45% compared with the DUET baseline model.