SAGENet: Binaural Echo-Based 3D Depth Estimation with Sparse Angular Queries and Refined Geometric Cues

Abstract

In this paper, we propose SAGENet that utilizes only binaural echoes (i.e., for scenarios when vision perception seriously degrades) for scene depth estimation. Unlike previous methods that implicitly learn spatial features from echoes, which may cause shape and scale drift, SAGENet explicitly extracts spatial cues, effectively enhancing depth estimation accuracy. First, we leverage signal processing to generate coarse 2D geometric cues, which contain scene scale and shape information, as additional input for the 3D depth estimation network. This approach aids the network in better reconstructing depth information from the scene. Given the substantial noise in the 2D geometric cues, we design a geometric cue consistency denoising loss function to help the network accurately interpret the scale and shape information embedded in the features. Second, we initialize learnable queries with angular spectrum peaks and fuse them with audio features via self-attention to guide the network to focus on the first few reflections echo dominant feature, while effectively suppressing interference from reverberation. Finally, Our experimental results on the Replica and real-world BatVision datasets show that the proposed method outperforms the existing binaural echo-based methods (including BatVision) by more than 5% and 10% in absolute relative error, respectively. To benefit the community, we open-source the code at https://github.com/zjuersdsd/SAGENet.git.