Efficient Joint Estimation of Optical Flow and Stereo Disparity With Event Cameras

Abstract

Optical flow and stereo disparity, both are fundamental in the perception pipeline of robotic systems, enabling 3D understanding and motion estimation of the robot itself and the dynamic objects around. In this context, event cameras offer great potential to reduce latency and improve the efficiency of the perception pipeline. However, existing event-based approaches often deal with flow and disparity estimation tasks separately, potentially leading to redundant computations and reduced efficiency. In this work, we developed a joint flow and depth estimation network, featuring shared feature encoders for high computational efficiency. Moreover, we introduce a novel Bidirectional Mamba module to enhance feature expressiveness by increasing the spatial receptive field to capture global context with significantly lower computational overhead than vision transformers. We further improve efficiency by incorporating <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">informed priors</i> to reduce the number of refinement iterations in the sequential estimation task. Additionally, we extend our framework to combine the estimated flow and disparity to predict 3D scene flow, an important task in many robotics applications.