RCVAFusion: 4D Radar and Camera Fusion With Virtual Points Association for 3D Object Detection

Abstract

4D millimeter-wave radar plays a critical role in object detection for autonomous driving and robotics under all-weather and all-lighting conditions. Recently, the virtual-point-based approaches have attracted widespread attention due to their ability to address radar data sparsity by complementing the depth of image instance points with the nearest 3D points. However, existing radar-camera fusion methods based on virtual points simply incorporate virtual points into the raw radar points as a form of data augmentation, overlooking the potential of virtual points that have an inherent association with both radar and images. To address these issues, we present a novel radar-camera fusion network, RCVAFusion, for 3D object detection. Specifically, we first design an association branch that employs Object Area Sampling (OAS) and Virtual-Raw Points Depth Lifting (VRPDL). This branch facilitates a deep interaction between radar geometric features and image semantic features through the medium of virtual points to generate an association feature. Then, we introduce the Dual-step Feature Aggregation (DFA) to promote feature fusion from radar, image, and association branches by establishing aggregation priorities based on feature similarity in two steps. Experimental results on the TJ4DRadSet and View-of-Delft (VoD) datasets demonstrate that our method efficiently fuses radar and camera through virtual points and achieves state-of-the-art performance.