MFT: Modal Fusion Transformer for Cross-Modal Fusion in 3D Object Detection

Abstract

Increasing attention has been garnered by LiDAR points and multi-view images fusion based on Transformer for supplementing another modality in 3D object detection. However, challenges persist for cross-modal fusion methods due to the heterogeneity of these two modalities, leading to issues such as inaccurate detection results encountered by Transformer-based methods. In this work, a one-way mid-level fusion based framework for 3D object detection named Modal Fusion Transformer (MFT) using LiDAR points and multi-view images is introduced. It comprises a Depth-Guided Generation(DGG) module, Position Encoding Generation (PEG) module and Cross Modal Fusion(CMF) module. Specifically, depth information from point cloud is utilized for both gathering the image depth map and initializing object queries in DGG. PEG unifies the form of position encoding from LiDAR features and multi-view image features. Depth and position information of object from images is aggregated to point clouds by CMF, which fully explores dual-modal information. Furthermore, a Modal Fusion Network with deformable attention named fast-MFT is introduced to reduce the relatively large computational cost associated with global attention. Our MFT and fast-MFT achieve competitive performance while maintaining a faster inference speed than other models.