StreamCMT: Prior-Guided Multimodal Temporal Fusion for Sparse 3D Object Detection

Abstract

Multimodal 3D detection is critical for autonomous driving reliability. While most existing methods boost accuracy via elaborate networks, they neglect inference speed which is essential for real-world deployment. Although existing decoder-based sparse query detection methods offer advantages in real-time performance, they suffer from limitations in convergence speed and cross-modal feature integration. To address these challenges of slow convergence and inadequate feature fusion, this paper proposes a Prior-Guided Position Embedding Module based on the Cross Modal Transformer (CMT) framework. The module reconstructs 3D sampling point distribution through spatial geometric priors, effectively improving model accuracy and accelerating convergence without incurring additional computational overhead. Concurrently, to enhance motion awareness, we integrate a Temporal Fusion Module that leverages historical frame information to optimize current detection performance. Experimental results demonstrate that StreamCMT achieves a detection accuracy of 72.5% NDS and 69.6% mAP on the nuScenes test set. On the validation set, compared to the baseline model, it improves NDS and mAP by 1.0% and 1.1% respectively, while increasing inference speed from 12.0 to 14.4 FPS. The model maintains a lightweight architecture while achieving an effective trade-off between detection accuracy and inference efficiency for autonomous driving perception systems.