PCMF2-Net: A Pyramid Cross-Modal Feature Fusion Network for Off-Road Freespace Detection

Abstract

Freespace detection plays an important role in autonomous driving. In recent years, deep learning based freespace detection methods have performed well in urban scenes. However, for off-road scenes, freespace detection poses significant challenges due to the complexity of the scenes and the lack of clear edges. The existing methods have not effectively fused LiDAR data and camera images. In this paper, we propose a Pyramid Cross-Modal Feature Fusion Network (PCMF2-Net) for off-road freespace detection. The dense depth maps are concatenated with RGB images and used as input along with surface normal maps. The dual branch CNN-Transformer encoder combines convolutional neural networks and transformers to extract local and global features from RGBD images and surface normal maps, respectively. Then, in the pyramid cross-modal feature fusion module, the multi-scale and multimodal encoder features are fused in a top-down manner. In addition, we also use an edge segmentation task and a two-step training strategy to further improve performance. Experiments on the off-road freespace detection dataset (ORFD) demonstrate that the proposed PCMF2-Net achieves a competitive result of 93.9% IoU at a speed of 23 Hz.