TheSHY-3D: Texture and Structure HarmonY for Multi-View 3D Object Detection

Abstract

The task of 3D perception from multi-view camera images is a crucial concern of research in autonomous driving, such as detecting road vehicles and surrounding pedestrians. Camera images contain rich texture information, as well as potential precise spatial data and relational positioning between objects. However, training that relies solely on image data often limits the performance of end-to-end models. This conflict is particularly prominent in balancing object structure perception and texture details extraction. To address this challenge, we propose TheSHY-3d, a novel architecture designed to fuse texture-rich image features with 3D spatial queries. Specifically, TheSHY-3D leverages a hybrid feature integration module, 3D anchor queries, and a harmonized 3D bounding box prediction module, without 3D data. While balanced performance in vehicle detection highlights the benefits through texture and structure harmony, TheSHY-3D achieves superior performance on the NuScenes with 73.13% NDS and 68.8% mAP.