SynthDrive: Scalable Real2Sim2Real Sensor Simulation Pipeline for High-Fidelity Asset Generation and Driving Data Synthesis

Abstract

In the field of autonomous driving, sensor simulation is essential for generating rare and diverse scenarios that are difficult to capture in real-world environments. Current solutions fall into two categories: 1) CG-based methods, such as CARLA, which lack diversity and struggle to scale to the vast array of rare cases required for robust perception training; and 2) learning-based approaches, such as NeuSim, which are limited to specific object categories (vehicles) and require extensive multi-sensor data, hindering their applicability to generic objects. To address these limitations, we propose SynthDrive, a scalable "real2sim2real" system that leverages 3D generation to automate asset mining, generation, and rare-case data synthesis.Our framework introduces two key innovations: 1) Automated Rare-Case Mining and Synthesis. Given a text prompt describing specific objects, SynthDrive automatically mines image data from the Internet and then generates corresponding high-fidelity 3D assets, which eliminates the need for costly manual data collection. By integrating these assets into existing street-view data, our pipeline produces photorealistic rare-case data, supporting rapid scaling to diverse assets including irregular obstacles and temporary traffic facilities. 2) High-Fidelity 3D Generation. We propose a hybrid asset generation pipeline that combines a geometry-aware LRM, iterative mesh optimization, and an improved texture fusion algorithm. Our approach achieves 0.0164 Chamfer Distance on the GSO dataset, outperforming InstantMesh by 14.1% in geometry accuracy, and achieves 19.05 PSNR (vs 16.84) for texture quality. This enables fine geometry details and high-resolution texture generation, which is essential for perception model training. Experiments demonstrate that SynthDrive-generated data improves the performance of downstream perception tasks (2D and 3D detection on rare objects) by 2-4% mAP. SynthDrive greatly lowers the data production cost and improves the diversity for corner-case data generation, showcasing extensive potential applications in the field of autonomous driving.