Sim2Real Domain Shifting: Hyper-Realistic Data Generation for Object Segmentation

Abstract

Object segmentation is a critical prerequisite for robotic tasks such as grasping and assembly. While high accuracy and reliability typically require extensive real-world data, its collection and annotation are costly. Although synthetic data generated through physically-based rendering mitigates this need, a persistent domain gap hinders model performance. This paper introduces a novel hyper-realistic synthetic data generation method to mitigate this gap with minimal real-world data. By extracting domain information from limited real scenes, we shift synthetic data toward the target domain. Realistic backgrounds are synthesized using generative models, while a two-stage style transfer, guided by anchor image styles, adapts foregrounds. Our method achieves performance comparable to models trained on thousands of real images using as few as one real image, significantly reducing the reliance on large-scale data collection.