Realism Assessment for Synthetic Images in Robot Vision through Performance Characterization

Abstract

Synthetic image generation plays a crucial role in the development of robot vision algorithms, circumventing manual data collection. However, the realism of synthetic images could affect the performance of the algorithms when applied in real-world settings. In this study, we propose a framework to quantitatively assess the realism of synthetic images using a set of realism metrics as a means of performance characterization. We use a commercial rendering engine as a test-bed for generating synthetic images and ascertain that a set of rendering parameters affect specific image metrics through statistical hypothesis testing. We demonstrate that this framework can be used to optimize rendering parameter values and generate synthetic datasets with improved performance on downstream robot vision tasks such as instance segmentation.