A probabilistic framework for intrinsic image decomposition from RGB-D streams

Abstract

Lighting and shading can have a great impact on a robot's ability to recognize, match, and classify objects in indoor scenes. Additionally, the realism of augmented reality applications benefit greatly from understanding the lighting and shading in a scene. To aid in extracting this information from a mobile platform we introduce a novel framework to solve the intrinsic image decomposition problem for RGB-D streams. In our pipeline, the task is formulated as a Bayesian estimation problem. Compared to frame-based methods that must solve a full conditional random field (CRF) optimization problem at each time step, our framework can utilize the knowledge of past frames to predict the intrinsic images at a given frame. Our approach produces more reliable and consistent predictions over time, and our filtering-based framework achieves significant performance gains. Furthermore, our framework can be easily integrated into standard perception loops in many robotic systems that use a similar recursive filtering structure. We show qualitative results on real data and generate quantitative results using ground truth from a photorealistic synthetic dataset produced using a state-of-the-art ray tracer and high fidelity 3D model.