LERPS: Lighting Estimation and Relighting for Photometric Stereo

Abstract

Photometric stereo is a method to obtain surface normals of an object using its images captured under varying illumination directions. The existing deep learning-based methods require multiple images of an object captured using complex image acquisition systems. In this work, we propose a deep learning framework to perform three tasks jointly: (i) lighting estimation, (ii) image relighting, and (iii) surface normal estimation, all from a single input image of an object with non-Lambertian surface and general reflectance. The network explicitly segregates global geometric features and local lighting-specific features of the object from a single image. The local features resemble attached shadows, shadings, and specular highlights, providing valuable lighting estimation and relighting cues. The global features capture the lighting-independent geometric attributes that effectively guide the surface normal estimation. The joint training transfers valuable insights to achieve significant improvements across all three tasks. We show that the proposed single-image-based relighting framework outperforms several existing photometric stereo methods which require multiple images of a static object.