InvGS: a Novel Real-Time Inverse Rendering Framework Utilizing 3D Gaussian Splatting

Abstract

In this paper, we address the problem of reconstructing geometry, material and illumination from multi-view images of a scene captured in an unknown environment. Compared to prevalent NeRF-based methods, which are typically hindered by slow training and time-intensive evaluating, our approach capitalizes on recent advancements in Volume Rendering: point-based 3D Gaussian Splatting (3DGS) techniques. However, the inherent lack of geometric priors in this point-based representation and the material-lighting ambiguity of the radiance field make it intractable for inverse rendering tasks. To overcome this challenge, our method adopt a two-step approach which we call InvGS. First, our method optimize the original parameters of the initial 3DGS point cloud using flattening and alignment regularization to make the Gaussians closer to the object surfaces for reconstructing a more compact geometry. In the second stage, our method jointly optimize lighting and material parameters of 3DGS point cloud through the differentiable rendering process, thereby following the physically-based rendering equation. Benefiting from the fast rasterization of 3DGS and real-time rendering techniques in physically-based rendering, InvGS can reasonably shade each Gaussian under image-based HDR illumination without compromising real-time performance. We demonstrate that our method is comparable to the state of the arts, and even outperform classic and deep learning-based approaches.