SCI-Gaussian: Optimizing 3D Gaussian Radiance Fields from a Snapshot Compressive Image

Abstract

Snapshot compressive imaging (SCI) is a compressed sensing (CS)-based high-speed imaging modality. Recent efforts have explored the underlying 3D representation from only an SCI image using neural radiance fields (NeRF), yet the training time, rendering computation cost, and reconstruction quality limitations are general issues that have limited wider adoption. This paper introduces SCI-Gaussian, the first 3D-aware SCI reconstruction based on 3D Gaussian splatting (3D-GS). This method utilizes an explicit 3D representation to achieve efficient and high-quality scene reconstruction. The motivation stems from the highly efficient representation and surprising quality of 3D-GS, despite when applied to SCI system, it encounters difficulties in generating point initialization for explicit Gaussians and accurate pose recovery from a single SCI measured image. Specifically, we effectively initialize these Gaussians through sampling a coarsely trained NeRF at various hash structures, then model the physical formation of the SCI measurement and jointly optimize Gaussians and camera trajectories with a bundle adjustment formulation during exposure time. Extensive experiments on synthetic and real-world datasets demonstrate that SCI-Gaussian outperforms the state-of-the-art (SOTA) methods, achieving comparable or better results with significantly 10× faster training and 1000× faster rendering speed than the most recent NeRF-based method.