NCDI-Diffusion: Neural Contextual and Directional Inversion for Novel View Synthesis through Diffusion Models

Abstract

Novel view synthesis typically requires a comprehensive set of multi-view images for either image-based rendering or scene representation-based optimization. However, achieving high-fidelity novel view rendering often demands a large number of images. To address this limitation, we propose NCDI-Diffusion, a novel diffusion-based view synthesis method that reduces the number of required images by leveraging the prior knowledge embedded in pre-trained diffusion models. Specifically, NCDI-Diffusion encapsulates both the contextual and directional information of a scene by utilizing neural descriptors, which are inversely derived from a limited set of positioned multi-view training images. These descriptors guide the diffusion model's image synthesis process, enabling the generation of high-quality novel views. Empirical results on the Forward-facing Dataset demonstrate the effectiveness of our approach to novel view synthesis.