A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields
Yiyu Zhuang, Qi Zhang, Xuan Wang, Hao Zhu, Ying Feng, Xiaoyu Li, Ying Shan, Xun Cao
Abstract
Recent advances in implicit neural representation have demonstrated the ability to recover detailed geometry and material from multi-view images. However, the use of simplified lighting models such as environment maps to represent non-distant illumination, or using a network to fit indirect light modeling without a solid basis, can lead to an undesirable decomposition between lighting and material. To address this, we propose a fully differentiable framework named Neural Illumination Fields (NeIF) that uses radiance fields as a lighting model to handle complex lighting in a physically based way. Together with integral lobe encoding for roughness-adaptive specular lobe and leveraging the pre-convolved background for accurate decomposition, the proposed method represents a significant step towards integrating physically based rendering into the NeRF representation. The experiments demonstrate the superior performance of novel-view rendering compared to previous works, and the capability to re-render objects under arbitrary NeRF-style environments opens up exciting possibilities for bridging the gap between virtual and real-world scenes.
BibTeX
@article{Zhuang_Zhang_Wang_Zhu_Feng_Li_Shan_Cao_2024, title={A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields}, volume={38}, url={https://ojs.aaai.org/index.php/AAAI/article/view/28618}, DOI={10.1609/aaai.v38i7.28618}, abstractNote={Recent advances in implicit neural representation have demonstrated the ability to recover detailed geometry and material from multi-view images. However, the use of simplified lighting models such as environment maps to represent non-distant illumination, or using a network to fit indirect light modeling without a solid basis, can lead to an undesirable decomposition between lighting and material. To address this, we propose a fully differentiable framework named Neural Illumination Fields (NeIF) that uses radiance fields as a lighting model to handle complex lighting in a physically based way. Together with integral lobe encoding for roughness-adaptive specular lobe and leveraging the pre-convolved background for accurate decomposition, the proposed method represents a significant step towards integrating physically based rendering into the NeRF representation. The experiments demonstrate the superior performance of novel-view rendering compared to previous works, and the capability to re-render objects under arbitrary NeRF-style environments opens up exciting possibilities for bridging the gap between virtual and real-world scenes.}, number={7}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Zhuang, Yiyu and Zhang, Qi and Wang, Xuan and Zhu, Hao and Feng, Ying and Li, Xiaoyu and Shan, Ying and Cao, Xun}, year={2024}, month={Mar.}, pages={7828-7836} }