Shape-prior Free Space-time Neural Radiance Field for 4D Semantic Reconstruction of Dynamic Scene from Sparse-View RGB Videos

Abstract

Many applications in Augmented/Virtual Reality or robotics require precise geometry modeling of individual elements in a dynamic scene under a sparse-view camera setup, without any prior information about their semantic labels or shapes. In our research, we introduce a 3D shape prior-free Neural Radiance Field-based technique for detailed geometry reconstruction under human-object interactions, offering an explicit surface reconstruction with semantic labels for reconstructed geometry. Our approach harnesses the capabilities of an Invertible Neural Network to learn a deformation function that effectively connects local (current-frame input) and canonical spaces for each of the components under motion. The deformation process is guided by temporal constraints from multi-frame, facilitating the precise reconstruction of the complex interactions between humans and objects. Our experimental evaluations highlight the effectiveness of our framework, demonstrating its ability to accurately represent both the comprehensive object-compositional scene and individual components over state-of-the-art methods, under complex interactions between the scene entities. This research is deemed to mark a significant stride in semantic 3D geometry modeling within dynamic interactive environments, relying solely on sparse multi-view RGB data.