Single-Shot Scene Reconstruction
Sergey Zakharov, Rares Andrei Ambrus, Vitor Campagnolo Guizilini, Dennis Park, Wadim Kehl, Fredo Durand, Joshua B. Tenenbaum, Vincent Sitzmann
Abstract
We introduce a novel scene reconstruction method to infer a fully editable and re-renderable model of a 3D road scene from a single image. We represent movable objects separately from the immovable background, and recover a full 3D model of each distinct object as well as their spatial relations in the scene. We leverage transformer-based detectors and neural implicit 3D representations and we build a Scene Decomposition Network (SDN) that reconstructs the scene in 3D. Furthermore, we show that this reconstruction can be used in an analysis-by-synthesis setting via differentiable rendering. Trained only on simulated road scenes, our method generalizes well to real data in the same class without any adaptation thanks to its strong inductive priors. Experiments on two synthetic-real dataset pairs (PD-DDAD and VKITTI-KITTI) show that our method can robustly recover scene geometry and appearance, as well as reconstruct and re-render the scene from novel viewpoints.
BibTeX
@inproceedings{
zakharov2021singleshot,
title={Single-Shot Scene Reconstruction},
author={Sergey Zakharov and Rares Andrei Ambrus and Vitor Campagnolo Guizilini and Dennis Park and Wadim Kehl and Fredo Durand and Joshua B. Tenenbaum and Vincent Sitzmann and Jiajun Wu and Adrien Gaidon},
booktitle={5th Annual Conference on Robot Learning },
year={2021},
url={https://openreview.net/forum?id=CGn3XKSf7vf}
}