Moving SLAM: Fully Unsupervised Deep Learning in Non-Rigid Scenes

Abstract

We propose a new deep learning framework to decompose monocular videos into 3D geometry (camera pose and depth), moving objects, and their motions, with no supervision. We build upon the idea of view synthesis, which uses classical camera geometry to re-render a source image from a different point-of-view to obtain supervisory signals, specified by a predicted relative 6-degree-of-freedom pose and depth map. However, the typical view synthesis equations rely on a strong assumption: that objects in scenes do not move. This rigid-world assumption limits the predictive power, and rules out learning about objects automatically. We propose a simple solution: minimize the synthesis error on small local regions of the image instead. While the scene as a whole may be non-rigid, it is always possible to find small regions that are approximately rigid, such as inside a moving object. Our network can learn a dense pose map describing poses for each local region. This represents a significantly richer model, including 6D object motions, with little additional complexity. We establish very competitive results on unsupervised odometry and depth prediction on KITTI. We also demonstrate new capabilities on EPIC-Kitchens, a challenging dataset of indoor videos, where there is no ground truth information for depth, odometry, object segmentation or motion - yet all are recovered automatically by our approach.