Semantic Decoupling Based Semantic Scene Completion From a Single Depth Image

Abstract

Semantic Scene Completion (SSC) is a task that simultaneously predicts the occupancy and semantic labels of the environment. Compared with separate processing, SSC leverages the coupled nature of scene completion and semantic segmentation. Although this multitask integration can utilize complementarity and correlation between tasks, it also increases the training difficulty. To address this, in this paper, we propose a Semantic Decoupling based Semantic Scene Completion (SD-SSC) network from a single depth image. The semantic segmentation task is decoupled from the semantic scene completion task, and we use 2D and 3D semantic supervision to simplify the scene completion task and improve SSC performance. Specifically, our network first performs 2D semantic segmentation on the depth image and transforms features into 3D voxel space as semantic priors. Then, the 3D SSC is performed based on the voxel features and the flipped Truncated Signed Distance Field (f-TSDF). We use multi-scale 3D semantic supervision to further enhance the semantic information and fuse semantic and geometric features through the Planar Attention Fusion Module (PAFM) to obtain accurate SSC results. The proposed SD-SSC network achieves state-of-the-art performance on the NYU dataset (51.1% mIoU) and the NYUCAD dataset (61.9% mIoU) among all single depth-image based methods. It is even better than most RGB-D fusion-based SSC methods. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Saturday-xql/SD-SSC</uri>.