Monocular 360 Depth Estimation via Spherical Fully-Connected CRFs

Abstract

Monocular 360° depth estimation poses significant challenges due to the inherent distortion of the equirectangular projection (ERP). This distortion separates adjacent spherical points after their projection onto the ERP plane, especially in the polar regions, resulting in insufficient spherical relationships. To address this issue, recent methods calculate spherical neighbors within the tangent domain. However, since the tangent patch and the sphere share only one common point, spherical relationships are established only among neighbors around this common point. In this letter, we propose Spherical Fully-Connected CRFs (<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SF-CRFs</b>). We start by evenly partitioning an ERP image into regular windows, where windows at the equator have broader spherical neighbors than those at the poles. To enhance spherical relationships, our SF-CRFs feature two key components. Firstly, to include sufficient spherical neighbors, we introduce a Spherical Window Transform (<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SWT</b>) module. This module replicates the equator window's spherical relationships across all other windows, leveraging the rotational invariance of the sphere. Remarkably, the transformation process is efficient, transforming all windows in a 512×1024 ERP image in just 0.038 seconds on a CPU. Secondly, we introduce a Planar-Spherical Interaction (<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PSI</b>) module to calculate the SF-CRFs, which facilitates the relationships between regular and transformed windows. By integrating SF-CRFs blocks into a decoder, we propose <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CRF360D</b>, a novel 360° depth estimation framework that achieves state-of-the-art performance across diverse datasets. Our CRF360D is compatible with different perspective image-trained backbones (<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e</i>.<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">g</i>., EfficientNet), serving as the encoder.