KLFormer: Karhunen-Loève Transform for Robust 3D Human Pose Estimation

Abstract

In the scope of 3D human pose estimation, the task encompasses estimating the 3D positions of key skeletal points (i.e., wrists, elbows, and knees) from a 2D image or video sequence. This technology demonstrates widespread applicability across diverse domains, encompassing domains such as kinematic analysis, virtual reality, augmented reality, and medical imaging analysis. The common approach is divided into two stages: i) 2D Keypoint Detection: Detecting 2D keypoints from images. ii) 2D-to-3D Lifting: Converting 2D keypoints into 3D coordinates. Present research predominantly concentrates on Stage 2 and leverages sophisticated deep learning architectures, notably Transformers, yet two key challenges exist: i) suboptimal performance in predicting local details; ii) susceptibility to noise interference. In this work, we propose a time-principal component fusion model with limb segment property tracking is proposed to address the challenges above. By integrating timedomain features with principal components extracted through the Karhunen-Loève Transform (KLT), the model aims to address challenges related to feature extraction and noise reduction. Furthermore, to address the issue of suboptimal performance in predicting local details, we devise a property transformer to track the lengths of limb segments and predict the fixed property. Extensive experiments demonstrate that KLFormer showcases state-of-the-art performance on the standard benchmark dataset, Human3.6M.