Facial Expression Recognition with DToF Sensing

Abstract

Facial Expression Recognition (FER) is crucial for understanding human emotions, with applications spanning from mental health assessment to marketing recommendation systems. However, existing camera-based methods raise privacy concerns, while RF-based approaches suffer from limited environmental generalizability and high cost. In this work, we propose ToFace, a FER system leveraging a low-cost (4.8$) Direct Time-of-Flight (DToF) sensor that has been available on commodity smartphones. This sensor provides an extremely low-resolution 8 × 8 depth map and a clear Field of View (FoV), significantly mitigating privacy concerns while avoiding the impact of ambient objects. Despite the benefits, the low-resolution depth map introduces significant challenges for precise expression recognition due to limited facial structure information. We first develop a physical model to extract additional spatial information from the intermediate sensor output, i.e., the transient histograms. We then propose a physics-integrated neural network to reconstruct a facial structure map comprising both depth and orientation for accurate expression recognition. We conduct real-world experiments with 12 users and compare our model with several baselines. The results demonstrate that ToFace achieves the highest recognition accuracy of 75%.