Heart Rate and Oxygen Saturation Estimation from Facial Video with Multimodal Physiological Data Generation

Abstract

Efforts to estimate multiple physiological parameters such as heart rate and oxygen saturation from facial videos have been made. However, training robust machine learning models for the estimation is challenging without large multimodal physiological datasets containing multiple physiological parameters and facial videos. In this paper, we propose a method to estimate heart rate and oxygen saturation from facial videos with multimodal physiological data generation. To collect sufficient datasets, the proposed method generates multimodal physiological datasets from several datasets containing a part of physiological modalities. Furthermore, to accurately estimate physiological parameters for unseen subjects, i.e., not included in the training data, we generate a multimodal physiological dataset for unseen subjects by using short facial videos of unseen subjects. Experimental results using three public datasets show the effectiveness of our multimodal physiological data generation.