Micro-expression recognition by fusing action unit detection and Spatio-temporal features

Abstract

Micro-expressions (MEs) are subtle and brief facial expressions that occur involuntarily and may reveal hidden emotions. Due to MEs' weak intensities, it is challenging to discriminate MEs from image noise through AU detection results or spatio-temporal features. To model authentic ME patterns rather than overfitting to noise, we propose a novel multiframe strategy that captures detailed motion patterns and a two-layered feature encoding scheme to model interactions across different parts of the feature maps. Furthermore, we propose a novel facial Action Unit Graph Convolutional Network (AU GCN) that can adapt to testing input data through an AU detection module and a learnable adjacent matrix with a transformer encoder. Finally, we fuse the enhanced spatiotemporal features and AU GCN results to recognize MEs. Experimental results show that our methods outperform SOTA in F1 scores on SAMM and CASME II datasets, and also achieves the highest accuracy on CASME II dataset.