Flipping Consistent and Counterfactual Attention Network for Facial Expression Recognition

Abstract

Facial expression recognition (FER) has recently made great process with attention mechanisms. However, existing methods mostly treat the network as a black box and ignore the causal relationship between modules. In this paper, we propose a flipping consistent and counterfactual attention network (FCCAN), which innovatively uses two causal inferences to guide the learning process. First, we construct an attention network with a multi-head attention learning (MAL) module to learn attention for different local features. Then, we measure and improve the quality of each component with causal reasoning. According to the consistency of features from flipped images, we design flipping consistency learning (FCL) module to force the network to extract comprehensive expression features. Based on counterfactual causality, we propose counterfactual attention learning (CAL) module to encourage the network to learn more effective attention. Extensive experiments implemented on FER datasets (including RAF-DB, FERPlus and AffectNet) show that our FCCAN outperforms state-of-the-art methods.