Sample-level Self-paced Learning to Tackle Multimodal Imbalance Problem

Abstract

The issue of multimodal imbalance has attracted widespread attention recently, and spurred the proposal of numerous dataset-level modulation strategies. However, we observe that the degree of modality imbalance may vary significantly across different samples, suggesting that dataset-level strategies may fail to learn the full spectrum of multimodal information present in certain samples. In this paper, we propose a sample-level multimodal self-paced learning strategy (SMSL). It first assesses the degree of modality imbalance in each sample and progressively learns the weaker modalities in these samples through self-paced learning with a two-phase modulation. This approach not only helps to fully leverage the multimodal information of each sample but also ensures the model's stability. Experiments on two benchmark datasets, CREMA-D and IEMOCAP, have demonstrated the robustness and effectiveness of SMSL.