Xiaokang Peng

Balanced Multimodal Learning via On-the-Fly Gradient Modulation

Audio-visual learning helps to comprehensively understand the world, by integrating different senses. Accordingly, multiple input modalities are expected to boost model performance, but we actually find that they are not fully exploited even when the multi-modal model outperforms its uni-modal count…