Tip the Scales: Achieving Balance in Adversarial Examples Across Modalities

Abstract

In the field of multimodal learning, controlling the training of unimodal encoders from different perspectives is a primary approach to addressing Training Imbalance. However, the inherent capacity limitations of the modality affect the model’s capability. Therefore, generating adversarial examples that can achieve balanced transferability remains a challenging and perplexing problem. In this paper, we propose the InterModality Balanced Attack (MOBA) to address this problem. MOBA leverages Aggregated Modality Perturbation (AMP), which exploits the unbalanced effects of text and image perturbations to maximize the impact on the victim model. AMP capitalizes on the intrinsic feature connections between modalities during the optimization process, adjusting perturbations through Cross-Modality Discrepancy Loss to enhance attack success rates. Additionally, we devise the Transferability-Enhanced Evolution (TEE) to overcome the issue of diminished attack transferability due to model capacity limitations. TEE employs Transfer-Driven Optimization Loss to alleviate overfitting in single models, thereby enhancing the generalization ability.