mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections

Chenliang Li; Haiyang Xu; Junfeng Tian; Wei Wang; Ming Yan; Bin Bi; Jiabo Ye; He Chen

EMNLP 2022main145 citations

mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections

Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, He Chen

Abstract

Large-scale pre-trained foundation models have been an emerging paradigm for building artificial intelligence (AI) systems, which can be quickly adapted to a wide range of downstream tasks. This paper presents mPLUG, a new vision-language foundation model for both cross-modal understanding and generation. Most existing pre-trained models suffer from inefficiency and linguistic signal overwhelmed by long visual sequences in cross-modal alignment. To address both problems, mPLUG introduces an effective and efficient vision-language architecture with novel cross-modal skip-connections.mPLUG is pre-trained end-to-end on large-scale image-text pairs with both discriminative and generative objectives. It achieves state-of-the-art results on a wide range of vision-language downstream tasks, including image captioning, image-text retrieval, visual grounding and visual question answering. mPLUG also demonstrates strong zero-shot transferability on vision-language and video-language tasks. The code and pre-trained models are available at https://github.com/alibaba/AliceMind

BibTeX

@inproceedings{li-etal-2022-mplug,
    title = "m{PLUG}: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections",
    author = "Li, Chenliang  and
      Xu, Haiyang  and
      Tian, Junfeng  and
      Wang, Wei  and
      Yan, Ming  and
      Bi, Bin  and
      Ye, Jiabo  and
      Chen, He  and
      Xu, Guohai  and
      Cao, Zheng  and
      Zhang, Ji  and
      Huang, Songfang  and
      Huang, Fei  and
      Zhou, Jingren  and
      Si, Luo",
    editor = "Goldberg, Yoav  and
      Kozareva, Zornitsa  and
      Zhang, Yue",
    booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.emnlp-main.488/",
    doi = "10.18653/v1/2022.emnlp-main.488",
    pages = "7241--7259"
}