FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction
Chen-Yu Lee, Chun-Liang Li, Hao Zhang, Timothy Dozat, Vincent Perot, Guolong Su, Xiang Zhang, Kihyuk Sohn
Abstract
The recent advent of self-supervised pre-training techniques has led to a surge in the use of multimodal learning in form document understanding. However, existing approaches that extend the mask language modeling to other modalities require careful multi-task tuning, complex reconstruction target designs, or additional pre-training data. In FormNetV2, we introduce a centralized multimodal graph contrastive learning strategy to unify self-supervised pre-training for all modalities in one loss. The graph contrastive objective maximizes the agreement of multimodal representations, providing a natural interplay for all modalities without special customization. In addition, we extract image features within the bounding box that joins a pair of tokens connected by a graph edge, capturing more targeted visual cues without loading a sophisticated and separately pre-trained image embedder. FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE and Payment benchmarks with a more compact model size.
BibTeX
@inproceedings{lee-etal-2023-formnetv2,
title = "{F}orm{N}et{V}2: Multimodal Graph Contrastive Learning for Form Document Information Extraction",
author = "Lee, Chen-Yu and
Li, Chun-Liang and
Zhang, Hao and
Dozat, Timothy and
Perot, Vincent and
Su, Guolong and
Zhang, Xiang and
Sohn, Kihyuk and
Glushnev, Nikolay and
Wang, Renshen and
Ainslie, Joshua and
Long, Shangbang and
Qin, Siyang and
Fujii, Yasuhisa and
Hua, Nan and
Pfister, Tomas",
editor = "Rogers, Anna and
Boyd-Graber, Jordan and
Okazaki, Naoaki",
booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.acl-long.501/",
doi = "10.18653/v1/2023.acl-long.501",
pages = "9011--9026"
}