Benchmarking Diverse-Modal Entity Linking with Generative Models
Sijia Wang, Alexander Hanbo Li, Henghui Zhu, Sheng Zhang, Pramuditha Perera, Chung-Wei Hang, Jie Ma, William Yang Wang
Abstract
Entities can be expressed in diverse formats, such as texts, images, or column names and cell values in tables. While existing entity linking (EL) models work well on per modality configuration, such as text-only EL, visual grounding or schema linking, it is more challenging to design a unified model for diverse modality configurations. To bring various modality configurations together, we constructed a benchmark for diverse-modal EL (DMEL) from existing EL datasets, covering all three modalities including text, image and table. To approach the DMEL task, we proposed a generative diverse-modal model (GDMM) following a multimodal-encoder-decoder paradigm. Pre-training GDMM with rich corpora builds a solid foundation for DMEL without storing the entire KB for inference. Fine-tuning GDMM builds a stronger DMEL baseline, outperforming state-of-the-art task-specific EL models by 8.51 F1 score on average. Additionally, extensive error analyses are conducted to highlight the challenge of DMEL, facilitating future researches on this task.
BibTeX
@inproceedings{wang-etal-2023-benchmarking,
title = "Benchmarking Diverse-Modal Entity Linking with Generative Models",
author = "Wang, Sijia and
Li, Alexander Hanbo and
Zhu, Henghui and
Zhang, Sheng and
Perera, Pramuditha and
Hang, Chung-Wei and
Ma, Jie and
Wang, William Yang and
Wang, Zhiguo and
Castelli, Vittorio and
Xiang, Bing and
Ng, Patrick",
editor = "Rogers, Anna and
Boyd-Graber, Jordan and
Okazaki, Naoaki",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.findings-acl.497/",
doi = "10.18653/v1/2023.findings-acl.497",
pages = "7841--7857"
}