MMAR: Multilingual and Multimodal Anaphora Resolution in Instructional Videos
Cennet Oguz, Pascal Denis, Simon Ostermann, Emmanuel Vincent, Natalia Skachkova, Josef Van Genabith
Abstract
Multilingual anaphora resolution identifies referring expressions and implicit arguments in texts and links to antecedents that cover several languages. In the most challenging setting, cross-lingual anaphora resolution, training data, and test data are in different languages. As knowledge needs to be transferred across languages, this task is challenging, both in the multilingual and cross-lingual setting. We hypothesize that one way to alleviate some of the difficulty of the task is to include multimodal information in the form of images (i.e. frames extracted from instructional videos). Such visual inputs are by nature language agnostic, therefore cross- and multilingual anaphora resolution should benefit from visual information. In this paper, we provide the first multilingual and multimodal dataset annotated with anaphoric relations and present experimental results for end-to-end multimodal and multilingual anaphora resolution. Given gold mentions, multimodal features improve anaphora resolution results by ~10 % for unseen languages.
BibTeX
@inproceedings{oguz-etal-2024-mmar,
title = "{MMAR}: Multilingual and Multimodal Anaphora Resolution in Instructional Videos",
author = "Oguz, Cennet and
Denis, Pascal and
Ostermann, Simon and
Vincent, Emmanuel and
Skachkova, Natalia and
Genabith, Josef Van",
editor = "Al-Onaizan, Yaser and
Bansal, Mohit and
Chen, Yun-Nung",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-emnlp.88/",
doi = "10.18653/v1/2024.findings-emnlp.88",
pages = "1618--1633"
}