M3: Multi-level dataset for Multi-document summarisation of Medical studies
Yulia Otmakhova, Karin Verspoor, Timothy Baldwin, Antonio Jimeno Yepes, Jey Han Lau
Abstract
We present M3 (Multi-level dataset for Multi-document summarisation of Medical studies), a benchmark dataset for evaluating the quality of summarisation systems in the biomedical domain. The dataset contains sets of multiple input documents and target summaries of three levels of complexity: documents, sentences, and propositions. The dataset also includes several levels of annotation, including biomedical entities, direction, and strength of relations between them, and the discourse relationships between the input documents (“contradiction” or “agreement”). We showcase usage scenarios of the dataset by testing 10 generic and domain-specific summarisation models in a zero-shot setting, and introduce a probing task based on counterfactuals to test if models are aware of the direction and strength of the conclusions generated from input studies.
BibTeX
@inproceedings{otmakhova-etal-2022-m3,
title = "{M}3: Multi-level dataset for Multi-document summarisation of Medical studies",
author = "Otmakhova, Yulia and
Verspoor, Karin and
Baldwin, Timothy and
Jimeno Yepes, Antonio and
Lau, Jey Han",
editor = "Goldberg, Yoav and
Kozareva, Zornitsa and
Zhang, Yue",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
month = dec,
year = "2022",
address = "Abu Dhabi, United Arab Emirates",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.findings-emnlp.286/",
doi = "10.18653/v1/2022.findings-emnlp.286",
pages = "3887--3901"
}