Predicting Discourse Trees from Transformer-based Neural Summarizers
Wen Xiao, Patrick Huber, Giuseppe Carenini
Abstract
Previous work indicates that discourse information benefits summarization. In this paper, we explore whether this synergy between discourse and summarization is bidirectional, by inferring document-level discourse trees from pre-trained neural summarizers. In particular, we generate unlabeled RST-style discourse trees from the self-attention matrices of the transformer model. Experiments across models and datasets reveal that the summarizer learns both, dependency- and constituency-style discourse information, which is typically encoded in a single head, covering long- and short-distance discourse dependencies. Overall, the experimental results suggest that the learned discourse information is general and transferable inter-domain.
BibTeX
@inproceedings{xiao-etal-2021-predicting,
title = "Predicting Discourse Trees from Transformer-based Neural Summarizers",
author = "Xiao, Wen and
Huber, Patrick and
Carenini, Giuseppe",
editor = "Toutanova, Kristina and
Rumshisky, Anna and
Zettlemoyer, Luke and
Hakkani-Tur, Dilek and
Beltagy, Iz and
Bethard, Steven and
Cotterell, Ryan and
Chakraborty, Tanmoy and
Zhou, Yichao",
booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
month = jun,
year = "2021",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.naacl-main.326/",
doi = "10.18653/v1/2021.naacl-main.326",
pages = "4139--4152"
}