Disordered-DABS: A Benchmark for Dynamic Aspect-Based Summarization in Disordered Texts
Abstract
Aspect-based summarization has seen significant advancements, especially in structured text. Yet, summarizing disordered, large-scale texts, like those found in social media and customer feedback, remains a significant challenge. Current research largely targets predefined aspects within structured texts, neglecting the complexities of dynamic and disordered environments. Addressing this gap, we introduce Disordered-DABS, a novel benchmark for dynamic aspect-based summarization tailored to unstructured text. Developed by adapting existing datasets for cost-efficiency and scalability, our comprehensive experiments and detailed human evaluations reveal that Disordered-DABS poses unique challenges to contemporary summarization models, including state-of-the-art language models such as GPT-3.5.
BibTeX
@inproceedings{guo-vosoughi-2024-disordered,
title = "Disordered-{DABS}: A Benchmark for Dynamic Aspect-Based Summarization in Disordered Texts",
author = "Guo, Xiaobo and
Vosoughi, Soroush",
editor = "Al-Onaizan, Yaser and
Bansal, Mohit and
Chen, Yun-Nung",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-emnlp.24/",
doi = "10.18653/v1/2024.findings-emnlp.24",
pages = "416--431"
}