SEGMENT+: Long Text Processing with Short-Context Language Models
Wei Shi, Shuang Li, Kerun Yu, Jinglei Chen, Zujie Liang, Xinhui Wu, Yuxi Qian, Feng Wei
Abstract
There is a growing interest in expanding the input capacity of language models (LMs) across various domains. However, simply increasing the context window does not guarantee robust performance across diverse long-input processing tasks, such as understanding extensive documents and extracting detailed information from lengthy and noisy data. In response, we introduce Segment+, a general framework that enables LMs to handle extended inputs within limited context windows efficiently. Segment+ utilizes structured notes and a filtering module to manage information flow, resulting in a system that is both controllable and interpretable. Our extensive experiments across various model sizes, focusing on long-document question-answering and Needle-in-a-Haystack tasks, demonstrate the effectiveness of Segment+ in improving performance.
BibTeX
@inproceedings{shi-etal-2024-segment,
title = "{SEGMENT}+: Long Text Processing with Short-Context Language Models",
author = "Shi, Wei and
Li, Shuang and
Yu, Kerun and
Chen, Jinglei and
Liang, Zujie and
Wu, Xinhui and
Qian, Yuxi and
Wei, Feng and
Zheng, Bo and
Liang, Jiaqing and
Chen, Jiangjie and
Xiao, Yanghua",
editor = "Al-Onaizan, Yaser and
Bansal, Mohit and
Chen, Yun-Nung",
booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.emnlp-main.926/",
doi = "10.18653/v1/2024.emnlp-main.926",
pages = "16605--16617"
}