Long Text Generation with Topic-aware Discrete Latent Variable Model
Erguang Yang, Mingtong Liu, Deyi Xiong, Yujie Zhang, Yufeng Chen, Jinan Xu
Abstract
Generating coherent long texts is an important yet challenging task, particularly forthe open-ended generation. Prior work based on discrete latent codes focuses on the modeling of discourse relation, resulting in discrete codes only learning shallow semantics (Ji and Huang, 2021). A natural text always revolves around several related topics and the transition across them is natural and smooth.In this work, we investigate whether discrete latent codes can learn information of topics. To this end, we build a topic-aware latent code-guided text generation model. To encourage discrete codes to model information about topics, we propose a span-level bag-of-words training objective for the model. Automatic and manual evaluation experiments show that our method can generate more topic-relevant and coherent texts.
BibTeX
@inproceedings{yang-etal-2022-long,
title = "Long Text Generation with Topic-aware Discrete Latent Variable Model",
author = "Yang, Erguang and
Liu, Mingtong and
Xiong, Deyi and
Zhang, Yujie and
Chen, Yufeng and
Xu, Jinan",
editor = "Goldberg, Yoav and
Kozareva, Zornitsa and
Zhang, Yue",
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
month = dec,
year = "2022",
address = "Abu Dhabi, United Arab Emirates",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.emnlp-main.554/",
doi = "10.18653/v1/2022.emnlp-main.554",
pages = "8100--8107"
}