Dynamic Structured Neural Topic Model with Self-Attention Mechanism
Nozomu Miyamoto, Masaru Isonuma, Sho Takase, Junichiro Mori, Ichiro Sakata
Abstract
This study presents a dynamic structured neural topic model, which can handle the time-series development of topics while capturing their dependencies. Our model captures the topic branching and merging processes by modeling topic dependencies based on a self-attention mechanism. Additionally, we introduce citation regularization, which induces attention weights to represent citation relations by modeling text and citations jointly. Our model outperforms a prior dynamic embedded topic model regarding perplexity and coherence, while maintaining sufficient diversity across topics. Furthermore, we confirm that our model can potentially predict emerging topics from academic literature.
BibTeX
@inproceedings{miyamoto-etal-2023-dynamic,
title = "Dynamic Structured Neural Topic Model with Self-Attention Mechanism",
author = "Miyamoto, Nozomu and
Isonuma, Masaru and
Takase, Sho and
Mori, Junichiro and
Sakata, Ichiro",
editor = "Rogers, Anna and
Boyd-Graber, Jordan and
Okazaki, Naoaki",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
month = jul,
year = "2023",
address = "Toronto, Canada",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.findings-acl.366/",
doi = "10.18653/v1/2023.findings-acl.366",
pages = "5916--5930"
}