AAAI 2025technical0 citations
Information-Theoretic Generative Clustering of Documents
Abstract
We present *generative clustering* (GC) for clustering a set of documents, X, by using texts Y generated by large language models (LLMs) instead of by clustering the original documents X. Because LLMs provide probability distributions, the similarity between two documents can be rigorously defined in an information-theoretic manner by the KL divergence. We also propose a natural, novel clustering algorithm by using importance sampling. We show that GC outperforms any previous clustering method, often by a large margin. Furthermore, we show an application to generative document retrieval in which documents are indexed via hierarchical clustering and our method improves the retrieval accuracy.
BibTeX
@article{Du_Tanaka-Ishii_2025, title={Information-Theoretic Generative Clustering of Documents}, volume={39}, url={https://ojs.aaai.org/index.php/AAAI/article/view/33802}, DOI={10.1609/aaai.v39i16.33802}, abstractNote={We present *generative clustering* (GC) for clustering a set of documents, X, by using texts Y generated by large language models (LLMs) instead of by clustering the original documents X. Because LLMs provide probability distributions, the similarity between two documents can be rigorously defined in an information-theoretic manner by the KL divergence. We also propose a natural, novel clustering algorithm by using importance sampling. We show that GC outperforms any previous clustering method, often by a large margin. Furthermore, we show an application to generative document retrieval in which documents are indexed via hierarchical clustering and our method improves the retrieval accuracy.}, number={16}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Du, Xin and Tanaka-Ishii, Kumiko}, year={2025}, month={Apr.}, pages={16408-16417} }