NeurIPS 2025poster0 citations
Attention-based clustering
Rodrigo Maulen-Soto, Pierre Marion, Claire Boyer
Abstract
Transformers have emerged as a powerful neural network architecture capable of tackling a wide range of learning tasks. In this work, we provide a theoretical analysis of their ability to automatically extract structure from data in an unsupervised setting. In particular, we demonstrate their suitability for clustering when the input data is generated from a Gaussian mixture model. To this end, we study a simplified two-head attention layer and define a population risk whose minimization with unlabeled data drives the head parameters to align with the true mixture centroids.
Theory of neural networksClusteringTransformersAttention-based modelsMixture modelsOptimizationUnsupervised learning
BibTeX
@inproceedings{
maulen-soto2025attentionbased,
title={Attention-based clustering},
author={Rodrigo Maulen-Soto and Pierre Marion and Claire Boyer},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=NRvxzOdSPU}
}