NAACL 2025short8 citations
Repetition Neurons: How Do Language Models Produce Repetitions?
Abstract
This paper introduces repetition neurons, which can be regarded as “skill neurons” responsible for the repetition problem in text generation tasks. These neurons are progressively activated more strongly as repetition continues, indicating that they perceive repetition as a task to copy the previous context repeatedly, similar to in-context learning. We identify these repetition neurons by comparing activation values before and after the onset of repetition in texts generated by recent pre-trained language models. We analyze the repetition neurons in three English and one Japanese pre-trained language models and observe similar patterns across them.
BibTeX
@inproceedings{hiraoka-inui-2025-repetition,
title = "Repetition Neurons: How Do Language Models Produce Repetitions?",
author = "Hiraoka, Tatsuya and
Inui, Kentaro",
editor = "Chiruzzo, Luis and
Ritter, Alan and
Wang, Lu",
booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)",
month = apr,
year = "2025",
address = "Albuquerque, New Mexico",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.naacl-short.41/",
pages = "483--495",
ISBN = "979-8-89176-190-2"
}