ICLR 2025poster6 citations
Memory Mosaics
Jianyu Zhang, Niklas Nolte, Ranajoy Sadhukhan, Beidi Chen, Leon Bottou
Abstract
Memory Mosaics are networks of associative memories working in concert to achieve a prediction task of interest. Like transformers, memory mosaics possess compositional capabilities and in-context learning capabilities. Unlike transformers, memory mosaics achieve these capabilities in comparatively transparent way (“predictive disentanglement”). We illustrate these capabilities on a toy example and also show that memory mosaics perform as well or better than transformers on medium-scale language modeling tasks.
predictive disentanglementAssociative memorylanguage model
BibTeX
@inproceedings{
zhang2025memory,
title={Memory Mosaics},
author={Jianyu Zhang and Niklas Nolte and Ranajoy Sadhukhan and Beidi Chen and Leon Bottou},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=IiagjrJNwF}
}