Sparse and Structured Hopfield Networks
Saul José Rodrigues dos Santos, Vlad Niculae, Daniel C McNamee, Andre Martins
Abstract
Modern Hopfield networks have enjoyed recent interest due to their connection to attention in transformers. Our paper provides a unified framework for sparse Hopfield networks by establishing a link with Fenchel-Young losses. The result is a new family of Hopfield-Fenchel-Young energies whose update rules are end-to-end differentiable sparse transformations. We reveal a connection between loss margins, sparsity, and exact memory retrieval. We further extend this framework to structured Hopfield networks via the SparseMAP transformation, which can retrieve pattern associations instead of a single pattern. Experiments on multiple instance learning and text rationalization demonstrate the usefulness of our approach.
BibTeX
@inproceedings{
santos2024sparse,
title={Sparse and Structured Hopfield Networks},
author={Saul Jos{\'e} Rodrigues dos Santos and Vlad Niculae and Daniel C McNamee and Andre Martins},
booktitle={Forty-first International Conference on Machine Learning},
year={2024},
url={https://openreview.net/forum?id=OdPlFWExX1}
}