CodeSSM: Towards State Space Models for Code Understanding
Shweta Verma, Abhinav Anand, Mira Mezini
Abstract
Although transformers dominate many code-specific tasks, they have significant limitations. This paper explores State Space Models (SSMs) as a promising alternative for code understanding tasks such as retrieval, classification, and clone detection. We introduce CodeSSM, the first SSM-based model trained on code corpora to assess its effectiveness. Our results demonstrate that SSMs are more sample-efficient and can extrapolate to longer contexts beyond the pretraining length. Extensive experiments show that SSMs offer a viable alternative to transformers, addressing several their limitations. Additionally, CodeSSM reduces memory usage by up to 64% compared to transformers at a context length of 2048, with greater savings as context length grows.The code is available [here](https://github.com/abx04/CodeSSM).
BibTeX
@inproceedings{emnlp2025_codessmtowardsst,
title = {CodeSSM: Towards State Space Models for Code Understanding},
author = {Shweta Verma and Abhinav Anand and Mira Mezini},
booktitle = {EMNLP 2025},
year = {2025}
}