LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Models
Yan Wang, Ling Ding, Tien N Nguyen, Shaohua Wang, Yanan Zheng
Abstract
Large Language Models for code often entail significant computational complexity, which grows significantly with the length of the input code sequence. We propose LeanCode for code simplification to reduce training and prediction time, leveraging code contexts in utilizing attention scores to represent the tokens’ importance. We advocate for the selective removal of tokens based on the average context-aware attention scores rather than average scores across all inputs. LeanCode uses the attention scores of ‘CLS’ tokens within the encoder for classification tasks, such as code search. It also employs the encoder-decoder attention scores to determine token significance for sequence-to-sequence tasks like code summarization. Our evaluation shows LeanCode‘s superiority over the SOTAs DietCode and SlimCode, with improvements of 60% and 16% for code search, and 29% and 27% for code summarization, respectively.
BibTeX
@inproceedings{wang-etal-2025-leancode,
title = "{LEANCODE}: Understanding Models Better for Code Simplification of Pre-trained Large Language Models",
author = "Wang, Yan and
Ding, Ling and
Nguyen, Tien N and
Wang, Shaohua and
Zheng, Yanan",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.acl-long.78/",
doi = "10.18653/v1/2025.acl-long.78",
pages = "1551--1567",
ISBN = "979-8-89176-251-0"
}