Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction
Yijun Liu, Yixuan Wang, Yuzhuang Xu, Shiyu Ji, Yang Xu, Qingfu Zhu, Wanxiang Che
Abstract
Large language models (LLMs) utilize key-value (KV) cache to store historical information during sequence processing. The size of KV cache grows linearly as the length of the sequence extends, which seriously affects memory usage and decoding efficiency. Current methods for KV cache eviction typically utilize the last window from the pre-filling phase as queries to compute the KV importance scores for eviction. Although this scheme is simple to implement, it tends to overly focus on local information, potentially leading to the neglect or omission of crucial global information. To mitigate this issue, we propose **Judge Q**, a novel training method which incorporates a soft token list. This method only tunes the model’s embedding layer at a low training cost. By concatenating the soft token list at the end of the input sequence, we train these tokens
BibTeX
@inproceedings{aaai2026_judgeqtrainableq,
title = {Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction},
author = {Yijun Liu and Yixuan Wang and Yuzhuang Xu and Shiyu Ji and Yang Xu and Qingfu Zhu and Wanxiang Che},
booktitle = {AAAI 2026},
year = {2026}
}