2024
Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression
ACL 2024long
Key-value (KV) caching is an important technique to accelerate the inference of large language models (LLMs), but incurs significant memory overhead. To compress the size of KV cache, existing methods often compromise precision or require extra data for calibration, limiting their practicality in LL…