Zhiqiang Ding

DynamicAttention: Dynamic KV Cache for Disaggregate LLM Inference

In large language model inference, efficient utilization of GPU memory is of utmost importance. Current systems suffer from unreasonable GPU memory allocation: excessive memory is idle under low load, while memory shortages occur under high load. Moreover, existing GPU memory management methods do n…