Chengye YU

SmartCache: Context-aware Semantic Cache for Efficient Multi-turn LLM Inference

Large Language Models (LLMs) for multi-turn conversations suffer from inefficiency: semantically similar queries across different user sessions trigger redundant computation and duplicate memory-intensive Key-Value (KV) caches. Existing optimizations such as prefix caching overlook semantic similari…