Attreyee Mukherjee

Cache Saver: A Modular Framework for Efficient, Affordable, and Reproducible LLM Inference

Inference constitutes the majority of costs throughout the lifecycle of a large language model (LLM). While numerous LLM inference engines focusing primarily on low-level optimizations have been developed, there is a scarcity of non-intrusive client-side frameworks that perform high-level optimizati