2026
TileSparse: Arithmetic-Intensity-Aware Sparse Attention for Compute-Bound LLM Decoding
ICML 2026poster
Sparse attention has emerged as a vital technique for long-context inference in Large Language Models (LLMs), effectively accelerating memory-bound decoding by reducing memory access for non-essential keys. However, the assumption that decoding attention is memory-bound has been shattered. The proli…