Peiyuan Zhou

SeerAttention: Self-distilled Attention Gating for Efficient Long-context Prefilling

Attention is the cornerstone of modern Large Language Models (LLMs). Yet its quadratic complexity hinders efficiency and scalability, especially for long-context processing. A promising approach is to leverage sparsity in attention. However, existing sparsity-based solutions predominantly rely on p…