2026
Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
ICML 2026poster
Large Language Models (LLMs) often assign disproportionate attention to the first token, a phenomenon known as the attention sink. Several recent approaches aim to address this issue, including Sink Attention in GPT-OSS and Gated Attention in Qwen3-Next. However, a comprehensive analysis of the rela…