Yilu Cao

Revealing Long-context Potential of Attention Heads via Frequency Kernels

Large language model (LLM) exists a subset of attention heads that are highly responsible for long-context processing. Existing work has identified different long-context heads in models, but their detection methods mainly rely on model inference on actual long texts and do not analyze the inherent …