Zengqi Yu

Cooperative or Competitive? Understanding the Interaction between Attention Heads From A Game Theory Perspective

Despite the remarkable success of attention-based large language models (LLMs), the precise interaction mechanisms between attention heads remain poorly understood. In contrast to prevalent methods that focus on individual head contributions, we rigorously analyze the intricate interplay among atten…