Joy Crosbie

Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning

Large language models (LLMs) have shown a remarkable ability to learn and perform complex tasks through in-context learning (ICL). However, a comprehensive understanding of its internal mechanisms is still lacking. This paper explores the role of induction heads in a few-shot ICL setting. We analyse…