Zizhong Li

Unveiling the Magic: Investigating Attention Distillation in Retrieval-Augmented Generation

Retrieval-augmented generation framework addresses the limitations of large language models by enabling real-time knowledge updates for more accurate answers. An efficient way in the training phase of retrieval-augmented models is attention distillation, which uses attention scores as supervision si…