Share Your Attention: Transformer Weight Sharing via Matrix-based Dictionary Learning
Magauiya Zhussip, Dmitriy Shopkhoev, Ammar Ali, Stamatios Lefkimmiatis
Abstract
Large language models (LLMs) have revolutionized AI applications, yet their high computational and memory demands hinder their widespread deployment. Existing compression techniques focus on intra-block optimizations (e.g., low-rank approximation or attention head pruning), while the repetitive layered structure of transformers implies significant inter-block redundancy - a dimension largely unexplored beyond key-value (KV) caching. Inspired by dictionary learning in convolutional networks, we propose a framework for structured weight sharing across transformer layers. Our approach decomposes attention projection matrices (Q, K, V, O) into shared dictionary atoms, reducing the attention module
BibTeX
@inproceedings{aaai2026_shareyourattenti,
title = {Share Your Attention: Transformer Weight Sharing via Matrix-based Dictionary Learning},
author = {Magauiya Zhussip and Dmitriy Shopkhoev and Ammar Ali and Stamatios Lefkimmiatis},
booktitle = {AAAI 2026},
year = {2026}
}