Axel Marmoret

TENSLORA: TENSOR ALTERNATIVES FOR LOW-RANK ADAPTATION

Low-Rank Adaptation (LoRA) is widely used to efficiently adapt Transformers by adding trainable low-rank matrices to attention projections. While effective, these matrices are considered independent for each attention projection (Query, Key, and Value) and each layer. Recent extensions have consider…