Group and Shuffle: Efficient Structured Orthogonal Parametrization
Mikhail Gorbunov, Kolya Yudin, Vera Soboleva, Aibek Alanov, Alexey Naumov, Maxim Rakhuba
Abstract
The increasing size of neural networks has led to a growing demand for methods of efficient finetuning. Recently, an orthogonal finetuning paradigm was introduced that uses orthogonal matrices for adapting the weights of a pretrained model. In this paper, we introduce a new class of structured matrices, which unifies and generalizes structured classes from previous works. We examine properties of this class and build a structured orthogonal parametrization upon it. We then use this parametrization to modify the orthogonal finetuning framework, improving parameter efficiency. We empirically validate our method on different domains, including adapting of text-to-image diffusion models and downstream task finetuning in language modeling. Additionally, we adapt our construction for orthogonal convolutions and conduct experiments with 1-Lipschitz neural networks.
BibTeX
@inproceedings{
gorbunov2024group,
title={Group and Shuffle: Efficient Structured Orthogonal Parametrization},
author={Mikhail Gorbunov and Kolya Yudin and Vera Soboleva and Aibek Alanov and Alexey Naumov and Maxim Rakhuba},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=7EQx56YSB2}
}