2021
Shapeshifter: a Parameter-efficient Transformer using Factorized Reshaped Matrices
NeurIPS 2021poster
Language models employ a very large number of trainable parameters. Despite being highly overparameterized, these networks often achieve good out-of-sample test performance on the original task and easily fine-tune to related tasks. Recent observations involving, for example, intrinsic dimension of…