Mohammad AkbarTajari

An Empirical Study on the Transferability of Transformer Modules in Parameter-efficient Fine-tuning

Parameter-efficient fine-tuning has garnered lots of attention in recent studies.On this subject, we investigate the capability of different transformer modules in transferring knowledge from a pre-trained model to a downstream task. Our empirical results suggest that every transformer module is a w…