Automatic Design of Adapter Architectures for Enhanced Parameter-Efficient Fine-Tuning

Abstract

Adapters, one of the most important parameter-efficient fine-tuning (PEFT) methods, achieve state-of-the-art (SOTA) performance through manual architecture design. To unlock the full potential of adapter tuning, we introduce the AutoAdapter framework, designated to design novel adapter architectures automatically. First, we discuss the adapter design choices and define a search space. Second, we propose CDARTS, an contribution-based differentiable neural architecture search (NAS) method, to enhance the search results. We conduct comprehensive experiments and analysis on the GLUE and SuperCLUE benchmark tasks, demonstrating that AutoAdapter effectively designs novel adapters that outperform recent baseline PEFT methods.