GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalization
Recently, Sharpness-Aware Minimization (SAM) algorithm has shown state-of-the-art generalization abilities in vision tasks. It demonstrates that flat minima tend to imply better generalization abilities. However, it has some difficulty implying SAM to some natural language tasks, especially to model…