SACR: Self-training with Saliency-Augmented Consistency Regularization for Few-Shot Learners

Abstract

Pre-trained language models have made significant strides in natural language processing tasks, enabling flexible fine-tuning for downstream applications. However, in few-shot learning scenarios, pre-trained models face challenges related to overfitting due to limited training samples, which hinders their ability to capture data diversity and robustly handle input variations. To overcome these limitations, we propose the Saliency-Augmented Consistency Regularization (SACR) framework, a novel self-training strategy designed to improve few-shot learning performance and robustness. SACR consists of three key components: (1) Saliency-guided Data Perturbation, which uses saliency analysis to identify and perturb words that significantly influence model predictions, generating semantically consistent pseudo-samples; (2) Semantic-equivalent Sample Mining, which employs K-means clustering to select semantically similar samples and prevent semantic shift; and (3) Consistency Training, where regularization ensures consistency between the semantics and prediction distributions of the original and perturbed samples. We conducted extensive experiments on 15 public datasets to evaluate our approach. SACR significantly outperforms strong baseline models and demonstrates superior generalization capabilities compared to state-of-the-art methods, achieving an average 1.0% improvement in classification accuracy.