Continuous Review and Timely Correction: Enhancing the Resistance to Noisy Labels via Self-Not-True Distillation

Abstract

Deep neural networks possess substantial learning capacities and robust expressive power, making them prone to overfitting mislabeled data. Fortunately, the memorization effect shows that the networks tend to memorize the clean data first, and then gradually memorize the mislabeled data. Correspondingly, early stopping is proposed and has proven to be effective in mitigating overfitting. However, the networks can still overfit some mislabeled data in the early training stage, resulting in forgotten knowledge of clean data. In addition, early stopping lacks correction of errors caused by mislabeled data. In this paper, we propose that the network should continuously review the knowledge it learned earlier to enhance clean data memorization while timely correcting the incorrect knowledge learned from the mislabeled data. To implement these two ideas, we first introduce self-distillation into training, which employs a teacher network from the previous stage to guide the current network, enhancing clean data memorization. Based on this, we further propose the not-true distillation. Before distilling knowledge from the teacher network, we mask the true class (i.e. label class) in the logits, focusing only on not-true classes to correct the accumulated incorrect knowledge. Extensive experiments on simulated and realworld benchmarks adequately validate the superior performance of our method.