Learning from Oblivion: Predicting Knowledge-Overflowed Weights via Retrodiction of Forgetting
Pre-trained weights have become a cornerstone of modern deep learning, enabling efficient knowledge transfer and improving downstream task performance, especially in data-scarce scenarios. However, a fundamental question remains: how can we obtain better pre-trained weights that encapsulate more kno