DSD²: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free?
Victor Quétu, Enzo Tartaglione
Abstract
Neoteric works have shown that modern deep learning models can exhibit a sparse double descent phenomenon. Indeed, as the sparsity of the model increases, the test performance first worsens since the model is overfitting the training data; then, the overfitting reduces, leading to an improvement in performance, and finally, the model begins to forget critical information, resulting in underfitting. Such a behavior prevents using traditional early stop criteria. In this work, we have three key contributions. First, we propose a learning framework that avoids such a phenomenon and improves generalization. Second, we introduce an entropy measure providing more insights into the insurgence of this phenomenon and enabling the use of traditional stop criteria. Third, we provide a comprehensive quantitative analysis of contingent factors such as re-initialization methods, model width and depth, and dataset noise. The contributions are supported by empirical evidence in typical setups. Our code is available at https://github.com/VGCQ/DSD2.
BibTeX
@article{Quétu_Tartaglione_2024, title={DSD²: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free?}, volume={38}, url={https://ojs.aaai.org/index.php/AAAI/article/view/29393}, DOI={10.1609/aaai.v38i13.29393}, abstractNote={Neoteric works have shown that modern deep learning models can exhibit a sparse double descent phenomenon. Indeed, as the sparsity of the model increases, the test performance first worsens since the model is overfitting the training data; then, the overfitting reduces, leading to an improvement in performance, and finally, the model begins to forget critical information, resulting in underfitting. Such a behavior prevents using traditional early stop criteria. In this work, we have three key contributions. First, we propose a learning framework that avoids such a phenomenon and improves generalization. Second, we introduce an entropy measure providing more insights into the insurgence of this phenomenon and enabling the use of traditional stop criteria. Third, we provide a comprehensive quantitative analysis of contingent factors such as re-initialization methods, model width and depth, and dataset noise. The contributions are supported by empirical evidence in typical setups. Our code is available at https://github.com/VGCQ/DSD2.}, number={13}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Quétu, Victor and Tartaglione, Enzo}, year={2024}, month={Mar.}, pages={14749-14757} }