The Surprising Effectiveness of Infinite-Width NTKs for Characterizing and Improving Model Training
Joshua DeOliveira, Walter Gerych, Elke Rundensteiner
Abstract
Developments in deep neural nets have trended towards increasingly larger overparameterized architectures, resulting in lengthy training sessions with ever more elusive training dynamics. Thus, ensuring these models learn accurate generalizable representations of data efficiently is challenging. Previous works have developed specialized techniques from data-pruning, architecture selection, pseudo-label generation, bias identification, or label refurbishment to improve downstream training. Problematically, most methods require prohibitively expensive iterative model training. In this paper, we demonstrate that we can exploit the recent neural tangent kernel (NTK) theory for understanding and improving model training behavior before ever training a model. First, we show a powerful signal derived from the NTK theory can be computed remarkably fast. We then leverage this signal for the design of a unified suite of surprisingly effective tools for the four important tasks of architecture selection, pseudo-label verification, bias identification, and label refurbishment, all requiring zero model training.
BibTeX
@article{DeOliveira_Gerych_Rundensteiner_2025, title={The Surprising Effectiveness of Infinite-Width NTKs for Characterizing and Improving Model Training}, volume={39}, url={https://ojs.aaai.org/index.php/AAAI/article/view/33786}, DOI={10.1609/aaai.v39i15.33786}, abstractNote={Developments in deep neural nets have trended towards increasingly larger overparameterized architectures, resulting in lengthy training sessions with ever more elusive training dynamics. Thus, ensuring these models learn accurate generalizable representations of data efficiently is challenging. Previous works have developed specialized techniques from data-pruning, architecture selection, pseudo-label generation, bias identification, or label refurbishment to improve downstream training. Problematically, most methods require prohibitively expensive iterative model training. In this paper, we demonstrate that we can exploit the recent neural tangent kernel (NTK) theory for understanding and improving model training behavior before ever training a model. First, we show a powerful signal derived from the NTK theory can be computed remarkably fast. We then leverage this signal for the design of a unified suite of surprisingly effective tools for the four important tasks of architecture selection, pseudo-label verification, bias identification, and label refurbishment, all requiring zero model training.}, number={15}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={DeOliveira, Joshua and Gerych, Walter and Rundensteiner, Elke}, year={2025}, month={Apr.}, pages={16262-16270} }