2026
Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks
ICML 2026poster
Deeper modern architectures are costly to train, making hyperparameter transfer preferable to expensive repeated tuning. Maximal Update Parametrization ($\mu$P) helps explain why many hyperparameters transfer across width. Yet depth scaling is less understood for modern architectures, whose computat…