Shirui Bian

Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networks

Deeper modern architectures are costly to train, making hyperparameter transfer preferable to expensive repeated tuning. Maximal Update Parametrization ($\mu$P) helps explain why many hyperparameters transfer across width. Yet depth scaling is less understood for modern architectures, whose computat…