On the Angular Update and Hyperparameter Tuning of a Scale-Invariant Network
"Modern deep neural networks are equipped with normalization layers such as batch normalization or layer normalization to enhance and stabilize training dynamics. If a network contains such normalization layers, the optimization objective is invariant to the scale of the neural network parameters. T…