Zhixing Lu

The Orthogonality of Weight Vectors: The Key Characteristics of Normalization and Residual Connections

Normalization and residual connections find extensive application within the intricate architecture of deep neural networks, contributing significantly to their heightened performance. Nevertheless, the precise factors responsible for this elevated performance have remained elusive. Our theoretical…