2025
Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training
ICML 2025poster
In a neural network with ReLU activations, the number of piecewise linear regions in the output can grow exponentially with depth. However, this is highly unlikely to happen when the initial parameters are sampled randomly, which therefore often leads to the use of networks that are unnecessarily la…