Julian Kranz

SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures

We study gradient flows for loss landscapes of fully connected feedforward neural networks with commonly used continuously differentiable activation functions such as the logistic, hyperbolic tangent, softplus or GELU function. We prove that the gradient flow either converges to a critical point or…