Improving the Stochastic Gradient Descent's Test Accuracy by Manipulating the ℓ∞ Norm of its Gradient Approximation
The stochastic gradient descent (SGD) is a simple yet very influential algorithm used to find the minimum of a loss (cost) function which is dependent on datasets with large cardinality, such in cases typically associated with deep learning (DL). There exists several variants/improvements over the "…