The Unexpected Deterministic and Universal Behavior of Large Softmax Classifiers
Mohamed El Amine Seddik, Cosme Louart, Romain COUILLET, Mohamed Tamaazousti
Abstract
This paper provides a large dimensional analysis of the Softmax classifier. We discover and prove that, when the classifier is trained on data satisfying loose statistical modeling assumptions, its weights become deterministic and solely depend on the data statistical means and covariances. As a striking consequence, despite the implicit and non-linear nature of the underlying optimization problem, the performance of the Softmax classifier is the same as if performed on a mere Gaussian mixture model, thereby disrupting the intuition that non-linearities inherently extract advanced statistical features from the data. Our findings are theoretically as well as numerically sustained on CNN representations of images produced by GANs.
BibTeX
@InProceedings{pmlr-v130-el-amine-seddik21a,
title = { The Unexpected Deterministic and Universal Behavior of Large Softmax Classifiers },
author = {El Amine Seddik, Mohamed and Louart, Cosme and COUILLET, Romain and Tamaazousti, Mohamed},
booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics},
pages = {1045--1053},
year = {2021},
editor = {Banerjee, Arindam and Fukumizu, Kenji},
volume = {130},
series = {Proceedings of Machine Learning Research},
month = {13--15 Apr},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v130/el-amine-seddik21a/el-amine-seddik21a.pdf},
url = {https://proceedings.mlr.press/v130/el-amine-seddik21a.html},
abstract = { This paper provides a large dimensional analysis of the Softmax classifier. We discover and prove that, when the classifier is trained on data satisfying loose statistical modeling assumptions, its weights become deterministic and solely depend on the data statistical means and covariances. As a striking consequence, despite the implicit and non-linear nature of the underlying optimization problem, the performance of the Softmax classifier is the same as if performed on a mere Gaussian mixture model, thereby disrupting the intuition that non-linearities inherently extract advanced statistical features from the data. Our findings are theoretically as well as numerically sustained on CNN representations of images produced by GANs. }
}