ICLR 2023top-5%31 citations
SAM as an Optimal Relaxation of Bayes
Thomas Möllenhoff, Mohammad Emtiyaz Khan
Abstract
Sharpness-aware minimization (SAM) and related adversarial deep-learning methods can drastically improve generalization, but their underlying mechanisms are not yet fully understood. Here, we establish SAM as a relaxation of the Bayes objective where the expected negative-loss is replaced by the optimal convex lower bound, obtained by using the so-called Fenchel biconjugate. The connection enables a new Adam-like extension of SAM to automatically obtain reasonable uncertainty estimates, while sometimes also improving its accuracy. By connecting adversarial and Bayesian methods, our work opens a new path to robustness.
bayesian deep learningsharpness-aware minimizationvariational bayesconvex duality
BibTeX
@inproceedings{
m{\"o}llenhoff2023sam,
title={{SAM} as an Optimal Relaxation of Bayes},
author={Thomas M{\"o}llenhoff and Mohammad Emtiyaz Khan},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=k4fevFqSQcX}
}