2026
A Constrained Optimization Perspective of Unrolled Transformers
ICML 2026spotlight
We introduce a constrained optimization framework for training transformers that behave like optimization descent algorithms. Specifically, we enforce layerwise descent constraints on the objective function and replace standard empirical risk minimization (ERM) with a primal-dual training scheme. Th…