2021
Catformer: Designing Stable Transformers via Sensitivity Analysis
ICML 2021spotlight
Transformer architectures are widely used, but training them is non-trivial, requiring custom learning rate schedules, scaling terms, residual connections, careful placement of submodules such as normalization, and so on. In this paper, we improve upon recent analysis of Transformers and formalize a…