← Search

Harshith Goka

2 accepted papers

2024

Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models

ICML 2024poster

In spite of their huge success, transformer models remain difficult to scale in depth. In this work, we develop a unified signal propagation theory and provide formulae that govern the moments of the forward and backward signal through the transformer model. Our framework can be used to understand a…