Jason S. Wang

Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape View

Training language models currently requires pre-determining a fixed compute budget because the typical cosine learning rate schedule depends on the total number of steps. In contrast, the Warmup-Stable-Decay (WSD) schedule uses a constant learning rate to produce a main branch of iterates that can i…