2023
Transcending Scaling Laws with 0.1% Extra Compute
EMNLP 2023long main
Scaling language models improves performance but comes with significant computational costs. This paper proposes UL2R, a method that substantially improves existing language models and their scaling curves with a relatively tiny amount of extra compute. The key idea is to continue training a state-o…