Sayak Mallick

LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws

Scaling laws guide the development of large language models (LLMs) by offering estimates for the optimal balance of model size, tokens, and compute. More recently, loss-to-loss scaling laws that relate losses across pretraining datasets and downstream tasks have emerged as a powerful tool for unders…