Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruning
Predicting model performance at larger scales enables the design of training strategies and architectures tailored to specific performance targets. Empirical scaling law research identifies functional forms to aid this prediction task. These describe the relationship between loss and compute using a…