Position: Enough of Scaling LLMs! Lets Focus on Downscaling
Yash Goel, Ayan Sengupta, Tanmoy Chakraborty
Abstract
We challenge the dominant focus on neural scaling laws and advocate for a paradigm shift toward downscaling in the development of large language models (LLMs). While scaling laws have provided critical insights into performance improvements through increasing model and dataset size, we emphasize the significant limitations of this approach, particularly in terms of computational inefficiency, environmental impact, and deployment constraints. To address these challenges, we propose a holistic framework for downscaling LLMs that seeks to maintain performance while drastically reducing resource demands. This paper outlines practical strategies for transitioning away from traditional scaling paradigms, advocating for a more sustainable, efficient, and accessible approach to LLM development.
BibTeX
@inproceedings{
goel2025position,
title={Position: Enough of Scaling {LLM}s! Lets Focus on Downscaling},
author={Yash Goel and Ayan Sengupta and Tanmoy Chakraborty},
booktitle={Forty-second International Conference on Machine Learning Position Paper Track},
year={2025},
url={https://openreview.net/forum?id=CYJlJgEzZs}
}