Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding

Taowen Liu; Marta Andronic; Deniz Gunduz; George Anthony Constantinides

EMNLP 20250 citations

Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding

Taowen Liu, Marta Andronic, Deniz Gunduz, George Anthony Constantinides

Abstract

LLM training is resource-intensive. Quantized training improves computational and memory efficiency but introduces quantization noise, which can hinder convergence and degrade model accuracy. Stochastic Rounding (SR) has emerged as a theoretically attractive alternative to deterministic rounding, offering unbiased gradient estimates. However, its interaction with other training factors—especially batch size—remains underexplored. In this paper, we present a theoretical and empirical study of mini-batch stochastic gradient descent (SGD) with SR, showing that increased batch sizes can compensate for reduced precision during backpropagation. Furthermore, we show that quantizing weights and activations impacts gradient variance in distinct ways. Our experiments validate these theoretical insights. Our experiments validate these theoretical insights.

BibTeX

@inproceedings{emnlp2025_trainingwithfewe,
  title = {Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding},
  author = {Taowen Liu and Marta Andronic and Deniz Gunduz and George Anthony Constantinides},
  booktitle = {EMNLP 2025},
  year = {2025}
}