2025
FP4 All the Way: Fully Quantized Training of Large Language Models
NeurIPS 2025spotlight
We demonstrate, for the first time, fully quantized training (FQT) of large language models (LLMs) using predominantly 4-bit floating-point (FP4) precision for weights, activations, and gradients on datasets up to 200 billion tokens. We extensively investigate key design choices for FP4, including b…