INSTANT: Compressing Gradients and Activations for Resource-Efficient Training
Tuan-Kiet Doan, Trung-Hieu Tran, Enzo Tartaglione, Nikola Simidjievski, Van-Tam Nguyen
Abstract
Deep learning has advanced at an unprecedented pace. This progress has led to a significant increase in its complexity. However, despite extensive research on accelerating inference, training deep models directly within a resource-constrained budget remains a considerable challenge due to its high computational and memory requirements. In this paper, we introduce INSTANT (compressIng gradieNtS and acTivAtions for resource-efficieNt Training), a method designed to address both the computational and the memory bottlenecks when training. INSTANT reduces resource demands during backpropagation by projecting gradients and activations into a low-rank subspace and performing computation within that compressed representation. Experimental results demonstrate that INSTANT achieves a $15\times$ reduction in computational cost and $32\times$ reduction in activation memory with negligible impact on model performance. The code will be made publicly available upon the paper's acceptance.
BibTeX
@inproceedings{
doan2026instant,
title={{INSTANT}: Compressing Gradients and Activations for Resource-Efficient Training},
author={Tuan-Kiet Doan and Trung-Hieu Tran and Enzo Tartaglione and Nikola Simidjievski and Van-Tam Nguyen},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=P2q6Y7UweV}
}