Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning
Le-Trung Nguyen, Aël Quélennec, Van-Tam Nguyen, Enzo Tartaglione
Abstract
On-device learning has emerged as a promising direction for AI development, particularly because of its potential to reduce latency issues and mitigate privacy risks associated with device-server communication, while improving energy efficiency. Despite these advantages, significant memory and computational constraints still represent major challenges for its deployment. Drawing on previous studies on low-rank decomposition methods that address activation memory bottlenecks in backpropagation, we propose a novel shortcut approach as an alternative. Our analysis and experiments demonstrate that our method can reduce activation memory usage, even up to $120.09\times$ compared to vanilla training, while also reducing overall training FLOPs up to $1.86\times$ when evaluated on traditional benchmarks.
BibTeX
@inproceedings{
nguyen2025beyond,
title={Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning},
author={Le-Trung Nguyen and A{\"e}l Qu{\'e}lennec and Van-Tam Nguyen and Enzo Tartaglione},
booktitle={Forty-second International Conference on Machine Learning},
year={2025},
url={https://openreview.net/forum?id=YWUYqN2laV}
}