CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation
Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Mingsong Yan, Zi Yang, Paul D. Hovland, Bogdan Nicolae, Franck Cappello
Abstract
The full-size MLPs and the projection layers in attention introduce tremendous model sizes of large language models (LLMs), consuming extensive computational resources in pre-training. We empirically observe that the activations of pre-trained LLMs exhibit low-rank property. Motivated by such observations, we propose **CoLA** and its memory-efficient implementation, **CoLA-M**, to replace these full-size layers with compute-efficient **auto-encoders** that naturally enforce low-rank activations throughout training. This fundamental architectural change eliminates the activation redundancy and significantly boosts model capacity and training efficiency. Experiments on LLaMA models with 60 million to 7 billion parameters show that CoLA reduces the computing cost by 2 × and improves training throughput by 1.86 × while maintaining full-rank level performance. CoLA-M further squeezes memory cost without sacrificing throughput, offering a pre-training approach with collectively superior parameter, computing, and memory efficiency. The LLMs produced are also 2 × smaller, enabling faster inference with lower memory cost on resource-constrained platforms.
BibTeX
@inproceedings{emnlp2025_colacomputeeffic,
title = {CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation},
author = {Ziyue Liu and Ruijie Zhang and Zhengyang Wang and Mingsong Yan and Zi Yang and Paul D. Hovland and Bogdan Nicolae and Franck Cappello and Sui Tang and Zheng Zhang},
booktitle = {EMNLP 2025},
year = {2025}
}