Forget the Data and Fine-Tuning! Just Fold the Network to Compress
Dong Wang, Haris Šikić, Lothar Thiele, Olga Saukh
Abstract
We introduce model folding, a novel data-free model compression technique that merges structurally similar neurons across layers, significantly reducing the model size without the need for fine-tuning or access to training data. Unlike existing methods, model folding preserves data statistics during compression by leveraging k-means clustering, and using novel data-free techniques to prevent variance collapse or explosion. Our theoretical framework and experiments across standard benchmarks, including ResNet18 and LLaMA-7B, demonstrate that model folding achieves comparable performance to data-driven compression techniques and outperforms recently proposed data-free methods, especially at high sparsity levels. This approach is particularly effective for compressing large-scale models, making it suitable for deployment in resource-constrained environments.
BibTeX
@inproceedings{
wang2025forget,
title={Forget the Data and Fine-Tuning! Just Fold the Network to Compress},
author={Dong Wang and Haris {\v{S}}iki{\'c} and Lothar Thiele and Olga Saukh},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=W2Wkp9MQsF}
}