Maximizing the Effectiveness of Larger BERT Models for Compression
Wen-Shu Fan, Su Lu, Shangyu Xing, Xin-Chun Li, De-Chuan Zhan
Abstract
Knowledge distillation (KD) is a widely used approach for BERT compression, where a larger BERT model serves as a teacher to transfer knowledge to a smaller student model. Prior works have found that distilling a larger BERT with superior performance may degrade student’s performance than a smaller BERT. In this paper, we investigate the limitations of existing KD methods for larger BERT models. Through Canonical Correlation Analysis, we identify that these methods fail to fully exploit the potential advantages of larger teachers. To address this, we propose an improved distillation approach that effectively enhances knowledge transfer. Comprehensive experiments demonstrate the effectiveness of our method in enabling larger BERT models to distill knowledge more efficiently.
BibTeX
@inproceedings{fan-etal-2025-maximizing,
title = "Maximizing the Effectiveness of Larger {BERT} Models for Compression",
author = "Fan, Wen-Shu and
Lu, Su and
Xing, Shangyu and
Li, Xin-Chun and
Zhan, De-Chuan",
editor = "Che, Wanxiang and
Nabende, Joyce and
Shutova, Ekaterina and
Pilehvar, Mohammad Taher",
booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = jul,
year = "2025",
address = "Vienna, Austria",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.acl-long.1067/",
doi = "10.18653/v1/2025.acl-long.1067",
pages = "21975--21990",
ISBN = "979-8-89176-251-0"
}