Jack of All Trades, Master of None: PMP-Guided Adaptive Multi-Teacher Distillation with Meta-Learning
Sisi Zhang, Zechao Lin, Xingbin Wang, Yulan Su, Yan Wang, Rui Hou, Dan Meng
Abstract
To enhance the robustness and accuracy of the small model, existing approaches combine adversarial training with knowledge distillation, introducing a comprehensive single-teacher model to improve the performance of the student model (small model). However, due to the limited knowledge of a teacher model, it appears "knowledge gain saturation" phenomenon. Therefore, we propose a PMP-Guided Adaptive Multi-Teacher Distillation with Meta-Learning. Pontryagin’s Maximum Principle is employed to solve the issue of inconsistent teaching objectives among teachers causing distinct optimization directions. Meanwhile, Meta-learning-network is designed to tackle the problem of a student struggling to balance the learned knowledge. A series of experiments conducted on public datasets demonstrate that our approach outperforms the state-of-the-art methods against various adversarial attacks.
BibTeX
@inproceedings{icassp2025_jackofalltradesm,
title = {Jack of All Trades, Master of None: PMP-Guided Adaptive Multi-Teacher Distillation with Meta-Learning},
author = {Sisi Zhang and Zechao Lin and Xingbin Wang and Yulan Su and Yan Wang and Rui Hou and Dan Meng},
booktitle = {ICASSP 2025},
year = {2025}
}