2023
Towards Lossless Head Pruning through Automatic Peer Distillation for Language Models
IJCAI 2023poster
Pruning has been extensively studied in Transformer-based language models to improve efficiency. Typically, we zero (prune) unimportant model weights and train a derived compact model to improve final accuracy. For pruned weights, we treat them as useless and discard them. This usually leads to sign…