Malthe Have Musaeus

Iterative Structured Knowledge Distillation: Optimizing Language Models Through Layer-by-Layer Distillation

Traditional language model compression techniques, like knowledge distillation, require a fixed architecture, limiting flexibility, while structured pruning methods often fail to preserve performance. This paper introduces Iterative Structured Knowledge Distillation (ISKD), which integrates knowledg…