2026
xLSTM Distillation: Achieving Teacher-Student Parity Through Efficient Hybrid Architectures
Lukas Hauzenberger, Niklas Schmidinger, Thomas Schmied, Anamaria-Roberta Hartl, David Stap, Pieter-Jan Hoedt +3
ICML 2026poster
There have been numerous attempts to distill quadratic attention-based LLMs into sub-quadratic linearized architectures. However, despite extensive research, such distilled models often fail to match the performance of their teacher LLMs on various downstream tasks. We set out the goal of lossless d…