Anamaria-Roberta Hartl

xLSTM Distillation: Achieving Teacher-Student Parity Through Efficient Hybrid Architectures

There have been numerous attempts to distill quadratic attention-based LLMs into sub-quadratic linearized architectures. However, despite extensive research, such distilled models often fail to match the performance of their teacher LLMs on various downstream tasks. We set out the goal of lossless d…