Sam Blouir

Birdie: Advancing State Space Language Modeling with Dynamic Mixtures of Training Objectives

Efficient state space models (SSMs), including linear recurrent neural networks and linear attention variants, have emerged as potential alternative language models to Transformers. While efficient, SSMs struggle with tasks requiring in-context retrieval, such as text copying and associative recall,…