Hitesh Ballani

Implicit Language Models are RNNs: Balancing Parallelization and Expressivity

State-space models (SSMs) and transformers dominate the language modeling landscape. However, they are constrained to a lower computational complexity than classical recurrent neural networks (RNNs), limiting their expressivity. In contrast, RNNs lack parallelization during training, raising fundame…