Anunay Yadav

RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling

Transformers have become the cornerstone of modern large-scale language models, but their reliance on softmax attention poses a computational bottleneck at both training and inference. Recurrent models offer high efficiency, but compressing the full sequence into a fixed-size and holistic representa…