2026
Push, Pop, Parallelize: Stack-Augmented Linear Attention via the Delta Rule
ICML 2026poster
Linear attention architectures based on the Delta rule, such as DeltaNet and RWKV-7, combine Transformers' training scalability with RNNs' inference efficiency and can provably solve regular language tasks. However, due to their fixed-size state, these models fundamentally struggle to capture the re…