Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions
Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber
Abstract
Recent studies of the computational power of recurrent neural networks (RNNs) reveal a hierarchy of RNN architectures, given real-time and finite-precision assumptions. Here we study auto-regressive Transformers with linearised attention, a.k.a. linear Transformers (LTs) or Fast Weight Programmers (FWPs). LTs are special in the sense that they are equivalent to RNN-like sequence processors with a fixed-size state, while they can also be expressed as the now-popular self-attention networks. We show that many well-known results for the standard Transformer directly transfer to LTs/FWPs. Our formal language recognition experiments demonstrate how recently proposed FWP extensions such as recurrent FWPs and self-referential weight matrices successfully overcome certain limitations of the LT, e.g., allowing for generalisation on the parity problem. Our code is public.
BibTeX
@inproceedings{
irie2023practical,
title={Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions},
author={Kazuki Irie and R{\'o}bert Csord{\'a}s and J{\"u}rgen Schmidhuber},
booktitle={The 2023 Conference on Empirical Methods in Natural Language Processing},
year={2023},
url={https://openreview.net/forum?id=Q2Wu2Cfp2x}
}