EMNLP 2023long findings0 citations
Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation
Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, Peter Ramadge
Abstract
Unlike recurrent models, conventional wisdom has it that Transformers cannot perfectly model regular languages. Inspired by the notion of working memory, we propose a new Transformer variant named RegularGPT. With its novel combination of Weight-Sharing, Adaptive-Depth, and Sliding-Dilated-Attention, RegularGPT constructs working memory along the depth dimension, thereby enabling efficient and successful modeling of regular languages such as PARITY. We further test RegularGPT on the task of natural language length extrapolation and surprisingly find that it rediscovers the local windowed attention effect deemed necessary in prior work for length extrapolation.
TransformerAlgorithmic reasoningLength extrapolationWorking memory
BibTeX
@inproceedings{
chi2023transformer,
title={Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation},
author={Ta-Chung Chi and Ting-Han Fan and Alexander Rudnicky and Peter Ramadge},
booktitle={The 2023 Conference on Empirical Methods in Natural Language Processing},
year={2023},
url={https://openreview.net/forum?id=e8wYLib8HC}
}