2021
How Do Neural Sequence Models Generalize? Local and Global Cues for Out-of-Distribution Prediction
EMNLP 2021main
After a neural sequence model encounters an unexpected token, can its behavior be predicted? We show that RNN and transformer language models exhibit structured, consistent generalization in out-of-distribution contexts. We begin by introducing two idealized models of generalization in next-word pre…