Andrew Mattarella-Micke

Do Long-Range Language Models Actually Use Long-Range Context?

Language models are generally trained on short, truncated input sequences, which limits their ability to use discourse-level information present in long-range context to improve their predictions. Recent efforts to improve the efficiency of self-attention have led to a proliferation of long-range Tr…