2021
Multi-timescale Representation Learning in LSTM Language Models
ICLR 2021poster
Language models must capture statistical dependencies between words at timescales ranging from very short to very long. Earlier work has demonstrated that dependencies in natural language tend to decay with distance between words according to a power law. However, it is unclear how this knowledge ca…