Vy Ai Vo

Multi-timescale Representation Learning in LSTM Language Models

Language models must capture statistical dependencies between words at timescales ranging from very short to very long. Earlier work has demonstrated that dependencies in natural language tend to decay with distance between words according to a power law. However, it is unclear how this knowledge ca…