A Statistical Investigation of Long Memory in Language and Music
Alexander Greaves-Tunnell, Zaid Harchaoui
Abstract
Representation and learning of long-range dependencies is a central challenge confronted in modern applications of machine learning to sequence data. Yet despite the prominence of this issue, the basic problem of measuring long-range dependence, either in a given data source or as represented in a trained deep model, remains largely limited to heuristic tools. We contribute a statistical framework for investigating long-range dependence in current applications of deep sequence modeling, drawing on the well-developed theory of long memory stochastic processes. This framework yields testable implications concerning the relationship between long memory in real-world data and its learned representation in a deep learning architecture, which are explored through a semiparametric framework adapted to the high-dimensional setting.
BibTeX
@InProceedings{pmlr-v97-greaves-tunnell19a,
title = {A Statistical Investigation of Long Memory in Language and Music},
author = {Greaves-Tunnell, Alexander and Harchaoui, Zaid},
booktitle = {Proceedings of the 36th International Conference on Machine Learning},
pages = {2394--2403},
year = {2019},
editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan},
volume = {97},
series = {Proceedings of Machine Learning Research},
month = {09--15 Jun},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v97/greaves-tunnell19a/greaves-tunnell19a.pdf},
url = {https://proceedings.mlr.press/v97/greaves-tunnell19a.html},
abstract = {Representation and learning of long-range dependencies is a central challenge confronted in modern applications of machine learning to sequence data. Yet despite the prominence of this issue, the basic problem of measuring long-range dependence, either in a given data source or as represented in a trained deep model, remains largely limited to heuristic tools. We contribute a statistical framework for investigating long-range dependence in current applications of deep sequence modeling, drawing on the well-developed theory of long memory stochastic processes. This framework yields testable implications concerning the relationship between long memory in real-world data and its learned representation in a deep learning architecture, which are explored through a semiparametric framework adapted to the high-dimensional setting.}
}