DeepSITH: Efficient Learning via Decomposition of What and When Across Time Scales

Abstract

Extracting temporal relationships over a range of scales is a hallmark of human perception and cognition---and thus it is a critical feature of machine learning applied to real-world problems. Neural networks are either plagued by the exploding/vanishing gradient problem in recurrent neural networks (RNNs) or must adjust their parameters to learn the relevant time scales (e.g., in LSTMs). This paper introduces DeepSITH, a deep network comprising biologically-inspired Scale-Invariant Temporal History (SITH) modules in series with dense connections between layers. Each SITH module is simply a set of time cells coding what happened when with a geometrically-spaced set of time lags. The dense connections between layers change the definition of what from one layer to the next. The geometric series of time lags implies that the network codes time on a logarithmic scale, enabling DeepSITH network to learn problems requiring memory over a wide range of time scales. We compare DeepSITH to LSTMs and other recent RNNs on several time series prediction and decoding tasks. DeepSITH achieves results comparable to state-of-the-art performance on these problems and continues to perform well even as the delays are increased.