Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting
Bong Gyun Kang, Dongjun Lee, HyunGi Kim, Dohyun Chung, Sungroh Yoon
Abstract
Sequence modeling faces challenges in capturing long-range dependencies across diverse tasks. Recent linear and transformer-based forecasters have shown superior performance in time series forecasting. However, they are constrained by their inherent inability to effectively address long-range dependencies in time series data, primarily due to using fixed-size inputs for prediction. Furthermore, they typically sacrifice essential temporal correlation among consecutive training samples by shuffling them into mini-batches. To overcome these limitations, we introduce a fast and effective Spectral Attention mechanism, which preserves temporal correlations among samples and facilitates the handling of long-range information while maintaining the base model structure. Spectral Attention preserves long-period trends through a low-pass filter and facilitates gradient to flow between samples. Spectral Attention can be seamlessly integrated into most sequence models, allowing models with fixed-sized look-back windows to capture long-range dependencies over thousands of steps. Through extensive experiments on 11 real-world time series datasets using 7 recent forecasting models, we consistently demonstrate the efficacy of our Spectral Attention mechanism, achieving state-of-the-art results.
BibTeX
@inproceedings{
kang2024introducing,
title={Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting},
author={Bong Gyun Kang and Dongjun Lee and HyunGi Kim and Dohyun Chung and Sungroh Yoon},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=dxyNVEBQMp}
}