Sparse-VQ Transformer: An FFN-Free Framework with Vector Quantization for Enhanced Time Series

Abstract

Time series analysis is vital for numerous applications, and transformers have become increasingly prominent in this domain. Leading methods customize the transformer architecture from NLP and CV, utilizing a patching technique to convert continuous signals into segments. Yet, time series data is uniquely challenging due to significant distribution shifts and intrinsic noise levels. To address these two challenges, we introduce the Sparse Vector Quantized FFN-Free Transformer (Sparse-VQ). Our methodology capitalizes on a sparse vector quantization technique to reduce noise impact and capture sufficient statistics for forecasting, serving as an alternative to the Feed-Forward layer (FFN) in the transformer architecture. Our FFN-free approach trims the parameter size, enhancing computational efficiency and reducing overfitting. Through evaluations across ten benchmark datasets, including a newly introduced CAISO dataset, Sparse-VQ surpasses leading models with a 7.84% and 4.17% decrease in MAE for univariate and multivariate time series forecasting. It can be seamlessly integrated with existing transformer-based models to elevate their performance. Our source code and the new dataset are available at: https://github.com/Yanjun-Zhao/Sparse-VQ.