GeMIMO: Searching the Cores of X-formers for Time Series Forecasting

Abstract

In recent years, Transformer-based models have been widely used in time series forecasting tasks, demonstrating exceptional performance. However, these models lack interpretability, making it difficult to identify which components play a core role in predictions and which are redundant. To address this limitation, we introduce the GeMIMO method, which utilizes a heuristic algorithm to identify and refine masks. We applied GeMIMO to the Transformer model, and the results showed that temporal encoding had no significant effect, while positional encoding was only effective in the decoder. Moreover, the experiments suggest that the attention mechanism is not the most critical component of the Transformer; instead, the feedforward neural network and normalization layers are key to improving predictive accuracy. By applying the masks learned through GeMIMO to optimize the Transformer, we observed an 28% improvement in prediction efficiency and an 25% increase in accuracy, demonstrating the effectiveness of GeMIMO.