Classification of Eye-Tracking Data Based on Spatiotemporal Attention Encoding

Abstract

Eye movement classification can decode cognitive processes, offering valuable insights for a wide range of applications. However, existing eye movement classification models primarily focus on static fixation-based features and often neglect the encoding of spatiotemporal eye movement features, which are crucial for accurately reconstructing visual attention. To address this limitation, we propose a spatiotemporal attention encoding (STAE) model that jointly captures both spatial and temporal features for eye-tracking classification. First, we utilize a Vision Transformer (ViT) to extract spatial features from fixations by taking global competition into consideration. We then introduce a global weighting Gated Recurrent Unit (GRU) model to capture temporal correlations from the feature sequence. Specifically, we propose a hidden-state-based weighting to fuse the influence of different fixations on the current fixation. In the experiment, we evaluated our model on three tasks: autism spectrum disorder (ASD) identification, visual task classification, and age classification. Experiential results across three databases demonstrate that our model outperforms existing methods and shows strong adaptability across various eye movement classification tasks. The code is available at https://github.com/HectorTo/spatiotemporal-attention-encoding-STAE-.