SNNPTrack: Spiking Neural Network Based Prompt for High-Accuracy RGBE Tracking

Abstract

RGBE object tracking is an emerging field that integrates RGB frames and event data to achieve more robust tracking results, particularly in challenging scenarios. However, existing methodologies predominantly focus on transforming sparse event streams into event frames, thereby neglecting the potential of rich temporal information. To address this limitation, we introduce Spiking Neural Network-based Prompt Tracking (SNNPTrack), a hybrid framework designed for temporally adaptive RGBE tracking, aimed at achieving high-accuracy tracking performance. SNNPTrack includes a Leaky Integrate-and-Fire (LIF)-based Spiking Neural Network (SNN) module for temporal feature extraction, a Cross-Modality Fusion module for feature fusion across both domains, and a pre-trained RGB-based transformer model for dual-modal feature extraction and interaction. Extensive experimental evaluations demonstrate that, with only a modest increase in the number of parameters, our SNNPTrack framework surpasses state-of-the-art methods on the VisEvent, FE108, and COESOT datasets, highlighting its potential as a promising solution for real-world tracking applications.