Positional Encoding for Spiking Transformers

Abstract

Spiking Neural Networks (SNNs) demonstrate superior energy efficiency over conventional Artificial Neural Networks (ANNs). Recent advances in Transformer-based SNNs have shown encouraging performance by seamlessly integrating spike-driven computation with Transformer architectures. Positional information plays a crucial role in sequential modeling tasks. However, existing positional encoding methods designed for ANNs are fundamentally incompatible with SNNs, as they interfere with the spike-driven computation paradigm, highlighting the need for SNN-specific solutions. We propose Spiking Positional Encoding (SPE), a novel positional encoding method specifically designed for Spiking Transformers that effectively captures relative positional information. Its key component is the Positional Encoding Leaky Integrate-and-Fire (PE-LIF) neuron layer, which encodes positional information directly into neuron thresholds. Through continuous spike firing and membrane potential reset processes, this positional information is implicitly embedded into the emitted spike trains while maintaining compatibility with the spike-driven computation paradigm. Comprehensive experiments across thirteen datasets, including the GLUE and other widely-adopted Natural Language Processing benchmarks, demonstrate that SPE consistently outperforms existing positional encoding methods. SPE provides a tailored positional encoding solution for Spiking Transformers, bridging the performance gap between ANNs and SNNs, thus advancing neuromorphic computing applications in sequential modeling tasks.