SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks
Xiubo Liang, Jinxing Han, Yuke Li, Haoqi Zhu, Yu Zhao, Hongzhi Wang
Abstract
Spiking Neural Networks (SNNs) enable event-driven computation with sparse activations, but building multimodal Transformers on SNNs is hindered by unstable training in deep spiking stacks and a mismatch between dense softmax attention and spike-based communication. We propose SMM Transformer, an SNN-based multimodal Transformer framework that combines (i) a Parallel LIF with Multistage Learnable Parameters (PLMP) neuron and a tailored P-STBP algorithm to stabilize training, (ii) a spike-driven attention approximation (SMSA) with a lightweight self-compensation branch, and (iii) a spiking mixture-of-experts (SMoE) module for modality-aware fusion. Across visual and multimodal benchmarks, SMM Transformer achieves competitive accuracy compared to ANN baselines while reducing the estimated compute energy of the attention module by up to 97\% under a standard MAC/AC cost model.
BibTeX
@inproceedings{
liang2026smm,
title={{SMM} Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks},
author={Xiubo Liang and Jinxing Han and Yuke Li and Haoqi Zhu and Yu Zhao and Hongzhi Wang},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=aQmrtnFhqw}
}