Sign-Mamba: Advanced Mamba-Based Sign Language Generation

Abstract

In the field of sign language generation, Transformer-based models have been widely studied, but their quadratic computational complexity poses challenges. State Space Models (SSMs), like Mamba, offer a promising alternative with efficient long-range interaction modeling and linear complexity. In this study, we propose a two-stage generative framework for sign language generation, called Sign-Mamba, based on the state selection mechanism of SSM. In the first stage, we designed a Mamba-based encoder-decoder architecture, where the encoder captures the latent space representation and a symmetric decoder reconstructs it back into sign language skeletal points. In the second stage, the sign language latent space predicted by the Mamba-based latent predictor is used as a condition to guide the reconstruction network in generating more precise skeletal point sequences. We conducted comprehensive experiments on the PHOENIX and How2Sign datasets, and the results indicate that Sign-Mamba demonstrates competitive performance in sign language generation tasks. Project page: https://peterfanfan.github.io/Sign-Mamba/