3DSignDiff: Towards 3D Sign Language Gesture Generation

Abstract

Sign language is vital for communication within the deaf and hard-of-hearing community. Deep learning techniques, such as Generative Adversarial Networks and diffusion models, have shown promise in generating sign language video content. However, existing methods typically focus on 2D sign language video generation, with limited exploration into 3D sign language generation. A prominent issue is the lack of an available 3D sign language dataset for training and evaluating 3D sign language video generation models. To address this, we introduce a new dataset called How2Gesture, which contains long sequences of 3D sign language actions along with detailed text translations. We also propose a novel 3D sign language generation paradigm, named 3DSignDiff. This model is based on a Transformer encoder, directly conditioning on long text sequences, and uses a diffusion model as a denoiser. It also employs text context enhancement techniques to strengthen the guidance of text on sign language actions. Additionally, an advanced joint generation approach is used to generate human sign language actions in stages. Extensive experiments conducted on the open-source How2Gesture dataset validate the superiority of our model. How2Geature dataset is available at https://github.com/lifeisawar41/3Dsigndiff.