Towards Dynamic Skeleton-based Handshape Subunits for Sign Language Assessment

Abstract

Sign languages convey information through multiple channels. The handshape channel is an important manual component for conveying the message. In the literature, it is mainly modeled as a sequence of images of discrete postures even in the case of dynamic gestures, leading to blurring problems in detection. Furthermore, to model these discrete postures using deep learning frame-level labeling of the sign language videos is also required, which is time consuming and human intensive. In this paper, as opposed to modeling the handshape information through images of discrete postures, we propose dynamic modeling through skeletal information. More precisely, we develop an approach that combines HamNoSys-based prior knowledge and sign language data to derive dynamic handshape units by modeling skeletal features using hidden Markov models. We demonstrate the effectiveness of the proposed approach through sign language assessment study, sign language recognition, and handshape recognition analysis on the SMILE DSGS corpus.