Learning Motion Disfluencies for Automatic Sign Language Segmentation

Abstract

We introduce a novel technique for the automatic detection of word boundaries within continuous sentence expressions in Japanese Sign Language from three-dimensional body joint positions. First, the flow of signed sentence data within a temporal neighborhood is determined utilizing the spatial correlations between line segments of inter-joint pairs. Next, a frame-wise binary random forest classifier is trained to distinguish word and non-word frame content based on the extracted spatio-temporal features. The output of the classifier is used to propose an automatic word synthesis that achieves reliable and accurate sentence segmentation with an average frame-wise F1 score of 0.89. Evaluation with a baseline data set furthermore shows that the proposed approach can easily be adapted to distinguish between motion transitions and motion primitives for a coarse-action domain.