Automatic Temporal Segmentation of Hand Movements for Hand Positions Recognition in French Cued Speech

Abstract

In the context of Cued Speech (CS) recognition, the recognition of lips and hand movements is a key task. As we know, a good temporal segmentation is necessary for the supervised recognition system. However, lips and hand streams cannot share the same temporal segmentation since they are not synchronized. In this work, we propose a hand preceding model to predict temporal segmentations of hand movements automatically by exploring the relationship between hand preceding time and the vowel positions in sentences. To evaluate the performance of the proposed method, we apply the hand preceding model to a multi -speakers database. Hand positions recognition is realized with the multi-Gaussian and Long-Short Term Memory (LSTM). The results show that using the predicted temporal segmentation significantly improves the recognition performance compared with that using the audio based segmentation. To the best of our knowledge, this is the first automatic method to predict the temporal segmentation for hand movements only from the audio based segmentation in CS.