Meta-level tracking for gestural intent recognition

Abstract

In this paper, a novel mode-driven switching state space approach is proposed for the joint tracking and recognition of gestural commands. Gestures are modeled as spatio-temporal patterns comprised of syntactic sub-units called gesturelets. These gesturelets are directional vectors modulating a switching state space model. Stochastic context-free grammars (SCFG) are used as generative models for command gestures which impart a scale-invariant modeling framework. This translates into a method that is user-independent and robust to the signing variation between and among users. In addition to the modeling framework, we also design a library of useful gestural patterns that cannot be represented by regular grammars (hidden Markov models). Our approach combines tracking and recognition in a single framework and is able to deal with a high perplexity dataset. We demonstrate the effectiveness of our approach by comparing SCFG models with HMM models on synthetic gesture trajectories.