Keeping Your Eyes on the Fingertip: A Two-Stage In-Air Handwritten Recognition Method

Abstract

As human-computer interaction methods diversify, in-air handwritten recognition has gained growing research interest. However, existing camera-based in-air handwritten recognition models typically overlook a crucial fact: the background information in the video, despite its large presence, is irrelevant to the recognition task. In this study, we propose a two-stage model that first extracts spatial features around the writer’s index fingertip in each video frame by a fingertip feature extraction module, then encodes these features along the temporal dimension through a text recognition module, and ultimately performs CTC decoding for prediction. Our method enables the text recognition module to focus on features near the writer’s index fingertip in the video, allowing the module to concentrate on the recognition task without being distracted by background features. Multiple experiments demonstrate superior recognition accuracy compared to state-of-the-art methods, while utilizing only about half the parameters. Our project is open-sourced at https://github.com/Mayo001/FETR.