LERMO: A Novel Web Game for AI-Enhanced Sign Language Recognition
Adilson Medronha, Luís Lima, Janaína Claudio, Lucas Kupssinskü, Rodrigo C. Barros
Abstract
Sign language is a visual and gestural communication system used by deaf and hearing-impaired people. Despite numerous deep learning methods proposed for automatic interpretation, a gap persists in developing applications that effectively utilize these models for assisting sign language studies and inclusion. We introduce LERMO (https://lermo.app/), a web game merging machine learning and gamification to enhance sign language fingerspelling. Inspired by Wordle™, LERMO offers an interactive word-guessing game where users can play using a video camera. We create a new dataset of labeled landmark fingerspelling and design our model to ensure optimal speed and efficiency to run on a web browser. We survey approximately 40 users, which find LERMO user-friendly and innovative. From those, 95% believe LERMO could be used to enhance fingerspelling skills.
BibTeX
@article{Medronha_Lima_Claudio_Kupssinskü_Barros_2024, title={LERMO: A Novel Web Game for AI-Enhanced Sign Language Recognition}, volume={38}, url={https://ojs.aaai.org/index.php/AAAI/article/view/30384}, DOI={10.1609/aaai.v38i21.30384}, abstractNote={Sign language is a visual and gestural communication system used by deaf and hearing-impaired people. Despite numerous deep learning methods proposed for automatic interpretation, a gap persists in developing applications that effectively utilize these models for assisting sign language studies and inclusion. We introduce LERMO (https://lermo.app/), a web game merging machine learning and gamification to enhance sign language fingerspelling. Inspired by Wordle™, LERMO offers an interactive word-guessing game where users can play using a video camera. We create a new dataset of labeled landmark fingerspelling and design our model to ensure optimal speed and efficiency to run on a web browser. We survey approximately 40 users, which find LERMO user-friendly and innovative. From those, 95% believe LERMO could be used to enhance fingerspelling skills.}, number={21}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Medronha, Adilson and Lima, Luís and Claudio, Janaína and Kupssinskü, Lucas and Barros, Rodrigo C.}, year={2024}, month={Mar.}, pages={23352-23359} }