Modeling Intensification for Sign Language Generation: A Computational Approach
Mert Inan, Yang Zhong, Sabit Hassan, Lorna Quandt, Malihe Alikhani
Abstract
End-to-end sign language generation models do not accurately represent the prosody in sign language. A lack of temporal and spatial variations leads to poor-quality generated presentations that confuse human interpreters. In this paper, we aim to improve the prosody in generated sign languages by modeling intensification in a data-driven manner. We present different strategies grounded in linguistics of sign language that inform how intensity modifiers can be represented in gloss annotations. To employ our strategies, we first annotate a subset of the benchmark PHOENIX-14T, a German Sign Language dataset, with different levels of intensification. We then use a supervised intensity tagger to extend the annotated dataset and obtain labels for the remaining portion of it. This enhanced dataset is then used to train state-of-the-art transformer models for sign language generation. We find that our efforts in intensification modeling yield better results when evaluated with automatic metrics. Human evaluation also indicates a higher preference of the videos generated using our model.
BibTeX
@inproceedings{inan-etal-2022-modeling,
title = "Modeling Intensification for Sign Language Generation: A Computational Approach",
author = "Inan, Mert and
Zhong, Yang and
Hassan, Sabit and
Quandt, Lorna and
Alikhani, Malihe",
editor = "Muresan, Smaranda and
Nakov, Preslav and
Villavicencio, Aline",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2022",
month = may,
year = "2022",
address = "Dublin, Ireland",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.findings-acl.228/",
doi = "10.18653/v1/2022.findings-acl.228",
pages = "2897--2911"
}