PhISANet: Phonetically Informed Speech Animation Network
Salvador Medina, Sarah L. Taylor, Carsten Stoll, Gareth Edwards, Alex Hauptmann, Shinji Watanabe, Iain A. Matthews
Abstract
Realistic animation is crucial for immersive and seamless human-avatar interactions as digital avatars become more prevalent. This work presents PhISANet, an encoder-decoder model that realistically animates the face and tongue solely from speech. PhISANet leverages neural audio representations trained on vast amounts of speech to map the speech signal into animation parameters that control the lower face and tongue of realistic 3D models. By integrating a novel multi-task learning strategy during the training phase, PhISANet reincorporates the phonetic information from the input speech, improving articulation in the generated animations. A thorough quantitative and qualitative study validates this improvement, and it determines that WavLM and Whisper features are ideal for training a generalizable speech-animation model regardless of gender, age, and language.
BibTeX
@inproceedings{icassp2024_phisanetphonetic,
title = {PhISANet: Phonetically Informed Speech Animation Network},
author = {Salvador Medina and Sarah L. Taylor and Carsten Stoll and Gareth Edwards and Alex Hauptmann and Shinji Watanabe and Iain A. Matthews},
booktitle = {ICASSP 2024},
year = {2024}
}