Lightweight neural front-ends for low-resource on-device Text-to-Speech

Abstract

We propose a lightweight neural front-end framework for on-device speech generation and highlight its benefits towards low-resource language scaling. While data-driven models have shown potential in front-end literature, especially since they can enable fast language expansion, they are often extremely large and of high latency. There is limited work focusing on their usability in real-time settings, and none for on-device TTS applications. At the cost of small performances trade-offs, we build lightweight neural Grapheme-to-Phoneme and verbalization models which achieve, on average across three languages, a p90 latency reduction of 95.98% per token on single-threaded CPU@2.30GHz, with respect to a traditional transformer-based baseline, while having 99.26% less parameters. Additionally, leveraging pre-trained teacher models to bootstrap lightweight students, we enable low-resource language scaling on both Grapheme-to-Phoneme conversion and verbalization.