InstantSpeech: Instant Synchronous Text-to-Speech Synthesis for LLM-driven Voice Chatbots

Abstract

Chatbots powered by large language models (LLMs) offer natural, human-like interactions. However, traditional text-to-speech (TTS) models paired with LLMs typically wait for the entire sentence to be generated before starting synthesis, leading to increased response latency. Although word-by-word speech synthesis models have been proposed to address this issue, they still face challenges, such as relying on autoregressive architectures to maintain smooth transitions between words or conditioning on auxiliary features from LLM for naturalness. To overcome these limitations, we introduce InstantSpeech, a novel low-latency synchronous speech synthesis model. InstantSpeech employs a fully parallel architecture, combining a causal transformer-based acoustic model with a causal convolution-based vocoder, enabling it to start streaming speech synthesis immediately after the LLM generates the initial words. Furthermore, we utilize knowledge distillation to enhance speech quality under limited lookahead. Experimental results show that InstantSpeech can deliver high-quality speech with consistently low speech response latency when integrated with LLMs of various sizes.