Dual Decoder for Fast Inference in Natural Language Generation

Abstract

Natural language generation is an important task in natural language processing and has been applied in various scenarios. Most state-of-the-art generation models, however, are usually slow at inference time mainly due to the sequential dependencies of autoregressive generation and the use of more and more large-scale decoder models. To this end, we propose a Dual-Decoder (Dude) model to speed up the decoder without sacrificing the overall model performance. Dude model is composed of a semantic decoder and an output decoder, which are able to capture the long-term semantic dependencies and predict the target sequence fast as well. We evaluate Dude model on three natural language generation tasks including Neural Machine Translation, Text Summarization and Question Generation. The experimental results demonstrate that our model achieves 1.43× faster inference speed than the standard baseline while maintaining comparable performance, and even 1.62× faster on longer sequence generation tasks.