Non-Autoregressive Multimodal Machine Translation

Abstract

Performing better text translation by integrating auxiliary inputs from visual information has gained widespread attention in recent years. While existing methods outperform the text-only translation models, the step-by-step generative style reduces the inference speed, which limits their applicability in real-world scenarios. In this paper, we propose the non-autoregressive language model (NA-LM) for multimodal machine translation. With NA-LM, we develop a Non-Autoregressive Multimodal Transformer (NA-MMT), which accelerates the generative translation via a parallel multimodal decoder. To retain the translation performance, we improve the NA-MMT in twofold: 1) We preprocess the image into a refined sequence of visual entities with length encoding to reduce irrelevant information; 2) We design cross fertility and cross-modal gate attention for multimodal decoder to enhance the generative quality. Experiments on Multi30k datasets show the NA-MMT can generate high-quality translation with over 11× speedup than the baselines, which is strongly competitive.