DiMS: Distilling Multiple Steps of Iterative Non-Autoregressive Transformers for Machine Translation
The computational benefits of iterative non-autoregressive transformers decrease as the number of decoding steps increases. As a remedy, we introduce Distill Multiple Steps (DiMS), a simple yet effective distillation technique to decrease the number of required steps to reach a certain translation q…