6 accepted papers
The foundation of current large language model applications lies in the generative language model, which typically employs an autoregressive token generation approach. However, this model faces two key limitations: its unidirectional causal attention mechanism restricts semantic expressiveness, and…
Masked language model pretraining provides a standardized way to learn contextualized semantic representations, which reconstructs corrupted text sequences by estimating the conditional probabilities of randomly masked tokens given the context. We attempt to exploit language knowledge from the model…
Knowledge transfer can boost neural machine translation (NMT), for example, by finetuning a pretrained masked language model (LM). However, it may suffer from the forgetting problem and the structural inconsistency between pretrained LMs and NMT models. Knowledge distillation (KD) may be a potential…
Self-attention mechanism has been shown to be an effective approach for capturing global context dependencies in sequence modeling, but it suffers from quadratic complexity in time and memory usage. Due to the sparsity of the attention matrix, much computation is redundant. Therefore, in this paper,…
Connectionist Temporal Classification (CTC) model has achieved state-of-the-art LVCSR performance. However, due to the introduction of the blank symbol, word-level confidence measures (CM) based on CTC model can not be easily calculated by directly using the traditional phone posterior normalization…
Although context-dependent DNN-HMM systems have achieved significant improvements over GMM-HMM systems, there still exists big performance degradation if the acoustic condition of the test data mismatches that of the training data. Hence, adaptation and adaptive training of DNN are of great research…