CSNet: Contrastive Siamese Network for Robust SLU

Abstract

Automatic speech recognition (ASR) results based on clean references are much more accurate than those based on ASR transcripts in spoken language understanding (SLU). Effective utilization of manually-checked clean transcripts is key to improving SLU performance. This paper proposes a siamese network with contrastive learning to enhance SLU effects. A siamese network on sentence pairs that are composed of ASR transcripts and clean transcripts is used for the SLU task. During training, contrastive learning brings closer the sentence-level semantic representations of ASR transcripts and clean transcripts. During inference, k-nearest neighbors (KNN) semantic search via the siamese network first finds the pseudo clean transcript, then forms a sentence pair based on the ASR transcript and pseudo clean transcript for prediction. Experiments on three benchmark datasets prove the effectiveness of our proposed approach, which improves the Intent Classification (IC) performance by over 1.3% on the SLURP dataset.