Word and Class Common Space Embedding for Code-switch Language Modelling

Abstract

Code-switch language modelling is challenging due to limited linguistic resources and less predictable word sequences. Many state-of-the-art systems rely on linguistic information such as Part-of-Speech (POS) or classes to generalize the lexicon. Such systems generally use multi-task learning or conditional network to improve over baseline RNN language model by providing a better word prediction. To overcome the data sparsity through continuous space modelling and back-off mechanism, we propose to constrain the word and class embedding in a common space by means of cross-lingual word embedding, and to make use of the predicted class embedding as a back-off scheme when word prediction model is weak. The proposed word and class Common Space embedding Language Model (CSLM) is able to model word prediction better and is more robust when only sparse training data are available. The CSLM outperforms the state-of-the-art language model by 9.7% on the code-switch SEAME corpus.