Neural Codes to Factor Language in Multilingual Speech Recognition

Abstract

In the past, we adapted neural network based multilingual acoustic models using language codes. In this work, we study the extracted language codes and the language properties they encode: We use the codes to generate language prototype vectors, which represent the features of a language. Computing distances between prototype vectors shows that languages from the same family have smaller distances. This structure found within the feature representation supports the assumption that language codes do encode language information and not other properties like, e.g. channel characteristics, and in addition providing a richer language representation than the language identity alone.The network architecture of our system is based on a factorized model, which consists of multiple language dependent subnets. While we recently demonstrated that this approach enables multilingual setups to outperform monolingual ones, we here propose further optimizations. We evaluated using a) more language dependent subnets and b) wider BiLSTM layers. Our results indicate that using a larger number of language dependent subnets increases the system performance and renders phonetic pretraining superfluous. In addition, increasing the size of the hidden layers further improved the performance, with the system now outperforming the monolingual baseline by 6.3% relative.