A Training Method Using DNN-guided Layerwise Pretraining for Deep Gaussian Processes

Abstract

This paper proposes a novel framework of training method of deep Gaussian processes (DGPs). DGPs are deep architecture models based on stacked multiple GPs, which can overcome the limitation of single-layer GPs. Although a stochastic variational inference (SVI)- based method has been proposed for DGP training with an arbitrary amount of training data, it does not always update model parameters appropriately due to repeated Monte Carlo sampling of multiple GPs. To resolve this problem, we propose a pretraining method to determine initial parameters of DGPs. In the proposed method, layer-wise training of single-layer GPs is performed using the hidden-layer values obtained by a deep neural network (DNN) that has an analogous structure to a target DGP model. The proposed method utilizes the characteristics of single-layer GPs and deep neural network (DNN) whose training is easier than DGP's. Experimental results using two speech synthesis databases with approximately 600 K and 1.4 M training data points, respectively, which were composed of hundreds of input and output features, gave the effectiveness of the proposed method.