Evaluation of weight sparsity regularizion schemes of deep neural networks applied to functional neuroimaging data

Abstract

The paper presented a systematic evaluation of the weight sparsity regularization schemes for the deep neural networks applied to the whole brain resting-state functional magnetic resonance imaging data. The weight sparsity regularization was deployed between the visible and hidden layers of the Gaussian-Bernoulli restricted Boltzmann machine (GB-RBM), in which the L0-norm based non-zero value ratio and L1-/L2-norm based Hoyer's sparseness were used to define the weight sparsity. Also, the weight sparsity regularization schemes between the two consecutive layers (i.e. layer-wise) and between the layer and the node in the subsequent layer (i.e. node-wise) were compared in terms of the convergence property. Finally, the reproducibility of 10 sets of weight features extracted from the GB-RBMs trained using 10 sets of random initial weights was evaluated.