Simplified learning with binary orthogonal constraints

Abstract

Deep architecture based Deep Brief Nets (DBNs) has shown its data modelling power by stacking up several Restricted Boltzmann Machines (RBMs). However, the multiple-layer structure used in DBN brings expensive computation, and furthermore leads to slow convergence. This is because the pretraining stage is usually implemented in a data-driven way, and class information attached to the training data is only used for fine-tuning. In this paper, we aim to simplify a multiple-layer DBN to a one-layer structure. We use class information as a constraint to the hidden layer during pre-training. For each training instance and its corresponding class, a binary sequence will be generated in order to adapt the output of hidden layer. We test our approaches on four data sets: basic, MNIST, basic negative MNIST, rotation MNIST and rectangle (tall vs. wide rectangles). The obtained results show that the adapted one-layer structure can compete with a three-layer, DBN.