Integration of orthogonal feature detectors in parameter learning of artificial neural networks to improve robustness and the evaluation on hand-written digit recognition tasks

Abstract

We propose to use orthogonal feature detectors in artificial neural networks for the robustness of performance under noisy conditions. The motivation is grounded on the principle that orthogonal decomposition is the most efficient among all representation of a signal. In this paper, we incorporate orthogonalization in the process of learning the network weights. In our implementation, the constraint of orthogonality is enforced by applying Gram-Schmidt processes to the feature detectors during network training. The proposed method is evaluated on MNIST database for hand-written digit recognition. The images in the training set are not corrupted, while the images in the test set are artificially corrupted with white noises. Experimental results show that the proposed orthogonalization method achieves 56.4% relative improvement in recognition error rate over a conventional learning method without orthogonalization. Given that the clean training data and the noisy test data are clearly mismatched, such an improvement with artificial neural networks is indeed very remarkable. For engineering insight, we devise a visualization tool which illuminates interesting features of the neurons learned by the proposed method.