Deterministic Transform Based Weight Matrices for Neural Networks

Abstract

We propose to use deterministic transforms as weight matrices for several feedforward neural networks. The use of deterministic transforms helps to reduce the computational complexity in two ways: (1) matrix-vector product complexity in forward pass, helping real time complexity, and (2) fully avoiding backpropagation in the training stage. For each layer of a feedforward network, we pro-pose two unsupervised methods to choose the most appropriate deterministic transform from a set of transforms (a bag of well-known transforms). Experimental results show that the use of deterministic transforms is as good as traditional random matrices in the sense of providing similar classification performance.