Trainable Co-Occurrence Activation Unit for Improving Convnet

Abstract

A deep neural network is one of the promising approach to produce state-of-the-art performance on various fields such as pattern recognition and signal processing. While the network architecture is intensively studied, as to the network components, non-linear activation functions are the main subject of research in the literature. Most of the activation functions, such as a rectified linear unit (ReLU), operate on each of feature channels in an element-wise manner and thus can be regarded as extracting occurrence characteristics from the input feature map. In this paper, we propose a co-occurrence activation unit to work across feature channels by extending the element-wise activation function. In contrast to the original co-occurrence formulation applied to hand-crafted feature extraction methods, the proposed co-occurrence unit is trainable by a gradient-based optimization through back-propagation learning and exploits the co-occurrence relationships among the feature channels. The experimental results on image classification datasets show that the proposed co-occurrence activation unit embedded into various types of ConvNets favorably improve classification performance.