Information and Regularization in Restricted Boltzmann Machines

Abstract

Recent works suggests an interesting interplay between the information flow between inputs features and hidden representations of a learning and the ability of the algorithm to generalize from trained samples to unobserved data. For instance, some of regularization techniques used to control generalization are expected to impact the corresponding information metrics. In this work, we study mutual information in Restricted Boltzmann Machines (RBM) and its relationship with the different regularization techniques. Our results show some evidence on interesting connections between the mutual information (inputs and its representations) with relevant parameters such as: network dimension, matrix norms and dropout probability, which are known to influence the generalization ability of the network. Results are empirically corroborated with a numerical study.