Optimal One-hot Logistic Regression for Tree-based Distribution Classification

Abstract

The importance of features interactions for classification tasks between two classes is proven by the abundance of high-performance machine learning methods using them. In this paper, we take our inspiration from Bayesian Networks (BNs), a famous classification method whose main asset is the graphical representation of links between features. In the case of discrete feature, we assume that the features follow a tree-based distribution under each class. Then, we show that the optimal Bayes classifier coincides with a logistic regression whose features are one-hot encoded with a specific scheme. We show the advantages of such an approach both theoretically and numerically, especially showing that the learning step always converges toward a unique solution. Simulated experiments confirm the efficiency of the one-hot logistic regression with feature interaction encoding.