Invariant Random Forest: Tree-Based Model Solution for OOD Generalization
Abstract
Out-Of-Distribution (OOD) generalization is an essential topic in machine learning. However, recent research is only focusing on the corresponding methods for neural networks. This paper introduces a novel and effective solution for OOD generalization of decision tree models, named Invariant Decision Tree (IDT). IDT enforces a penalty term with regard to the unstable/varying behavior of a split across different environments during the growth of the tree. Its ensemble version, the Invariant Random Forest (IRF), is constructed. Our proposed method is motivated by a theoretical result under mild conditions, and validated by numerical tests with both synthetic and real datasets. The superior performance compared to non-OOD tree models implies that considering OOD generalization for tree models is absolutely necessary and should be given more attention.
BibTeX
@article{Liao_Wu_Yan_2024, title={Invariant Random Forest: Tree-Based Model Solution for OOD Generalization}, volume={38}, url={https://ojs.aaai.org/index.php/AAAI/article/view/29283}, DOI={10.1609/aaai.v38i12.29283}, abstractNote={Out-Of-Distribution (OOD) generalization is an essential topic in machine learning. However, recent research is only focusing on the corresponding methods for neural networks. This paper introduces a novel and effective solution for OOD generalization of decision tree models, named Invariant Decision Tree (IDT). IDT enforces a penalty term with regard to the unstable/varying behavior of a split across different environments during the growth of the tree. Its ensemble version, the Invariant Random Forest (IRF), is constructed. Our proposed method is motivated by a theoretical result under mild conditions, and validated by numerical tests with both synthetic and real datasets. The superior performance compared to non-OOD tree models implies that considering OOD generalization for tree models is absolutely necessary and should be given more attention.}, number={12}, journal={Proceedings of the AAAI Conference on Artificial Intelligence}, author={Liao, Yufan and Wu, Qi and Yan, Xing}, year={2024}, month={Mar.}, pages={13772-13781} }