Gradient Boosting Performs Gaussian Process Inference

Aleksei Ustimenko; Artem Beliakov; Liudmila Prokhorenkova

ICLR 2023poster3 citations

Gradient Boosting Performs Gaussian Process Inference

Aleksei Ustimenko, Artem Beliakov, Liudmila Prokhorenkova

Abstract

This paper shows that gradient boosting based on symmetric decision trees can be equivalently reformulated as a kernel method that converges to the solution of a certain Kernel Ridge Regression problem. Thus, we obtain the convergence to a Gaussian Process' posterior mean, which, in turn, allows us to easily transform gradient boosting into a sampler from the posterior to provide better knowledge uncertainty estimates through Monte-Carlo estimation of the posterior variance. We show that the proposed sampler allows for better knowledge uncertainty estimates leading to improved out-of-domain detection.

gradient boostinggaussian processknowledge uncertaintykernel gradient boosting

BibTeX

@inproceedings{
ustimenko2023gradient,
title={Gradient Boosting Performs Gaussian Process Inference},
author={Aleksei Ustimenko and Artem Beliakov and Liudmila Prokhorenkova},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=3VKiaagxw1S}
}