A Direct Approximation of AIXI Using Logical State Abstractions
Samuel Yang-Zhao, Tianyu Wang, Kee Siong Ng
Abstract
We propose a practical integration of logical state abstraction with AIXI, a Bayesian optimality notion for reinforcement learning agents, to significantly expand the model class that AIXI agents can be approximated over to complex history-dependent and structured environments. The state representation and reasoning framework is based on higher-order logic, which can be used to define and enumerate complex features on non-Markovian and structured environments. We address the problem of selecting the right subset of features to form state abstractions by adapting the $\Phi$-MDP optimisation criterion from state abstraction theory. Exact Bayesian model learning is then achieved using a suitable generalisation of Context Tree Weighting over abstract state sequences. The resultant architecture can be integrated with different planning algorithms. Experimental results on controlling epidemics on large-scale contact networks validates the agent's performance.
BibTeX
@inproceedings{
yang-zhao2022a,
title={A Direct Approximation of {AIXI} Using Logical State Abstractions},
author={Samuel Yang-Zhao and Tianyu Wang and Kee Siong Ng},
booktitle={Advances in Neural Information Processing Systems},
editor={Alice H. Oh and Alekh Agarwal and Danielle Belgrave and Kyunghyun Cho},
year={2022},
url={https://openreview.net/forum?id=92leLHqlcvv}
}