A Marginal Log-Likelihood Approach for the Estimation of Discount Factors of Multiple Experts in Inverse Reinforcement Learning

Abstract

We focus on multiple experts performing a task in a Markov decision process (MDP) environment. A probabilistic assignment of trajectories to clusters and a mathematical framework which leverages the utility function are employed to jointly estimate the discount factor and reward. We treat the number of clusters as a hyperparameter which can be "freely" selected by the problem designer. In this work, we specifically treat the cluster of trajectories as a latent variable in the adapted maximum entropy inverse reinforcement learning (IRL) formulation; the introduction of this latent variable adds to the complexity of the IRL problem. To manage such complexity, we optimize a marginal log-likelihood function via Expectation Maximization. To test our approach, we have utilized behavioral data generated from three MDP environments. Experimental works show that our approach is promising towards the estimation of discount factors in IRL for non-interacting multiple experts.