A Predictive Reward Function for Human-Like Driving Based on a Transition Model of Surrounding Environment

Abstract

Driving is a complex task that requires the perception of the surrounding environment, decision making and control of the vehicle. Human drivers predict how surrounding objects move and decide an appropriate driving behavior. As with human drivers, autonomous driving vehicles should consider the condition of the surrounding environment and behave naturally so as not to disturb the traffic flow. We propose a reward function for learning how natural the driving is based on the hypothesis that the movement of surrounding vehicles becomes unpredictable when the ego vehicle takes an unnatural driving behavior. The reward function is based on the prediction error of a deep predictive network that models the transition of the surrounding environment. Occupancy grid image is used to perceive the surrounding environment and the predictions up to two seconds are used to calculate the reward function. We evaluated the reward function using both simulated and the real world data. We trained the prediction network using real driving data and trained a reinforcement learning agent based on the reward function. Then we compared the speed planned by the agent and a human driver, which showed a correlation of 0.52. We also confirmed the benefit of taking prediction into account by observing the behavior of the agent in a specific traffic scenario.