IR-VIC: Unsupervised Discovery of Sub-goals for Transfer in RL

Nirbhay Modhe; Prithvijit Chattopadhyay; Mohit Sharma; Abhishek Das; Devi Parikh; Dhruv Batra; Ramakrishna Vedantam

IJCAI 2020poster0 citations

IR-VIC: Unsupervised Discovery of Sub-goals for Transfer in RL

Nirbhay Modhe, Prithvijit Chattopadhyay, Mohit Sharma, Abhishek Das, Devi Parikh, Dhruv Batra, Ramakrishna Vedantam

Abstract

We propose a novel framework to identify sub-goals useful for exploration in sequential decision making tasks under partial observability. We utilize the variational intrinsic control framework (Gregor et.al., 2016) which maximizes empowerment -- the ability to reliably reach a diverse set of states and show how to identify sub-goals as states with high necessary option information through an information theoretic regularizer. Despite being discovered without explicit goal supervision, our sub-goals provide better exploration and sample complexity on challenging grid-world navigation tasks compared to supervised counterparts in prior work.

Machine Learning: Deep Reinforcement LearningMachine Learning: Reinforcement LearningMachine Learning: Unsupervised Learning

BibTeX

@inproceedings{ijcai2020p280,
  title     = {IR-VIC: Unsupervised Discovery of Sub-goals for Transfer in RL},
  author    = {Modhe, Nirbhay and Chattopadhyay, Prithvijit and Sharma, Mohit and Das, Abhishek and Parikh, Devi and Batra, Dhruv and Vedantam, Ramakrishna},
  booktitle = {Proceedings of the Twenty-Ninth International Joint Conference on
               Artificial Intelligence, {IJCAI-20}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Christian Bessiere},
  pages     = {2022--2028},
  year      = {2020},
  month     = {7},
  note      = {Main track},
  doi       = {10.24963/ijcai.2020/280},
  url       = {https://doi.org/10.24963/ijcai.2020/280},
}