Information-Theoretic Methods in Deep Neural Networks: Recent Advances and Emerging Opportunities

Shujian Yu; Luis Sanchez Giraldo; Jose Principe

IJCAI 2021poster20 citations

Information-Theoretic Methods in Deep Neural Networks: Recent Advances and Emerging Opportunities

Shujian Yu, Luis Sanchez Giraldo, Jose Principe

Abstract

We present a review on the recent advances and emerging opportunities around the theme of analyzing deep neural networks (DNNs) with information-theoretic methods. We first discuss popular information-theoretic quantities and their estimators. We then introduce recent developments on information-theoretic learning principles (e.g., loss functions, regularizers and objectives) and their parameterization with DNNs. We finally briefly review current usages of information-theoretic concepts in a few modern machine learning problems and list a few emerging opportunities.

Machine learning: General

BibTeX

@inproceedings{ijcai2021p633,
  title     = {Information-Theoretic Methods in Deep Neural Networks: Recent Advances and Emerging Opportunities},
  author    = {Yu, Shujian and Sanchez Giraldo, Luis and Principe, Jose},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Zhi-Hua Zhou},
  pages     = {4669--4678},
  year      = {2021},
  month     = {8},
  note      = {Survey Track},
  doi       = {10.24963/ijcai.2021/633},
  url       = {https://doi.org/10.24963/ijcai.2021/633},
}