Analysis of Deep Neural Networks with Extended Data Jacobian Matrix
Shengjie Wang, Abdel-rahman Mohamed, Rich Caruana, Jeff Bilmes, Matthai Plilipose, Matthew Richardson, Krzysztof Geras, Gregor Urban
Abstract
Deep neural networks have achieved great successes on various machine learning tasks, however, there are many open fundamental questions to be answered. In this paper, we tackle the problem of quantifying the quality of learned wights of different networks with possibly different architectures, going beyond considering the final classification error as the only metric. We introduce \emphExtended Data Jacobian Matrix to help analyze properties of networks of various structures, finding that, the spectrum of the extended data jacobian matrix is a strong discriminating factor for networks of different structures and performance. Based on such observation, we propose a novel regularization method, which manages to improve the network performance comparably to dropout, which in turn verifies the observation.
BibTeX
@InProceedings{pmlr-v48-wanga16,
title = {Analysis of Deep Neural Networks with Extended Data Jacobian Matrix},
author = {Wang, Shengjie and Mohamed, Abdel-rahman and Caruana, Rich and Bilmes, Jeff and Plilipose, Matthai and Richardson, Matthew and Geras, Krzysztof and Urban, Gregor and Aslan, Ozlem},
booktitle = {Proceedings of The 33rd International Conference on Machine Learning},
pages = {718--726},
year = {2016},
editor = {Balcan, Maria Florina and Weinberger, Kilian Q.},
volume = {48},
series = {Proceedings of Machine Learning Research},
address = {New York, New York, USA},
month = {20--22 Jun},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v48/wanga16.pdf},
url = {https://proceedings.mlr.press/v48/wanga16.html},
abstract = {Deep neural networks have achieved great successes on various machine learning tasks, however, there are many open fundamental questions to be answered. In this paper, we tackle the problem of quantifying the quality of learned wights of different networks with possibly different architectures, going beyond considering the final classification error as the only metric. We introduce \emphExtended Data Jacobian Matrix to help analyze properties of networks of various structures, finding that, the spectrum of the extended data jacobian matrix is a strong discriminating factor for networks of different structures and performance. Based on such observation, we propose a novel regularization method, which manages to improve the network performance comparably to dropout, which in turn verifies the observation.}
}