Analysis of Deep Convolutional Neural Networks Using Tensor Kernels and Matrix-Based Entropy
Research output: Contribution to journal › Journal article › Research › peer-review
Documents
- Fulltext
Final published version, 913 KB, PDF document
Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently to gain insight into, among others, DNNs’ generalization ability. However, it is by no means obvious how to estimate the mutual information (MI) between each hidden layer and the input/desired output to construct the IP. For instance, hidden layers with many neurons require MI estimators with robustness toward the high dimensionality associated with such layers. MI estimators should also be able to handle convolutional layers while at the same time being computationally tractable to scale to large networks. Existing IP methods have not been able to study truly deep convolutional neural networks (CNNs). We propose an IP analysis using the new matrix-based Rényi’s entropy coupled with tensor kernels, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data. Our results shed new light on previous studies concerning small-scale DNNs using a completely new approach. We provide a comprehensive IP analysis of large-scale CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks.
Original language | English |
---|---|
Article number | 899 |
Journal | Entropy |
Volume | 25 |
Issue number | 6 |
Number of pages | 21 |
ISSN | 1099-4300 |
DOIs | |
Publication status | Published - 2023 |
Bibliographical note
Publisher Copyright:
© 2023 by the authors.
- deep learning, information plane, information theory, kernels methods
Research areas
ID: 360255530