Deep neural networks excel in supervised learning tasks but are constrained by the need for
extensive labeled data. Self-supervised learning emerges as a promising alternative …

The generalization and learning speed of a multi-class neural network can often be
significantly improved by using soft targets that are a weighted average of the hard targets …

Despite their great success, there is still no comprehensive theoretical understanding of
learning with Deep Neural Networks (DNNs) or their inner organization. Previous work …

We review the current literature concerned with information plane (IP) analyses of neural
network (NN) classifiers. While the underlying information bottleneck theory and the claim …

Numerous deep learning algorithms have been inspired by and understood via the notion of
information bottleneck, where unnecessary information is (often implicitly) minimized while …

This paper proposes an Information Bottleneck theory based filter pruning method that uses
a statistical measure called Mutual Information (MI). The MI between filters and class labels …

Whatever information a deep neural network has gleaned from training data is encoded in
its weights. How this information affects the response of the network to future data remains …

A novel functional estimator for Rényi's α-entropy and its multivariate extension was recently
proposed in terms of the normalized eigenspectrum of a Hermitian matrix of the projected …

Although deep neural networks have been immensely successful, there is no
comprehensive theoretical understanding of how they work or are structured. As a result …

We present a review on the recent advances and emerging opportunities around the theme
of analyzing deep neural networks (DNNs) with information-theoretic methods. We first …