Overview frequency principle/spectral bias in deep learning
Understanding deep learning is increasingly emergent as it penetrates more and more into
industry and science. In recent years, a research line from Fourier analysis sheds light on …
industry and science. In recent years, a research line from Fourier analysis sheds light on …
Understanding multi-phase optimization dynamics and rich nonlinear behaviors of relu networks
The training process of ReLU neural networks often exhibits complicated nonlinear
phenomena. The nonlinearity of models and non-convexity of loss pose significant …
phenomena. The nonlinearity of models and non-convexity of loss pose significant …
Empirical phase diagram for three-layer neural networks with infinite width
Substantial work indicates that the dynamics of neural networks (NNs) is closely related to
their initialization of parameters. Inspired by the phase diagram for two-layer ReLU NNs with …
their initialization of parameters. Inspired by the phase diagram for two-layer ReLU NNs with …
Embedding principle of loss landscape of deep neural networks
Understanding the structure of loss landscape of deep neural networks (DNNs) is obviously
important. In this work, we prove an embedding principle that the loss landscape of a DNN" …
important. In this work, we prove an embedding principle that the loss landscape of a DNN" …
Mathematical introduction to deep learning: methods, implementations, and theory
This book aims to provide an introduction to the topic of deep learning algorithms. We review
essential components of deep learning algorithms in full mathematical detail including …
essential components of deep learning algorithms in full mathematical detail including …
Implicit regularization of dropout
It is important to understand how dropout, a popular regularization method, aids in achieving
a good generalization solution during neural network training. In this work, we present a …
a good generalization solution during neural network training. In this work, we present a …
On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks
In this article we study fully-connected feedforward deep ReLU ANNs with an arbitrarily
large number of hidden layers and we prove convergence of the risk of the GD optimization …
large number of hidden layers and we prove convergence of the risk of the GD optimization …
Towards understanding the condensation of neural networks at initial training
Empirical works show that for ReLU neural networks (NNs) with small initialization, input
weights of hidden neurons (the input weight of a hidden neuron consists of the weight from …
weights of hidden neurons (the input weight of a hidden neuron consists of the weight from …
Loss spike in training neural networks
In this work, we investigate the mechanism underlying loss spikes observed during neural
network training. When the training enters a region with a lower-loss-as-sharper (LLAS) …
network training. When the training enters a region with a lower-loss-as-sharper (LLAS) …
Convergence to good non-optimal critical points in the training of neural networks: Gradient descent optimization with one random initialization overcomes all bad non …
Gradient descent (GD) methods for the training of artificial neural networks (ANNs) belong
nowadays to the most heavily employed computational schemes in the digital world. Despite …
nowadays to the most heavily employed computational schemes in the digital world. Despite …