[HTML][HTML] Landscape and training regimes in deep learning
Deep learning algorithms are responsible for a technological revolution in a variety of tasks
including image recognition or Go playing. Yet, why they work is not understood. Ultimately …
including image recognition or Go playing. Yet, why they work is not understood. Ultimately …
Convex analysis of the mean field langevin dynamics
As an example of the nonlinear Fokker-Planck equation, the mean field Langevin dynamics
recently attracts attention due to its connection to (noisy) gradient descent on infinitely wide …
recently attracts attention due to its connection to (noisy) gradient descent on infinitely wide …
Towards a mathematical understanding of neural network-based machine learning: what we know and what we don't
The purpose of this article is to review the achievements made in the last few years towards
the understanding of the reasons behind the success and subtleties of neural network …
the understanding of the reasons behind the success and subtleties of neural network …
Mean-field Langevin dynamics: Time-space discretization, stochastic gradient, and variance reduction
The mean-field Langevin dynamics (MFLD) is a nonlinear generalization of the Langevin
dynamics that incorporates a distribution-dependent drift, and it naturally arises from the …
dynamics that incorporates a distribution-dependent drift, and it naturally arises from the …
Feature-learning networks are consistent across widths at realistic scales
We study the effect of width on the dynamics of feature-learning neural networks across a
variety of architectures and datasets. Early in training, wide neural networks trained on …
variety of architectures and datasets. Early in training, wide neural networks trained on …
Dynamics in deep classifiers trained with the square loss: Normalization, low rank, neural collapse, and generalization bounds
We overview several properties—old and new—of training overparameterized deep
networks under the square loss. We first consider a model of the dynamics of gradient flow …
networks under the square loss. We first consider a model of the dynamics of gradient flow …
Phase diagram for two-layer relu neural networks at infinite-width limit
How neural network behaves during the training over different choices of hyperparameters
is an important question in the study of neural networks. In this work, inspired by the phase …
is an important question in the study of neural networks. In this work, inspired by the phase …
Uniform-in-time propagation of chaos for the mean-field gradient Langevin dynamics
The mean-field Langevin dynamics is characterized by a stochastic differential equation that
arises from (noisy) gradient descent on an infinite-width two-layer neural network, which can …
arises from (noisy) gradient descent on an infinite-width two-layer neural network, which can …
Embedding principle of loss landscape of deep neural networks
Understanding the structure of loss landscape of deep neural networks (DNNs) is obviously
important. In this work, we prove an embedding principle that the loss landscape of a DNN" …
important. In this work, we prove an embedding principle that the loss landscape of a DNN" …
Neural collapse in deep homogeneous classifiers and the role of weight decay
A Rangamani… - ICASSP 2022-2022 IEEE …, 2022 - ieeexplore.ieee.org
Neural Collapse is a phenomenon recently discovered in deep classifiers where the last
layer activations collapse onto their class means, while the means and last layer weights …
layer activations collapse onto their class means, while the means and last layer weights …