Representations and generalization in artificial and brain neural networks
Humans and animals excel at generalizing from limited data, a capability yet to be fully
replicated in artificial intelligence. This perspective investigates generalization in biological …
replicated in artificial intelligence. This perspective investigates generalization in biological …
Learning single-index models with shallow neural networks
Single-index models are a class of functions given by an unknown univariate``link''function
applied to an unknown one-dimensional projection of the input. These models are …
applied to an unknown one-dimensional projection of the input. These models are …
Feature learning via mean-field langevin dynamics: classifying sparse parities and beyond
Neural network in the mean-field regime is known to be capable of\textit {feature learning},
unlike the kernel (NTK) counterpart. Recent works have shown that mean-field neural …
unlike the kernel (NTK) counterpart. Recent works have shown that mean-field neural …
How two-layer neural networks learn, one (giant) step at a time
We investigate theoretically how the features of a two-layer neural network adapt to the
structure of the target function through a few large batch gradient descent steps, leading to …
structure of the target function through a few large batch gradient descent steps, leading to …
On learning gaussian multi-index models with gradient flow
We study gradient flow on the multi-index regression problem for high-dimensional
Gaussian data. Multi-index functions consist of a composition of an unknown low-rank linear …
Gaussian data. Multi-index functions consist of a composition of an unknown low-rank linear …
From high-dimensional & mean-field dynamics to dimensionless odes: A unifying approach to sgd in two-layers networks
This manuscript investigates the one-pass stochastic gradient descent (SGD) dynamics of a
two-layer neural network trained on Gaussian data and labels generated by a similar …
two-layer neural network trained on Gaussian data and labels generated by a similar …
On the different regimes of stochastic gradient descent
A Sclocchi, M Wyart - … of the National Academy of Sciences, 2024 - National Acad Sciences
Modern deep networks are trained with stochastic gradient descent (SGD) whose key
hyperparameters are the number of data considered at each step or batch size B, and the …
hyperparameters are the number of data considered at each step or batch size B, and the …
Rigorous dynamical mean-field theory for stochastic gradient descent methods
We prove closed-form equations for the exact high-dimensional asymptotics of a family of
first-order gradient-based methods, learning an estimator (eg, M-estimator, shallow neural …
first-order gradient-based methods, learning an estimator (eg, M-estimator, shallow neural …
Bottleneck structure in learned features: Low-dimension vs regularity tradeoff
A Jacot - Advances in Neural Information Processing …, 2023 - proceedings.neurips.cc
Previous work has shown that DNNs withlarge depth $ L $ and $ L_ {2} $-regularization are
biased towards learninglow-dimensional representations of the inputs, which can be …
biased towards learninglow-dimensional representations of the inputs, which can be …
Should Under-parameterized Student Networks Copy or Average Teacher Weights?
B Simsek, A Bendjeddou… - Advances in Neural …, 2024 - proceedings.neurips.cc
Abstract Any continuous function $ f^* $ can be approximated arbitrarily well by a neural
network with sufficiently many neurons $ k $. We consider the case when $ f^* $ itself is a …
network with sufficiently many neurons $ k $. We consider the case when $ f^* $ itself is a …