We perform an analysis of the average generalization dynamics of large neural networks
trained using gradient descent. We study the practically-relevant “high-dimensional” regime …

Generalization is one of the most important problems in neural-network research. It is
influenced by several factors in the network design, such as network size, weight decay …

At the heart of machine learning lies the question of generalizability of learned rules over
previously unseen data. While over-parameterized models based on neural networks are …

Deep neural networks achieve stellar generalisation even when they have enough
parameters to easily fit all their training data. We study this phenomenon by analysing the …

A longstanding goal in deep learning research has been to precisely characterize training
and generalization. However, the often complex loss landscapes of neural networks have …

Why does training deep neural networks using stochastic gradient descent (SGD) result in a
generalization error that does not worsen with the number of parameters in the network? To …

The classical bias-variance trade-off predicts that bias decreases and variance increase with
model complexity, leading to a U-shaped risk curve. Recent work calls this into question for …

This paper investigates the generalization properties of two-layer neural networks in high-
dimensions, ie when the number of samples $ n $, features $ d $, and neurons $ h $ tend to …

For certain infinitely-wide neural networks, the neural tangent kernel (NTK) theory fully
characterizes generalization, but for the networks used in practice, the empirical NTK only …

A theoretical understanding of generalization remains an open problem for many machine
learning models, including deep networks where overparameterization leads to better …