Despite extensive studies, the underlying reason as to why overparameterizedneural
networks can generalize remains elusive. Existing theory shows that common stochastic …

Deep neural networks can achieve remarkable generalization performances while
interpolating the training data. Rather than the U-curve emblematic of the bias-variance …

We give a simple proof for the global convergence of gradient descent in training deep
ReLU networks with the standard square loss, and show some of its improvements over the …

Modern machine learning methods are often overparametrized, allowing adaptation to the
data at a fine level. This can seem puzzling; in the worst case, such models do not need to …

A recent line of work studies overparametrized neural networks in the “kernel regime,” ie
when during training the network behaves as a kernelized linear predictor, and thus, training …

The classical bias-variance trade-off predicts that bias decreases and variance increase with
model complexity, leading to a U-shaped risk curve. Recent work calls this into question for …

The vanishing gradient problem was a major obstacle for the success of deep learning. In
recent years it was gradually alleviated through multiple different techniques. However the …

We develop a convex analytic approach to analyze finite width two-layer ReLU networks.
We first prove that an optimal solution to the regularized training problem can be …

Several machine learning problems can be naturally defined over graph data. Recently,
many researchers have been focusing on the definition of neural networks for graphs. The …

In this paper we propose to study generalization of neural networks on small algorithmically
generated datasets. In this setting, questions about data efficiency, memorization …