The largest eigenvalue of the Hessian, or sharpness, of neural networks is a key quantity to
understand their optimization dynamics. In this paper, we study the sharpness of deep linear …

We analyze speed of convergence to global optimum for gradient descent training a deep
linear neural network (parameterized as $ x\mapsto W_N W_ {N-1}\cdots W_1 x $) by …

We analyze the global convergence of gradient descent for deep linear residual networks by
proposing a new initialization: zero-asymmetric (ZAS) initialization. It is motivated by …

When optimizing over-parameterized models, such as deep neural networks, a large set of
parameters can achieve zero training error. In such cases, the choice of the optimization …

We provide a detailed asymptotic study of gradient flow trajectories and their implicit
optimization bias when minimizing the exponential loss over" diagonal linear networks". This …

While training error of most deep neural networks degrades as the depth of the network
increases, residual networks appear to be an exception. We show that the main reason for …

In this paper, we study the implicit regularization of the gradient descent algorithm in
homogeneous neural networks, including fully-connected and convolutional neural …

Neural networks trained via gradient descent with random initialization and without any
regularization enjoy good generalization performance in practice despite being highly …

While deep learning is successful in a number of applications, it is not yet well understood
theoretically. A theoretical characterization of deep learning should answer questions about …

Over the past few years, an extensively studied phenomenon in training deep networks is
the implicit bias of gradient descent towards parsimonious solutions. In this work, we …