Large language models (LLMs) with hundreds of billions of parameters have sparked a new
wave of exciting AI applications. However, they are computationally expensive at inference …

Despite the great empirical success of deep reinforcement learning, its theoretical
foundation is less well understood. In this work, we make the first attempt to theoretically …

Benign overfitting, the phenomenon where interpolating models generalize well in the
presence of noisy data, was first observed in neural network models trained with gradient …

A recent line of research on deep learning focuses on the extremely over-parameterized
setting, and shows that when the network width is larger than a high degree polynomial of …

Deep residual networks (ResNets) have demonstrated better generalization performance
than deep feedforward networks (FFNets). However, the theory behind such a phenomenon …

The implicit biases of gradient-based optimization algorithms are conjectured to be a major
factor in the success of modern deep learning. In this work, we investigate the implicit bias of …

Residual neural networks are state-of-the-art deep learning models. Their continuous-depth
analog, neural ordinary differential equations (ODEs), are also widely used. Despite their …

Although the optimization objectives for learning neural networks are highly non-convex,
gradient-based methods have been wildly successful at learning neural networks in …

Finding parameters in a deep neural network (NN) that fit training data is a nonconvex
optimization problem, but a basic first-order optimization method (gradient descent) finds a …

Generalization analyses of deep learning typically assume that the training converges to a
fixed point. But, recent results indicate that in practice, the weights of deep neural networks …