Gradient descent finds a global minimum in training deep neural networks despite the
objective function being non-convex. The current paper proves gradient descent achieves …

One of the mysteries in the success of neural networks is randomly initialized first order
methods like gradient descent can achieve zero training loss even though the objective …

Neural network training relies on our ability to find" good" minimizers of highly non-convex
loss functions. It is well known that certain network architecture designs (eg, skip …

We describe the new field of the mathematical analysis of deep learning. This field emerged
around a list of research questions that were not answered within the classical framework of …

Geometric deep learning is an umbrella term for emerging techniques attempting to
generalize (structured) deep neural models to non-Euclidean domains, such as graphs and …

“Memorization can be loosely defined as learning the frequent co-occurrence of items or
features and exploiting the correlation available in the historical data. Generalization, on the …

Conventional wisdom in deep learning states that increasing depth improves
expressiveness but complicates optimization. This paper suggests that, sometimes …

In recent years, stochastic gradient descent (SGD) based techniques has become the
standard tools for training neural networks. However, formal theoretical understanding of …

In this paper, we consider regression problems with one-hidden-layer neural networks
(1NNs). We distill some properties of activation functions that lead to local strong convexity …

We consider the optimization problem associated with training simple ReLU neural networks
of the form $\mathbf {x}\mapsto\sum_ {i= 1}^{k}\max\{0,\mathbf {w} _i^\top\mathbf {x}\} $ with …