Optimization is a critical component in deep learning. We think optimization for neural
networks is an interesting topic for theoretical research due to various reasons. First, its …

When and why can a neural network be successfully trained? This article provides an
overview of optimization algorithms and theory for training neural networks. First, we discuss …

Adaptive gradient methods such as AdaGrad and its variants update the stepsize in
stochastic gradient descent on the fly according to the gradients received along the way; …

Recent works have shown that stochastic gradient descent (SGD) achieves the fast
convergence rates of full-batch gradient descent for over-parameterized models satisfying …

We propose a stochastic variant of the classical Polyak step-size (Polyak, 1987) commonly
used in the subgradient method. Although computing the Polyak step-size requires …

Multirobot systems have shown great potential in dealing with complicated tasks that are
impossible for a single robot to achieve. One essential problem encountered in …

This paper describes a novel algorithmic framework to minimize a finite-sum of functions
available over a network of nodes. The proposed framework, that we call GT-VR, is …

Backtracking linesearch is the de facto approach for minimizing continuously differentiable
functions with locally Lipschitz gradient. In recent years, it has been shown that in the convex …

Classical stochastic gradient methods for optimization rely on noisy gradient approximations
that become progressively less accurate as iterates approach a solution. The large noise …

We propose an adaptive variance-reduction method, called AdaSpider, for minimization of $
L $-smooth, non-convex functions with a finite-sum structure. In essence, AdaSpider …