Traditional analyses in non-convex optimization typically rely on the smoothness
assumption, namely requiring the gradients to be Lipschitz. However, recent evidence …

In this work, we describe a generic approach to show convergence with high probability for
both stochastic convex and non-convex optimization with sub-Gaussian noise. In previous …

Adaptive gradient methods are workhorses in deep learning. However, the convergence
guarantees of adaptive gradient methods for nonconvex optimization have not been …

Nonconvex-concave min-max problem arises in many machine learning applications
including minimizing a pointwise maximum of a set of nonconvex functions and robust …

We study convergence rates of AdaGrad-Norm as an exemplar of adaptive stochastic
gradient methods (SGD), where the step sizes change based on observed stochastic …

Very little is known about the training dynamics of adaptive gradient methods like Adam in
deep learning. In this paper, we shed light on the behavior of these algorithms in the full …

This draft textbook is extracted from lecture notes from a class that I have taught
(unfortunately online, but this gave me an opportunity to write more detailed notes) during …

Performance of recommender systems (RS) relies heavily on the amount of training data
available. This poses a chicken-and-egg problem for early-stage products, whose amount of …

We provide a simple convergence proof for AdaGrad optimizing non-convex objectives
under only affine noise variance and bounded smoothness assumptions. The proof is …

Currently, research on gesture recognition systems has been on the rise due to the
capabilities these systems provide to the field of human–machine interaction, however …