Stochastic optimization lies at the heart of machine learning, and its cornerstone is
stochastic gradient descent (SGD), a method introduced over 60 years ago. The last eight …

Federated learning is a key scenario in modern large-scale machine learning where the
data remains distributed over a large number of clients and the task is to learn a centralized …

Variance reduction has emerged in recent years as a strong competitor to stochastic
gradient descent in non-convex problems, providing the first algorithms to improve upon the …

Nesterov's momentum trick is famously known for accelerating gradient descent, and has
been proven useful in building fast iterative algorithms. However, in the stochastic setting …

We analyze the stochastic average gradient (SAG) method for optimizing the sum of a finite
number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG …

We consider the problem of minimizing the sum of two convex functions: one is the average
of a large number of smooth component functions, and the other is a general convex …

We design a stochastic algorithm to find $\varepsilon $-approximate local minima of any
smooth nonconvex function in rate $ O (\varepsilon^{-3.25}) $, with only oracle access to …

An up-to-date account of the interplay between optimization and machine learning,
accessible to students and researchers in both communities. The interplay between …

First-order stochastic methods are the state-of-the-art in large-scale machine learning
optimization owing to efficient per-iteration complexity. Second-order methods, while able to …

We consider nonconvex-concave minimax optimization problems of the form $\min_ {\bf
x}\max_ {\bf y\in {\mathcal Y}} f ({\bf x},{\bf y}) $, where $ f $ is strongly-concave in $\bf y $ but …