Stochastic optimization lies at the heart of machine learning, and its cornerstone is
stochastic gradient descent (SGD), a method introduced over 60 years ago. The last eight …

We study stochastic gradient descent (SGD) and the stochastic heavy ball method (SHB,
otherwise known as the momentum method) for the general stochastic approximation …

Abstract Stochastic Gradient Descent (SGD) is being used routinely for optimizing non-
convex functions. Yet, the standard convergence theory for SGD in the smooth non-convex …

The success of adversarial formulations in machine learning has brought renewed
motivation for smooth games. In this work, we focus on the class of stochastic Hamiltonian …

Single-call stochastic extragradient methods, like stochastic past extragradient (SPEG) and
stochastic optimistic gradient (SOG), have gained a lot of interest in recent years and are …

We present a unified theorem for the convergence analysis of stochastic gradient algorithms
for minimizing a smooth and convex loss plus a convex regularizer. We do this by extending …

Modern large-scale finite-sum optimization relies on two key aspects: distribution and
stochastic updates. For smooth and strongly convex problems, existing decentralized …

Federated learning (FL) is a subfield of machine learning where multiple clients try to
collaboratively learn a model over a network under communication constraints. We consider …

We propose a generic variance-reduced algorithm, which we call MUltiple RANdomized
Algorithm (MURANA), for minimizing a sum of several smooth functions plus a regularizer, in …

Variance reduction (VR) methods for finite-sum minimization typically require the knowledge
of problem-dependent constants that are often unknown and difficult to estimate. To address …