Data mining (DM) is an instrument of pattern detection and retrieval of knowledge from a
large quantity of data. Many robust early detection services and other health-related …

Stochastic Gradient Descent with a constant learning rate (constant SGD) simulates a
Markov chain with a stationary distribution. With this perspective, we derive several new …

Natural gradient descent is an optimization method traditionally motivated from the
perspective of information geometry, and works well for many applications as an alternative …

Minimax optimal convergence rates for numerous classes of stochastic convex optimization
problems are well characterized, where the majority of results utilize iterate averaged …

We develop the mathematical foundations of the stochastic modified equations (SME)
framework for analyzing the dynamics of stochastic gradient algorithms, where the latter is …

Bridging the gap between constant step size stochastic gradient descent and Markov chains
Page 1 The Annals of Statistics 2020, Vol. 48, No. 3, 1348–1382 https://doi.org/10.1214/19-AOS1850 …

In this paper, we study the implicit regularization of stochastic gradient descent (SGD)
through the lens of dynamical stability (Wu et al., 2018). We start by revising existing stability …

This draft textbook is extracted from lecture notes from a class that I have taught
(unfortunately online, but this gave me an opportunity to write more detailed notes) during …

Many convex problems in machine learning and computer science share the same
form:\begin {align*}\min_ {x}\sum_ {i} f_i (A_i x+ b_i),\end {align*} where $ f_i $ are convex …

This work characterizes the benefits of averaging techniques widely used in conjunction with
stochastic gradient descent (SGD). In particular, this work presents a sharp analysis of:(1) …