在机器学习领域中, 梯度下降算法是求解最优化问题最重要, 最基础的方法. 随着数据规模的不断
扩大, 传统的梯度下降算法已不能有效地解决大规模机器学习问题. 随机梯度下降算法在迭代 …

Recent advances in Transformers have come with a huge requirement on computing
resources, highlighting the importance of developing efficient training techniques to make …

We study nonconvex optimization problems, where the objective function is either an
average of n nonconvex functions or the expectation of some stochastic function. We …

SARAH and SPIDER are two recently developed stochastic variance-reduced algorithms,
and SPIDER has been shown to achieve a near-optimal first-order oracle complexity in …

Tasks across diverse application domains can be posed as large-scale optimization
problems, these include graphics, vision, machine learning, imaging, health, scheduling …

We analyze stochastic gradient algorithms for optimizing nonconvex, nonsmooth finite-sum
problems. In particular, the objective function is given by the summation of a differentiable …

We study the complexity of finding the global solution to stochastic nonconvex optimization
when the objective function satisfies global Kurdyka-{\L} ojasiewicz (KL) inequality and the …

This paper studies accelerated gradient methods for nonconvex optimization with Lipschitz
continuous gradient and Hessian. We propose two simple accelerated gradient methods …

Numerous tasks at the core of statistics, learning and vision areas are specific cases of ill-
posed inverse problems. Recently, learning-based (eg, deep) iterative methods have been …

Stochastic gradient descent (SGD) has been found to be surprisingly effective in training a
variety of deep neural networks. However, there is still a lack of understanding on how and …