Teams that have trained large Transformer-based models have reported training instabilities
at large scale that did not appear when training with the same hyperparameters at smaller …

We consider the problem of estimating the learning rate in adaptive methods, such as
AdaGrad and Adam. We propose Prodigy, an algorithm that provably estimates the distance …

This paper proposes a new easy-to-implement parameter-free gradient-based optimizer:
DoWG (Distance over Weighted Gradients). We prove that DoWG is efficient---matching the …

We introduce a technique for tuning the learning rate scale factor of any base optimization
algorithm and schedule automatically, which we call Mechanic. Our method provides a …

In this paper, we explore two fundamental first-order algorithms in convex optimization,
namely, gradient descent (GD) and proximal gradient method (ProxGD). Our focus is on …

We prove impossibility results for adaptivity in non-smooth stochastic convex optimization.
Given a set of problem parameters we wish to adapt to, we define a" price of adaptivity"(PoA) …

Line search (or backtracking) procedures have been widely employed into first-order
methods for solving convex optimization problems, especially those with unknown problem …

Building upon recent works on linesearch-free adaptive proximal gradient methods, this
paper proposes AdaPG, a framework that unifies and extends existing results by providing …

In the past several years, the convergence of the last iterate of the Stochastic Gradient
Descent (SGD) algorithm has triggered people's interest due to its good performance in …

Adaptive optimization methods are widely recognized as among the most popular
approaches for training Deep Neural Networks (DNNs). Techniques such as Adam …