Learning-rate-free learning by d-adaptation
A Defazio, K Mishchenko - International Conference on …, 2023 - proceedings.mlr.press
The speed of gradient descent for convex Lipschitz functions is highly dependent on the
choice of learning rate. Setting the learning rate to achieve the optimal convergence rate …
choice of learning rate. Setting the learning rate to achieve the optimal convergence rate …
Dog is sgd's best friend: A parameter-free dynamic step size schedule
We propose a tuning-free dynamic SGD step size formula, which we call Distance over
Gradients (DoG). The DoG step sizes depend on simple empirical quantities (distance from …
Gradients (DoG). The DoG step sizes depend on simple empirical quantities (distance from …
SGD with AdaGrad stepsizes: Full adaptivity with high probability to unknown parameters, unbounded gradients and affine variance
Abstract We study Stochastic Gradient Descent with AdaGrad stepsizes: a popular adaptive
(self-tuning) method for first-order stochastic optimization. Despite being well studied …
(self-tuning) method for first-order stochastic optimization. Despite being well studied …
Parameter-free regret in high probability with heavy tails
J Zhang, A Cutkosky - Advances in Neural Information …, 2022 - proceedings.neurips.cc
We present new algorithms for online convex optimization over unbounded domains that
obtain parameter-free regret in high-probability given access only to potentially heavy-tailed …
obtain parameter-free regret in high-probability given access only to potentially heavy-tailed …
Prodigy: An expeditiously adaptive parameter-free learner
K Mishchenko, A Defazio - arXiv preprint arXiv:2306.06101, 2023 - arxiv.org
We consider the problem of estimating the learning rate in adaptive methods, such as
AdaGrad and Adam. We propose Prodigy, an algorithm that provably estimates the distance …
AdaGrad and Adam. We propose Prodigy, an algorithm that provably estimates the distance …
Dowg unleashed: An efficient universal parameter-free gradient descent method
A Khaled, K Mishchenko, C Jin - Advances in Neural …, 2023 - proceedings.neurips.cc
This paper proposes a new easy-to-implement parameter-free gradient-based optimizer:
DoWG (Distance over Weighted Gradients). We prove that DoWG is efficient---matching the …
DoWG (Distance over Weighted Gradients). We prove that DoWG is efficient---matching the …
Mechanic: A learning rate tuner
We introduce a technique for tuning the learning rate scale factor of any base optimization
algorithm and schedule automatically, which we call Mechanic. Our method provides a …
algorithm and schedule automatically, which we call Mechanic. Our method provides a …
Nest your adaptive algorithm for parameter-agnostic nonconvex minimax optimization
Adaptive algorithms like AdaGrad and AMSGrad are successful in nonconvex optimization
owing to their parameter-agnostic ability–requiring no a priori knowledge about problem …
owing to their parameter-agnostic ability–requiring no a priori knowledge about problem …
The price of adaptivity in stochastic convex optimization
We prove impossibility results for adaptivity in non-smooth stochastic convex optimization.
Given a set of problem parameters we wish to adapt to, we define a" price of adaptivity"(PoA) …
Given a set of problem parameters we wish to adapt to, we define a" price of adaptivity"(PoA) …
A simple uniformly optimal method without line search for convex optimization
Line search (or backtracking) procedures have been widely employed into first-order
methods for solving convex optimization problems, especially those with unknown problem …
methods for solving convex optimization problems, especially those with unknown problem …