Understanding Adam optimizer via online learning of updates: Adam is FTRL in disguise
Despite the success of the Adam optimizer in practice, the theoretical understanding of its
algorithmic components still remains limited. In particular, most existing analyses of Adam …
algorithmic components still remains limited. In particular, most existing analyses of Adam …
General framework for online-to-nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimization
This work investigates the effectiveness of schedule-free methods, developed by A. Defazio
et al.(NeurIPS 2024), in nonconvex optimization settings, inspired by their remarkable …
et al.(NeurIPS 2024), in nonconvex optimization settings, inspired by their remarkable …
Adam with model exponential moving average is effective for nonconvex optimization
K Ahn, A Cutkosky - arXiv preprint arXiv:2405.18199, 2024 - arxiv.org
In this work, we offer a theoretical analysis of two modern optimization techniques for
training large and complex models:(i) adaptive optimization algorithms, such as Adam, and …
training large and complex models:(i) adaptive optimization algorithms, such as Adam, and …