Numerical validation is at the core of machine learning research as it allows us to assess the
actual impact of new methods, and to confirm the agreement between theory and practice …

Because the choice and tuning of the optimizer affects the speed, and ultimately the
performance of deep learning, there is significant past and recent research in this area. Yet …

Choosing the optimizer is considered to be among the most crucial design decisions in deep
learning, and it is not an easy one. The growing literature now lists hundreds of optimization …

Adaptive optimizers, such as Adam, have achieved remarkable success in deep learning. A
key component of these optimizers is the so-called preconditioning matrix, providing …

Efficient and automated design of optimizers plays a crucial role in full-stack AutoML
systems. However, prior methods in optimizer search are often limited by their scalability …

We present an open-source Python framework for NeuroEvolution Optimization with
Reinforcement Learning (NEORL) developed at the Massachusetts Institute of Technology …

In data-driven optimization, the sample performance of the obtained decision typically incurs
an optimistic bias against the true performance, a phenomenon commonly known as the …

Recently, flat-minima optimizers, which seek to find parameters in low-loss neighborhoods,
have been shown to improve a neural network's generalization performance over stochastic …

Abstract Recently, the Deep Learning community has become interested in evolutionary
optimization (EO) as a means to address hard optimization problems, eg meta-learning …

With advances in deep learning, exponential data growth and increasing model complexity,
developing efficient optimization methods are attracting much research attention. Several …