We present a stochastic first-order optimization method specialized for deep neural networks
(DNNs), ECCO-DNN. This method models the optimization variable trajectory as a …

The vast majority of successful deep neural networks are trained using variants of stochastic
gradient descent (SGD) algorithms. Recent attempts to improve SGD can be broadly …

Proper optimization of deep neural networks is an open research question since an optimal
procedure to change the learning rate throughout training is still unknown. Manually defining …

First-order optimization methods are currently the mainstream in training deep neural
networks (DNNs). Optimizers like Adam incorporate limited curvature information by …

Adaptive optimization methods are widely recognized as among the most popular
approaches for training Deep Neural Networks (DNNs). Techniques such as Adam …

Popular first-order stochastic optimization methods for deep neural networks (DNNs) are
usually either accelerated schemes (eg stochastic gradient descent (SGD) with momentum) …

Deep neural networks (DNNs) are widely used and demonstrated their power in many
applications, such as computer vision and pattern recognition. However, the training of these …

Deep neural networks (DNNs) have well-documented merits in learning nonlinear functions
in high-dimensional spaces. Stochastic gradient descent (SGD)-type optimization algorithms …

Interpretation of Deep Neural Networks (DNNs) training as an optimal control problem with
nonlinear dynamical systems has received considerable attention recently, yet the …

Because the choice and tuning of the optimizer affects the speed, and ultimately the
performance of deep learning, there is significant past and recent research in this area. Yet …