This paper introduces AdaSwarm, a novel gradient-free optimizer which has similar or even
better performance than the Adam optimizer adopted in neural networks. In order to support …

In this paper, we propose a general deep learning training framework XGrad which
introduces weight prediction into the popular gradient-based optimizers to boost their …

In recent years, deep learning has achieved remarkable success in various fields such as
image recognition, natural language processing, and speech recognition. The effectiveness …

Adaptive optimization algorithms, such as Adam and RMSprop, have shown better
optimization performance than stochastic gradient descent (SGD) in some scenarios …

The stochastic gradient descent (SGD) optimizers are generally used to train the
convolutional neural networks (CNNs). In recent years, several adaptive momentum based …

Adaptive first-order optimizers are fundamental tools in deep learning, although they may
suffer from poor generalization due to the nonuniform gradient scaling. In this work, we …

The role of optimizer in deep neural networks model impacts the accuracy of the model.
Deep learning comes under the umbrella of parametric approaches; however, it tries to relax …

Deep neural networks are traditionally trained using humandesigned stochastic optimization
algorithms, such as SGD and Adam. Recently, the approach of learning to optimize network …

Abstract Stochastic Gradient Decent (SGD) is a very popular basic optimizer applied in the
learning algorithms of deep neural networks. However, it has fixed-sized steps for every …

First-order gradient-based optimization algorithms have been of core practical importance in
the field of deep learning. In this paper, we propose a new weighting mechanism-based first …