Modern distributed training of machine learning models often suffers from high
communication overhead for synchronizing stochastic gradients and model parameters. In …

The large communication overhead has imposed a bottleneck on the performance of
distributed Stochastic Gradient Descent (SGD) for training deep neural networks. Previous …

The present paper develops a novel aggregated gradient approach for distributed machine
learning that adaptively compresses the gradient communication. The key idea is to first …

The interest for distributed stochastic optimization has raised to train complex Machine
Learning models with more data on distributed systems. Increasing the computation power …

Synchronous stochastic gradient descent (SGD) is the most common method used for
distributed training of deep learning models. In this algorithm, each worker shares its local …

Recent years have witnessed a growing list of systems for distributed data-parallel training.
Existing systems largely fit into two paradigms, ie, parameter server and MPI-style collective …

Large-scale distributed optimization is of great importance in various applications. For data-
parallel based distributed learning, the inter-node gradient communication often becomes …

To reduce the long training time of large deep neural network (DNN) models, distributed
synchronous stochastic gradient descent (S-SGD) is commonly used on a cluster of workers …

Gradient sparsification and quantization offer a promising prospect to alleviate the
communication overhead problem in distributed learning. However direct combination of the …

Distributed stochastic gradient descent (SGD) algorithms are becoming popular in speeding
up deep learning model training by employing multiple computational devices (named …