Training deep neural networks on large datasets can be accelerated by distributing the
computations across multiple worker nodes. The distributed stochastic gradient descent (D …

The large communication overhead has imposed a bottleneck on the performance of
distributed Stochastic Gradient Descent (SGD) for training deep neural networks. Previous …

To reduce the long training time of large deep neural network (DNN) models, distributed
synchronous stochastic gradient descent (S-SGD) is commonly used on a cluster of workers …

Large neural network models present a hefty communication challenge to distributed
Stochastic Gradient Descent (SGD), with a per-iteration communication complexity of O(n) …

Training large-scale machine learning models usually demands a distributed approach to
process the huge amount of training data efficiently. However, the high network …

This paper introduces CompSkipDSGD, a new algorithm for distributed stochastic gradient
descent that aims to improve communication efficiency by compressing and selectively …

Due to the explosion in the size of the training datasets, distributed learning has received
growing interest in recent years. One of the major bottlenecks is the large communication …

Synchronous stochastic gradient descent (SGD) is the most common method used for
distributed training of deep learning models. In this algorithm, each worker shares its local …

The communication bottleneck has been a critical problem in large-scale distributed deep
learning. In this work, we study distributed SGD with random block-wise sparsification as the …

Large-batch stochastic gradient descent (SGD) is widely used for training in distributed deep
learning because of its training-time efficiency, however, extremely large-batch SGD leads to …