Powerful computer clusters are used nowadays to train complex deep neural networks
(DNN) on large datasets. Distributed training increasingly becomes communication bound …

Powerful computer clusters are used nowadays to train complex deep neural networks
(DNN) on large datasets. Distributed training workloads increasingly become …

Currently, progressively larger deep neural networks are trained on ever growing data
corpora. In result, distributed training schemes are becoming increasingly relevant. A major …

Large-scale distributed training of Deep Neural Networks (DNNs) on state-of-the-art
platforms are expected to be severely communication constrained. To overcome this …

Due to the substantial computational cost, training state-of-the-art deep neural networks for
large-scale datasets often requires distributed training using multiple computation workers …

Distributed training performs data-parallel training of DNN models which is a necessity for
increasingly complex models and large datasets. Recent works are identifying major …

Gradient compression (eg, gradient quantization and gradient sparsification) is a core
technique in reducing communication costs in distributed learning systems. The recent trend …

Lossy gradient compression has become a practical tool to overcome the communication
bottleneck in centrally coordinated distributed training of machine learning models …

Distributed deep learning (DL) has become prevalent in recent years to reduce training time
by leveraging multiple computing devices (eg, GPUs/TPUs) due to larger models and …

Training deep neural networks on large datasets containing high-dimensional data requires
a large amount of computation. A solution to this problem is data-parallel distributed training …