This work presents a novel approach to distributed training of deep neural networks (DNNs)
that aims to overcome the issues related to mainstream approaches to data parallel training …

We consider distributed optimization under communication constraints for training deep
learning models. We propose a new algorithm, whose parameter updates rely on two forces …

Deep Learning (DL) has had an immense success in the recent past, leading to state-of-the-
art results in various domains, such as image recognition and natural language processing …

Abstract Machine learning, and in particular neural networks, require large amounts of data,
which is increasingly highly distributed (eg over user devices, or independent storage …

To cope with the ever growing availability of training data, there have been several
proposals to scale machine learning computation beyond a single server and distribute it …

Deep learning emerges as an important new resource-intensive workload and has been
successfully applied in computer vision, speech, natural language processing, and so on …

Big data has reignited research interest in machine learning. Massive quantities of data are
being generated regularly as a consequence of the development in the Internet, social …

Synchronous strategies with data parallelism are widely utilized in distributed training of
Deep Neural Networks (DNNs), largely owing to their easy implementation yet promising …

Distributed training of Deep Neural Networks (DNNs) on High-Performance Computing
(HPC) systems is becoming increasingly common. HPC systems dedicated entirely or …

We study distributed training of deep learning models in time-constrained environments. We
propose a new algorithm that periodically pulls workers towards the center variable …