Recently, the use of social networks such as Facebook, Twitter, and Sina Weibo has
become an inseparable part of our daily lives. It is considered as a convenient platform for …

Distributed machine learning (DML) techniques, such as federated learning, partitioned
learning, and distributed reinforcement learning, have been increasingly applied to wireless …

Synchronized stochastic gradient descent (SGD) optimizers with data parallelism are widely
used in training large-scale deep neural networks. Although using larger mini-batch sizes …

Initially developed for gaming and 3-D rendering, graphics processing units (GPUs) were
recognized to be a good fit to accelerate deep learning training. Its simple mathematical …

Data parallel training is widely used for scaling distributed deep neural network (DNN)
training. However, the performance benefits are often limited by the communication-heavy …

Distributed deep learning (DL) has become prevalent in recent years to reduce training time
by leveraging multiple computing devices (eg, GPUs/TPUs) due to larger models and …

Distributed synchronous stochastic gradient descent (S-SGD) with data parallelism has
been widely used in training large-scale deep neural networks (DNNs), but it typically …

Modern deep learning applications require increasingly more compute to train state-of-the-
art models. To address this demand, large corporations and institutions use dedicated High …

Cloud platforms are increasing their emphasis on sustainability and reducing their
operational carbon footprint. A common approach for reducing carbon emissions is to exploit …

Gradient sparsification is a promising technique to significantly reduce the communication
overhead in decentralized synchronous stochastic gradient descent (S-SGD) algorithms …