The surging scale of distributed training (DT) incurs significant communication overhead in
datacenters, while a promising solution is in-network aggregation (INA). It leverages …

Distributed computing has become a common practice nowadays, where recent focus has
been given to the usage of smart networking devices with in-network computing capabilities …

Communication is one of the key bottlenecks in the distributed training of large-scale
machine learning models, and lossy compression of exchanged information, such as …

Deep learning (DL) has become a key technique for solving complex problems in scientific
research and discovery. DL training for science is substantially challenging because it has to …

Over the last decade, we have witnessed growing data volumes generated and stored
across geographically distributed datacenters. Processing such geo-distributed datasets …

Iterative Synchronous Applications (ISApps) are popular in today's data centers, represented
by distributed deep learning (DL) training. In ISApps, multiple nodes carry out the computing …

With the growth of model/dataset/system size for distributed model training in datacenters,
the widely used Parameter Server (PS) architecture suffers from communication bottleneck …

Distributed machine learning is a mainstream system to learn insights for analytics and
intelligence services of many fronts (eg, health, streaming and business) from their massive …

Data parallelism and model parallelism are two typical parallel modes for distributed
machine learning (DML). Traditionally, DML mainly leverages data parallelism, which …

Abstract The bottleneck of Distributed Machine Learning (DML) has shifted from computation
to communication. Lots of works have focused on speeding up communication phase from …