Communication scheduling is crucial to accelerate the training of large deep learning
models, in which the transmission order of layer-wise deep neural network (DNN) tensors is …

Mixture of Expert (MoE) has received increasing attention for scaling DNN models to extra-
large size with negligible increases in computation. The MoE model has achieved the …

The communication bottleneck severely constrains the scalability of distributed deep
learning, and efficient communication scheduling accelerates distributed DNN training by …

In distributed deep learning with data parallelism, communication bottleneck throttles the
efficiency of model training. Recent studies adopt versatile gradient compression …

The communication bottleneck has severely restricted the scalability of distributed deep
learning. Tensor fusion improves the scalability of data parallelism by overlapping …

Large-scale datacenter networks are increasingly using in-network aggregation (INA) and
remote direct memory access (RDMA) techniques to accelerate deep neural network (DNN) …

Federated Learning enables collaboratively model training among a number of distributed
devices with the coordination of a centralized server, where each device alternatively …

Overlapping gradient communication with backward computation is a popular technique to
reduce communication cost in the widely adopted data parallel S-SGD training. However …

Deep Neural Networks (DNNs) find widespread applications across various domains,
including computer vision, recommendation systems, and natural language processing …

As deep learning (DL) algorithms evolve and data volumes expand, training deep neural
networks (DNNs) has become essential across various domains, delivering unprecedented …