As deep learning models and input data continue to scale at an unprecedented rate, it has
become inevitable to move towards distributed training platforms to fit the models and …

The allreduce operation is the most commonly used collective operation in distributed or
parallel applications. It aggregates data collected from distributed hosts and broadcasts the …

Training is an important aspect of deep learning to enable network models to be deployed.
To scale training, multiple GPUs are commonly used with data parallelism to exploit the …

As large language models (LLMs) become widespread in various application domains, a
critical challenge the AI community is facing is how to train these large AI models in a cost …

Ahstract-RDMA over Converged Ethernet (RoCE) has gained significant attraction for
datacenter networks due to its compatibility with conventional Ethernet-based fabric …

Sparse matrix dense matrix multiplication (SpMM) is commonly used in applications ranging
from scientific computing to graph neural networks. Typically, when SpMM is executed in a …

Recent dual in-line memory modules (DIMMs) are starting to support processing-in-memory
(PIM) by associating their memory banks with processing elements (PEs), allowing …

Modern Deep Learning (DL) models have grown to sizes requiring massive clusters of
specialized, high-end nodes to train. Designing such clusters to maximize both performance …

With the widespread adoption of Deep Learning (DL) models, the demand for DL
accelerator hardware has risen. On top of that, DL models are becoming massive in size. To …

The surge of artificial intelligence, specifically large language models, has led to a rapid
advent towards the development of large-scale machine learning training clusters …