Transformers have emerged as the architecture of choice for many state-of-the-art AI
models, showcasing exceptional performance across a wide range of AI applications …

While petabytes of data are generated each day by a number of independent computing
devices, only a few of them can be finally collected and used for deep learning (DL) due to …

Processing-In-Memory (PIM) architectures based on recent technology advances (eg,
Hybrid Memory Cube) demonstrate great potential for graph processing. However, existing …

Large deep learning models have shown great potential with state-of-the-art results in many
tasks. However, running these large models is quite challenging on an accelerator (GPU or …

This paper demonstrates the one-sided communication used in languages like UPC can
provide a significant performance advantage for bandwidth-limited applications. This is …

The design and implementation of a high performance communication network are critical
factors in determining the performance and cost-effectiveness of a largescale computing …

Large deep learning models have demonstrated strong ability to solve many tasks across a
wide range of applications. Those large models typically require training and inference to be …

Given the popularity of generative AI, Large Language Models (LLMs) often consume
hundreds or thousands of GPUs for parallelizing and accelerating the training process …

Message passing is a very popular style of parallel programming, used in a wide variety of
applications and supported by many APIs, such as BSD sockets, MPI and PVM. Its …

In the Fully Sharded Data Parallel (FSDP) training pipeline, collective operations can be
interleaved to maximize the communication/computation overlap. In this scenario …