Large deep learning models have shown great potential with state-of-the-art results in many tasks. However, running these large models is quite challenging on an accelerator (GPU or …
Transformer models have achieved state-of-the-art performance on various domains of applications and gradually becomes the foundations of the advanced large deep learning …
As deep learning models and input data continue to scale at an unprecedented rate, it has become inevitable to move towards distributed training platforms to fit the models and …
This paper argues that fair-sharing, the holy grail of congestion control algorithms for decades, is not necessarily a desirable property in Machine Learning (ML) training clusters …
Numerous microarchitectural optimizations unlocked tremendous processing power for deep neural networks that in turn fueled the AI revolution. With the exhaustion of such …
Distributed training is a solution to reduce DNN training time by splitting the task across multiple NPUs (eg, GPU/TPU). However, distributed training adds communication overhead …
As Deep Learning (DL) models grow larger and more complex, training jobs are increasingly distributed across multiple Computing Units (CU) such as GPUs and TPUs …
Graph Neural Networks (GCNs) have attracted wide attention and are applied to the real world. However, due to the ever-growing graph data with significant irregularities, off-chip …
T Yuan, Y Liu, X Ye, S Zhang, J Tan, B Chen… - 2024 USENIX Annual …, 2024 - usenix.org
Recent advancements in training large-scale models have centered on optimizing activation strategies and exploring various parallel training options. One research avenue focuses on …