Large foundation models, including large language models (LLMs), vision transformers
(ViTs), diffusion, and LLM-based multimodal models, are revolutionizing the entire machine …

Deep neural networks (DNNs) are becoming progressively large and costly to train. This
paper aims to reduce DNN training costs by leveraging preemptible instances on modern …

Training large-scale language models is increasingly critical in various domains, but it is
hindered by frequent failures, leading to significant time and economic costs. Current failure …

Deep Learning training jobs process large amounts of training data using many GPU
devices, often running for weeks or months. When hardware or software failures happen …

Training large Deep Neural Network (DNN) models requires thousands of GPUs for days or
weeks at a time. At these scales, failures are frequent and can have a big impact on training …

Given a Large Language Model (LLM) generation, how can we identify which training data
led to this generation? In this paper, we proposed RapidIn, a scalable framework adapting to …

Deep learning (DL) jobs use multi-dimensional parallelism, ie they combine data, model,
and pipeline parallelism, to use large GPU clusters efficiently. This couples jobs tightly to a …

Checkpointing (C) and restoring (R) are key components for GPU tasks. POS is an OS-level
GPU C/R system: It can transparently checkpoint or restore processes that use the GPU …

Distributed Deep Learning (DDL), as a paradigm, dictates the use of GPU-based clusters as
the optimal infrastructure for training large-scale Deep Neural Networks (DNNs). However …

With the widespread adoption of deep learning (DL) applications in recent years, training DL
models has become increasingly prevalent. Nevertheless, training these models is typically …