Training large-scale language models is increasingly critical in various domains, but it is
hindered by frequent failures, leading to significant time and economic costs. Current failure …

Large language models (LLMs) represented by chartGPT have achieved profound
applications and breakthroughs in various fields. This demonstrates that LLMs with …

Extensive system scales (ie thousands of GPU/TPUs) and prolonged training periods (ie
months of pretraining) significantly escalate the probability of failures when training large …

LLMs have seen rapid adoption in all domains. They need to be trained on high-end high-
performance computing (HPC) infrastructures and ingest massive amounts of input data …

Training Large Language Models (LLMs) presents significant memory challenges,
predominantly due to the growing size of weights and optimizer states. Common memory …

Getting large language models (LLMs) to perform well on the downstream tasks requires pre-
training over trillions of tokens. This typically demands a large number of powerful …

We present the design, implementation and engineering experience in building and
deploying MegaScale, a production system for training large language models (LLMs) at the …

Despite the effectiveness of deep neural networks in numerous natural language processing
applications, recent findings have exposed the vulnerability of these language models when …

Mixture-of-Experts (MoE) enjoys performance gain by increasing model capacity while
keeping computation cost constant. When comparing MoE to dense models, prior work …

Large language models (LLMs) suffer from catastrophic forgetting during continual learning.
Conventional rehearsal-based methods rely on previous training data to retain the model's …