Foundation model development attracts a rapidly expanding body of contributors, scientists,
and applications. To help shape responsible development practices, we introduce the …

Previous language model pre-training methods have uniformly applied a next-token
prediction loss to all training tokens. Challenging this norm, we posit that" Not all tokens in a …

Understanding how language model performance varies with scale is critical to benchmark
and algorithm development. Scaling laws are one approach to building this understanding …

Scaling laws have driven remarkable progress across machine learning domains like
language modeling and computer vision. However, the exploration of scaling laws in …

Kaplan et al. and Hoffmann et al. developed influential scaling laws for the optimal model
size as a function of the compute budget, but these laws yield substantially different …

Despite advancements in Natural Language Processing (NLP) and the growing availability
of pretrained models, the English language remains the primary focus of model …

Research on scaling large language models (LLMs) has primarily focused on model
parameters and training data size, overlooking the role of vocabulary size.% Intuitively …

Increasing the size of a Transformer model does not always lead to enhanced performance.
This phenomenon cannot be explained by the empirical scaling laws. Furthermore …

What makes a good Large Language Model (LLM)? That it performs well on the relevant
benchmarks--which hopefully measure, with some validity, the presence of capabilities that …

In this work, we consider whether pretraining on a pruned high-quality subset of a large-
scale text dataset can improve LLM performance. While existing work has shown that …