Multimodal large language models (MLLMs) enhance the capabilities of standard large
language models by integrating and processing data from multiple modalities, including text …

This is the interim publication of the first International Scientific Report on the Safety of
Advanced AI. The report synthesises the scientific understanding of general-purpose AI--AI …

Data curation is an essential component of large-scale pretraining. In this work, we
demonstrate that jointly selecting batches of data is more effective for learning than selecting …

Data plays a fundamental role in the training of Large Language Models (LLMs). Effective
data management, particularly in the formulation of a well-suited training dataset, holds …

Kaplan et al. and Hoffmann et al. developed influential scaling laws for the optimal model
size as a function of the compute budget, but these laws yield substantially different …

As training datasets become increasingly drawn from unstructured, uncontrolled
environments such as the web, researchers and industry practitioners have increasingly …

The rapid development of large language models (LLMs) has been witnessed in recent
years. Based on the powerful LLMs, multi-modal LLMs (MLLMs) extend the modality from …

Large Language Models have driven significant AI advancements, yet their training is
resource-intensive and highly sensitive to hyper-parameter selection. While scaling laws …

Scale has become a main ingredient in obtaining strong machine learning models. As a
result, understanding a model's scaling properties is key to effectively designing both the …

Knowledge distillation (KD) is the de facto standard for compressing large-scale models into
smaller ones. Prior works have explored ever more complex KD strategies involving different …