Are emergent abilities of large language models a mirage?
R Schaeffer, B Miranda… - Advances in Neural …, 2024 - proceedings.neurips.cc
Recent work claims that large language models display\textit {emergent abilities}, abilities
not present in smaller-scale models that are present in larger-scale models. What makes …
not present in smaller-scale models that are present in larger-scale models. What makes …
Broken neural scaling laws
We present a smoothly broken power law functional form (that we refer to as a Broken
Neural Scaling Law (BNSL)) that accurately models & extrapolates the scaling behaviors of …
Neural Scaling Law (BNSL)) that accurately models & extrapolates the scaling behaviors of …
Revisiting the minimalist approach to offline reinforcement learning
Recent years have witnessed significant advancements in offline reinforcement learning
(RL), resulting in the development of numerous algorithms with varying degrees of …
(RL), resulting in the development of numerous algorithms with varying degrees of …
Improving multimodal interactive agents with reinforcement learning from human feedback
An important goal in artificial intelligence is to create agents that can both interact naturally
with humans and learn from their feedback. Here we demonstrate how to use reinforcement …
with humans and learn from their feedback. Here we demonstrate how to use reinforcement …
Uncovering neural scaling laws in molecular representation learning
Abstract Molecular Representation Learning (MRL) has emerged as a powerful tool for drug
and materials discovery in a variety of tasks such as virtual screening and inverse design …
and materials discovery in a variety of tasks such as virtual screening and inverse design …
Beyond scale: the diversity coefficient as a data quality metric demonstrates llms are pre-trained on formally diverse data
Current trends to pre-train capable Large Language Models (LLMs) mostly focus on scaling
of model and dataset size. However, the quality of pre-training data is an important factor for …
of model and dataset size. However, the quality of pre-training data is an important factor for …
Pretraining on the test set is all you need
R Schaeffer - arXiv preprint arXiv:2309.08632, 2023 - arxiv.org
Inspired by recent work demonstrating the promise of smaller Transformer-based language
models pretrained on carefully curated data, we supercharge such approaches by investing …
models pretrained on carefully curated data, we supercharge such approaches by investing …
Offline actor-critic reinforcement learning scales to large models
JT Springenberg, A Abdolmaleki, J Zhang… - arXiv preprint arXiv …, 2024 - arxiv.org
We show that offline actor-critic reinforcement learning can scale to large models-such as
transformers-and follows similar scaling laws as supervised learning. We find that offline …
transformers-and follows similar scaling laws as supervised learning. We find that offline …
Scaling laws for single-agent reinforcement learning
J Hilton, J Tang, J Schulman - arXiv preprint arXiv:2301.13442, 2023 - arxiv.org
Recent work has shown that, in generative modeling, cross-entropy loss improves smoothly
with model size and training compute, following a power law plus constant scaling law. One …
with model size and training compute, following a power law plus constant scaling law. One …
Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations
Maximum Manifold Capacity Representations (MMCR) is a recent multi-view self-supervised
learning (MVSSL) method that matches or surpasses other leading MVSSL methods. MMCR …
learning (MVSSL) method that matches or surpasses other leading MVSSL methods. MMCR …