[PDF][PDF] Findings of the BabyLM Challenge: Sample-efficient pretraining on developmentally plausible corpora
Children can acquire language from less than 100 million words of input. Large language
models are far less data-efficient: they typically require 3 or 4 orders of magnitude more data …
models are far less data-efficient: they typically require 3 or 4 orders of magnitude more data …
Is Child-Directed Speech Effective Training Data for Language Models?
While high-performing language models are typically trained on hundreds of billions of
words, human children become fluent language users with a much smaller amount of data …
words, human children become fluent language users with a much smaller amount of data …
MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models
Pretraining data selection has the potential to improve language model pretraining efficiency
by utilizing higher-quality data from massive web data corpora. Current data selection …
by utilizing higher-quality data from massive web data corpora. Current data selection …
A surprisal oracle for when every layer counts
Active Curriculum Language Modeling (ACLM; Hong et al., 2023) is a learner directed
approach to training a language model. We proposed the original version of this process in …
approach to training a language model. We proposed the original version of this process in …
[PDF][PDF] Automatic Quality Estimation for Data Selection and Curriculum Learning
H Nguyen, L Yip, J DeBenedetto - csc.villanova.edu
The size of neural models within natural language processing has increased at a rapid pace
in recent years. With this increase in model size comes an increase in the amount of training …
in recent years. With this increase in model size comes an increase in the amount of training …