How can we accelerate progress towards human-like linguistic generalization?
T Linzen - arXiv preprint arXiv:2005.00955, 2020 - arxiv.org
This position paper describes and critiques the Pretraining-Agnostic Identically Distributed
(PAID) evaluation paradigm, which has become a central tool for measuring progress in …
(PAID) evaluation paradigm, which has become a central tool for measuring progress in …
[PDF][PDF] Findings of the BabyLM Challenge: Sample-efficient pretraining on developmentally plausible corpora
Children can acquire language from less than 100 million words of input. Large language
models are far less data-efficient: they typically require 3 or 4 orders of magnitude more data …
models are far less data-efficient: they typically require 3 or 4 orders of magnitude more data …
Finding syntax in human encephalography with beam search
Recurrent neural network grammars (RNNGs) are generative models of (tree, string) pairs
that rely on neural networks to evaluate derivational choices. Parsing with them using beam …
that rely on neural networks to evaluate derivational choices. Parsing with them using beam …
So cloze yet so far: N400 amplitude is better predicted by distributional information than human predictability judgements
JA Michaelov, S Coulson… - IEEE Transactions on …, 2022 - ieeexplore.ieee.org
More predictable words are easier to process—they are read faster and elicit smaller neural
signals associated with processing difficulty, most notably, the N400 component of the event …
signals associated with processing difficulty, most notably, the N400 component of the event …
Multilingual language models predict human reading behavior
We analyze if large language models are able to predict patterns of human reading
behavior. We compare the performance of language-specific and multilingual pretrained …
behavior. We compare the performance of language-specific and multilingual pretrained …
Learning to ignore: Long document coreference with bounded memory neural networks
Long document coreference resolution remains a challenging task due to the large memory
and runtime requirements of current models. Recent work doing incremental coreference …
and runtime requirements of current models. Recent work doing incremental coreference …
How well does surprisal explain N400 amplitude under different experimental conditions?
JA Michaelov, BK Bergen - arXiv preprint arXiv:2010.04844, 2020 - arxiv.org
We investigate the extent to which word surprisal can be used to predict a neural measure of
human language processing difficulty-the N400. To do this, we use recurrent neural …
human language processing difficulty-the N400. To do this, we use recurrent neural …
Different kinds of cognitive plausibility: Why are transformers better than RNNs at predicting N400 amplitude?
JA Michaelov, MD Bardolph, S Coulson… - arXiv preprint arXiv …, 2021 - arxiv.org
Despite being designed for performance rather than cognitive plausibility, transformer
language models have been found to be better at predicting metrics used to assess human …
language models have been found to be better at predicting metrics used to assess human …
Incremental, predictive parsing with psycholinguistically motivated tree-adjoining grammar
Psycholinguistic research shows that key properties of the human sentence processor are
incrementality, connectedness (partial structures contain no unattached nodes), and …
incrementality, connectedness (partial structures contain no unattached nodes), and …
Towards incremental transformers: An empirical analysis of transformer models for incremental NLU
Incremental processing allows interactive systems to respond based on partial inputs, which
is a desirable property eg in dialogue agents. The currently popular Transformer architecture …
is a desirable property eg in dialogue agents. The currently popular Transformer architecture …