Cloze distillation: Improving neural language models with human next-word prediction

T Eisape, N Zaslavsky, R Levy - 2020 - dspace.mit.edu
Contemporary autoregressive language models (LMs) trained purely on corpus data have
been shown to capture numerous features of human incremental processing. However, past …

A chimpanzee by any other name: The contributions of utterance context and information density on word choice

CL Jacobs, MC MacDonald - Cognition, 2023 - Elsevier
An important feature of language production is the flexibility of lexical selection; producers
could refer to an animal as chimpanzee, chimp, ape, she, and so on. Thus, a key question …

Humans and language models diverge when predicting repeating text

AR Vaidya, J Turek, AG Huth - arXiv preprint arXiv:2310.06408, 2023 - arxiv.org
Language models that are trained on the next-word prediction task have been shown to
accurately model human behavior in word prediction and reading speed. In contrast with …

Memory in humans and deep language models: Linking hypotheses for model augmentation

O Raccah, P Chen, TL Willke, D Poeppel… - arXiv preprint arXiv …, 2022 - arxiv.org
The computational complexity of the self-attention mechanism in Transformer models
significantly limits their ability to generalize over long temporal durations. Memory …

The Effect of Surprisal on Reading Times in Information Seeking and Repeated Reading

KG Klein, Y Meiri, O Shubi, Y Berzak - arXiv preprint arXiv:2410.08162, 2024 - arxiv.org
The effect of surprisal on processing difficulty has been a central topic of investigation in
psycholinguistics. Here, we use eyetracking data to examine three language processing …

Humans diverge from language models when predicting spoken language

TL Botch, ES Finn - ICLR 2024 Workshop on Representational Alignment - openreview.net
Humans communicate through both spoken and written language, often switching between
these modalities depending on their goals. The recent success of large language models …