Large language models (LLMs) have come closest among all models to date to mastering
human language, yet opinions about their linguistic and cognitive capabilities remain split …

Transformer language models have received widespread public attention, yet their
generated text is often surprising even to NLP researchers. In this survey, we discuss over …

Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved
great success and become a milestone in the field of artificial intelligence (AI). Owing to …

Transformer-based models have pushed state of the art in many areas of NLP, but our
understanding of what is behind their success is still limited. This paper is the first survey of …

Natural language is characterized by compositionality: the meaning of a complex expression
is constructed from the meanings of its constituent parts. To facilitate the evaluation of the …

Language understanding research is held back by a failure to relate language to the
physical world it describes and to the social interactions it facilitates. Despite the incredible …

Rapid progress in machine learning for natural language processing has the potential to
transform debates about how humans learn language. However, the learning environments …

While state-of-the-art neural network models continue to achieve lower perplexity scores on
language modeling benchmarks, it remains unknown whether optimizing for broad …

Modern deep neural networks achieve impressive performance in engineering applications
that require extensive linguistic skills, such as machine translation. This success has …

This position paper describes and critiques the Pretraining-Agnostic Identically Distributed
(PAID) evaluation paradigm, which has become a central tool for measuring progress in …