Can a mere next-token predictor faithfully model human intelligence? We crystallize this
intuitive concern, which is fragmented in the literature. As a starting point, we argue that the …

Modern language models rely on the transformer architecture and attention mechanism to
perform language understanding and text generation. In this work, we study learning a 1 …

Autoregressively trained transformers have brought a profound revolution to the world,
especially with their in-context learning (ICL) ability to address downstream tasks. Recently …

The transformer architecture has catalyzed revolutionary advances in language modeling.
However, recent architectural recipes, such as state-space models, have bridged the …

Large Language Models (LLMs) have the capacity to store and recall facts. Through
experimentation with open-source models, we observe that this ability to retrieve facts can …

The self-attention mechanism prevails in modern machine learning. It has an interesting
functionality of adaptively selecting tokens from an input sequence by modulating the …

Given a sequence of tokens, such as words, the task of next-token prediction is to predict the
next-token conditional probability distribution. Decoder-only transformers have become …

Large Language Models (LLMs) are capable of producing content that perpetuates
stereotypes, discrimination, and toxicity. The recently proposed moral self-correction is a …

Attention involves comparing query and key vectors in terms of a scalar product, $\mathbf
{Q}^ T\mathbf {K} $, together with a subsequent softmax normalization. Classicaly …

Chain-of-Thought (CoT) is an efficient prompting method that enables the reasoning ability
of large language models by augmenting the query using multiple examples with …