While recent language models have the ability to take long contexts as input, relatively little
is known about how well they use longer context. We analyze the performance of language …

Autoregressive transformers are spectacular models for short sequences but scale poorly to
long sequences such as high-resolution images, podcasts, code, or books. We proposed …

The advent of large language models (LLMs) has revolutionized natural language
processing, enabling the generation of coherent and contextually relevant human-like text …

Finetuning large language models (LLMs) on instructions leads to vast performance
improvements on natural language tasks. We apply instruction tuning using code …

Language models typically need to be trained or finetuned in order to acquire new
knowledge, which involves updating their weights. We instead envision language models …

Abstract We introduce the Block-Recurrent Transformer, which applies a transformer layer in
a recurrent fashion along a sequence, and has linear complexity with respect to sequence …

Retrieval-Augmented Generation (RAG) has recently emerged as a method to extend
beyond the pre-trained knowledge of Large Language Models by augmenting the original …

While Transformers have shown remarkable success in natural language processing, their
attention mechanism's large memory requirements have limited their ability to handle longer …

While Transformers have shown remarkable success in natural language processing, their
attention mechanism's large memory requirements have limited their ability to handle longer …

Why do large language models sometimes output factual inaccuracies and exhibit
erroneous reasoning? The brittleness of these models, particularly when executing long …