Since the proposal of transformers, these models have been limited to bounded input
lengths, because of their need to attend to every token in the input. In this work, we propose …

We introduce sub-sentence encoder, a contrastively-learned contextual embedding model
for fine-grained semantic representation of text. In contrast to the standard practice with …

While significant advancements have been made in compressed representations for text
embeddings in large language models (LLMs), the compression of visual tokens in large …

Fusion-in-Decoder (FiD) is an effective retrieval-augmented language model applied across
a variety of open-domain tasks, such as question answering, fact checking, etc. In FiD …

Current language models are unable to quickly learn new concepts on the fly, often
requiring a more involved finetuning process to learn robustly. Prompting in-context is not …

While significant advancements have been made in compressed representations for text
embeddings in large language models (LLMs), the compression of visual tokens in multi …

Text embedding has become a foundational technology in natural language processing
(NLP) during the deep learning era, driving advancements across a wide array of …

Large language models (LLMs) have achieved significant performance gains using
advanced prompting techniques over various tasks. However, the increasing length of …

Standard Transformer-based language models (LMs) scale poorly to long contexts. We
propose a solution based on dynamic contextual compression, which extends the Nugget …

We introduce a new dynamic vocabulary for language models. It can involve arbitrary text
spans during generation. These text spans act as basic generation bricks, akin to tokens in …