Modern language models rely on the transformer architecture and attention mechanism to perform language understanding and text generation. In this work, we study learning a 1 …
C Zheng, W Huang, R Wang, G Wu, J Zhu… - arXiv preprint arXiv …, 2024 - arxiv.org
Autoregressively trained transformers have brought a profound revolution to the world, especially with their in-context learning (ICL) ability to address downstream tasks. Recently …
M Li, X Zhang, Y Huang, S Oymak - arXiv preprint arXiv:2407.05591, 2024 - arxiv.org
The transformer architecture has catalyzed revolutionary advances in language modeling. However, recent architectural recipes, such as state-space models, have bridged the …
Large Language Models (LLMs) have the capacity to store and recall facts. Through experimentation with open-source models, we observe that this ability to retrieve facts can …
The self-attention mechanism prevails in modern machine learning. It has an interesting functionality of adaptively selecting tokens from an input sequence by modulating the …
Given a sequence of tokens, such as words, the task of next-token prediction is to predict the next-token conditional probability distribution. Decoder-only transformers have become …
G Liu, H Mao, J Tang, KM Johnson - arXiv preprint arXiv:2407.15286, 2024 - arxiv.org
Large Language Models (LLMs) are capable of producing content that perpetuates stereotypes, discrimination, and toxicity. The recently proposed moral self-correction is a …
C Gros - arXiv preprint arXiv:2407.18601, 2024 - arxiv.org
Attention involves comparing query and key vectors in terms of a scalar product, $\mathbf {Q}^ T\mathbf {K} $, together with a subsequent softmax normalization. Classicaly …
H Li, M Wang, S Lu, X Cui, PY Chen - High-dimensional Learning …, 2024 - openreview.net
Chain-of-Thought (CoT) is an efficient prompting method that enables the reasoning ability of large language models by augmenting the query using multiple examples with …