A survey of large language models
Language is essentially a complex, intricate system of human expressions governed by
grammatical rules. It poses a significant challenge to develop capable AI algorithms for …
grammatical rules. It poses a significant challenge to develop capable AI algorithms for …
Mamba: Linear-time sequence modeling with selective state spaces
Foundation models, now powering most of the exciting applications in deep learning, are
almost universally based on the Transformer architecture and its core attention module …
almost universally based on the Transformer architecture and its core attention module …
Rwkv: Reinventing rnns for the transformer era
Transformers have revolutionized almost all natural language processing (NLP) tasks but
suffer from memory and computational complexity that scales quadratically with sequence …
suffer from memory and computational complexity that scales quadratically with sequence …
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
M Reid, N Savinov, D Teplyashin, D Lepikhin… - arXiv preprint arXiv …, 2024 - arxiv.org
In this report, we present the latest model of the Gemini family, Gemini 1.5 Pro, a highly
compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning …
compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning …
Protein design with guided discrete diffusion
A popular approach to protein design is to combine a generative model with a discriminative
model for conditional sampling. The generative model samples plausible sequences while …
model for conditional sampling. The generative model samples plausible sequences while …
Hierarchically gated recurrent neural network for sequence modeling
Transformers have surpassed RNNs in popularity due to their superior abilities in parallel
training and long-term dependency modeling. Recently, there has been a renewed interest …
training and long-term dependency modeling. Recently, there has been a renewed interest …
A survey on vision mamba: Models, applications and challenges
Mamba, a recent selective structured state space model, performs excellently on long
sequence modeling tasks. Mamba mitigates the modeling constraints of convolutional …
sequence modeling tasks. Mamba mitigates the modeling constraints of convolutional …
Mamba-360: Survey of state space models as transformer alternative for long sequence modelling: Methods, applications, and challenges
BN Patro, VS Agneeswaran - arXiv preprint arXiv:2404.16112, 2024 - arxiv.org
Sequence modeling is a crucial area across various domains, including Natural Language
Processing (NLP), speech recognition, time series forecasting, music generation, and …
Processing (NLP), speech recognition, time series forecasting, music generation, and …
Laughing hyena distillery: Extracting compact recurrences from convolutions
Recent advances in attention-free sequence models rely on convolutions as alternatives to
the attention operator at the core of Transformers. In particular, long convolution sequence …
the attention operator at the core of Transformers. In particular, long convolution sequence …
Exposing attention glitches with flip-flop language modeling
Why do large language models sometimes output factual inaccuracies and exhibit
erroneous reasoning? The brittleness of these models, particularly when executing long …
erroneous reasoning? The brittleness of these models, particularly when executing long …