Length generalization, the ability to generalize from small training context sizes to larger
ones, is a critical challenge in the development of Transformer-based language models …

Length generalization, defined as the ability to extrapolate from shorter training sequences
to longer test ones, is a significant challenge for language models. This issue persists even …

Transformer-based Large Language Models (LLMs) are pioneering advances in many
natural language processing tasks, however, their exceptional capabilities are restricted …

Without positional information, attention-based Transformer neural networks are permutation-
invariant. Absolute or relative positional embeddings are the most popular ways to feed …

Abstract Relative Positional Encoding (RPE), which encodes the relative distance between
any pair of tokens, is one of the most successful modifications to the original Transformer. As …

The design choices in the Transformer attention mechanism, including weak inductive bias
and quadratic computational complexity, have limited its application for modeling long …

Preventing the performance decay of Transformers on inputs longer than those used for
training has been an important challenge in extending the context length of these models …

Transformers have impressive generalization capabilities on tasks with a fixed context
length. However, they fail to generalize to sequences of arbitrary length, even for seemingly …

Transformers provide a class of expressive architectures that are extremely effective for
sequence modeling. However, the key limitation of transformers is their quadratic memory …

The attention module, which is a crucial component in Transformer, cannot scale efficiently
to long sequences due to its quadratic complexity. Many works focus on approximating the …