Transformers have achieved great success in many artificial intelligence fields, such as
natural language processing, computer vision, and audio processing. Therefore, it is natural …

A long time ago in the machine learning literature, the idea of incorporating a mechanism
inspired by the human visual system into neural networks was introduced. This idea is …

Position encoding has recently been shown to be effective in transformer architecture. It
enables valuable supervision for dependency modeling between elements at different …

We propose a conditional positional encoding (CPE) scheme for vision Transformers. Unlike
previous fixed or learnable positional encodings, which are pre-defined and independent of …

Recent transformer-based models, especially patch-based methods, have shown huge
potentiality in vision tasks. However, the split fixed-size patches divide the input features into …

Natural language processing (NLP) based on deep learning provides a positive
performance for generative dialogue system, and the transformer model is a new boost in …

In this work, we investigate the positional encoding methods used in language pre-training
(eg, BERT) and identify several problems in the existing formulations. First, we show that in …

Transformers have impressive generalization capabilities on tasks with a fixed context
length. However, they fail to generalize to sequences of arbitrary length, even for seemingly …

Transformers are arguably the main workhorse in recent natural language processing
research. By definition, a Transformer is invariant with respect to reordering of the input …

ABSTRACT Various Position Embeddings (PEs) have been proposed in Transformer based
architectures (eg BERT) to model word order. These are empirically-driven and perform well …