We introduce Performers, Transformer architectures which can estimate regular (softmax)
full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to …

Transformer models have achieved state-of-the-art results across a diverse range of
domains. However, concern over the cost of training the attention mechanism to learn …

Due to the over-parameterization nature, neural networks are a powerful tool for nonlinear
function approximation. In order to achieve good generalization on unseen data, a suitable …

Despite the recent popularity of deep generative state space models, few comparisons have
been made between network architectures and the inference steps of the Bayesian filtering …

We introduce in this paper the mechanism of graph random features (GRFs). GRFs can be
used to construct unbiased randomized estimators of several important kernels defined on …

Transformer networks are able to capture patterns in data coming from many domains (text,
images, videos, proteins, etc.) with little or no change to architecture components. We …

The problem of efficient approximation of a linear operator induced by the Gaussian or
softmax kernel is often addressed using random features (RFs) which yield an unbiased …

State construction is important for learning in partially observable environments. A general
purpose strategy for state construction is to learn the state update using a Recurrent Neural …

Transformer networks are able to capture patterns in data coming from many domains (text,
images, videos, proteins, etc.) with little or no change to architecture components. We …

Monte Carlo sampling in high-dimensional, low-sample settings is important in many
machine learning tasks. We improve current methods for sampling in Euclidean spaces by …