With the growing model size of deep neural networks (DNN), deep learning training is
increasingly relying on handcrafted search spaces to find efficient parallelization execution …

Tensor Attention, a multi-view attention that is able to capture high-order correlations among
multiple modalities, can overcome the representational limitations of classical matrix …

The computational challenges of Large Language Model (LLM) inference remain a
significant barrier to their widespread deployment, especially as prompt lengths continue to …

We introduce, Q-Sparse, a simple yet effective approach to training sparsely-activated large
language models (LLMs). Q-Sparse enables full sparsity of activations in LLMs which can …

Transformers with linear attention (ie, linear transformers) and state-space models have
recently been suggested as a viable linear-time alternative to transformers with softmax …

The training and inference of large language models (LLMs) are together a costly process
that transports knowledge from raw data to meaningful computation. Inspired by the memory …

Empowering LLMs with the ability to utilize useful information from a long context is crucial
for many downstream applications. However, achieving long context lengths with the …