Transformers reign supreme in natural language processing, representing a milestone
innovation in deep learning. For high-performance model inference, optimizing the time …

Transformer has experienced tremendous success and revolutionized the field of natural
language processing (NLP). While GPU has become the de facto standard for deep learning …

The past several years have witnessed tremendous success of transformer models in
natural language processing (NLP), and their current landscape is increasingly diverse …

Multi-Head Attention (MHA) is a key component of Transformer. In MHA, attention heads
work independently, causing problems such as low-rank bottleneck of attention score …

Transformer neural networks (TNNs) are being applied across a widening range of
application domains, including natural language processing (NLP), machine translation, and …

Attention, as a core layer of the ubiquitous Transformer architecture, is the bottleneck for
large language models and long-context applications. FlashAttention elaborated an …

Transformer models have achieved remarkable success in various machine learning tasks
but suffer from high computational complexity and resource requirements. The quadratic …

Recently, the necessity of multiple attention heads in transformer architecture has been
questioned [1]. Removing less important heads from a large network is a promising strategy …

Models based on the attention mechanism, ie transformers, have shown extraordinary
performance in Natural Language Processing (NLP) tasks. However, their memory footprint …

Recently, Transformer has achieved state-of-the-arts results in several research areas such
as Natural Language Processing and Computer Vision. Due to Transformer has a very large …