This paper introduces PowerInfer, a high-speed Large Language Model (LLM) inference
engine on a personal computer (PC) equipped with a single consumer-grade GPU. The key …

Large language models (LLMs) power a new generation of interactive AI applications
exemplified by ChatGPT. The interactive nature of these applications demands low latency …

Symbolic music generation aims to generate music scores automatically. A recent trend is to
use Transformer or its variants in music generation, which is, however, suboptimal, because …

Sparse tensors are rapidly becoming critical components of modern deep learning
workloads. However, developing high-performance sparse operators can be difficult and …

The increasing demand for improving deep learning model performance has led to a
paradigm shift in supporting low-precision computation to harness the robustness of deep …

With the fast growth of parameter size, it becomes increasingly challenging to deploy large
generative models as they typically require large GPU memory consumption and massive …

Dynamic neural networks (NNs), which can adapt sparsely activated sub-networks to inputs
during inference, have shown significant advantages over static ones in terms of accuracy …

Dynamic sparsity, where the sparsity patterns are unknown until runtime, poses a significant
challenge to deep learning. The state-of-the-art sparsity-aware deep learning solutions are …

Unstructured sparse neural networks are an important class of machine learning (ML)
models, as they compact model size and reduce floating point operations. The execution …

Real world arrays often contain underlying structure, such as sparsity, runs of repeated
values, or symmetry. Specializing for structure yields significant speedups. But automatically …