Efficiently scaling transformer inference
We study the problem of efficient generative inference for Transformer models, in one of its
most challenging settings: large deep models, with tight latency targets and long sequence …
most challenging settings: large deep models, with tight latency targets and long sequence …
Coannotating: Uncertainty-guided work allocation between human and large language models for data annotation
Annotated data plays a critical role in Natural Language Processing (NLP) in training
models and evaluating their performance. Given recent developments in Large Language …
models and evaluating their performance. Given recent developments in Large Language …
Optimal clipping and magnitude-aware differentiation for improved quantization-aware training
Data clipping is crucial in reducing noise in quantization operations and improving the
achievable accuracy of quantization-aware training (QAT). Current practices rely on …
achievable accuracy of quantization-aware training (QAT). Current practices rely on …
Pokebnn: A binary pursuit of lightweight accuracy
Abstract Optimization of Top-1 ImageNet promotes enormous networks that may be
impractical in inference settings. Binary neural networks (BNNs) have the potential to …
impractical in inference settings. Binary neural networks (BNNs) have the potential to …
Understanding int4 quantization for language models: latency speedup, composability, and failure cases
Improving the deployment efficiency of transformer-based language models has been
challenging given their high computation and memory cost. While INT8 quantization has …
challenging given their high computation and memory cost. While INT8 quantization has …
Understanding int4 quantization for transformer models: Latency speedup, composability, and failure cases
Improving the deployment efficiency of transformer-based language models has been
challenging given their high computation and memory cost. While INT8 quantization has …
challenging given their high computation and memory cost. While INT8 quantization has …
4-bit conformer with native quantization aware training for speech recognition
Reducing the latency and model size has always been a significant research problem for
live Automatic Speech Recognition (ASR) application scenarios. Along this direction, model …
live Automatic Speech Recognition (ASR) application scenarios. Along this direction, model …
PikeLPN: Mitigating Overlooked Inefficiencies of Low-Precision Neural Networks
M Neseem, C McCullough, R Hsin… - Proceedings of the …, 2024 - openaccess.thecvf.com
Low-precision quantization is recognized for its efficacy in neural network optimization. Our
analysis reveals that non-quantized elementwise operations which are prevalent in layers …
analysis reveals that non-quantized elementwise operations which are prevalent in layers …
Distill or annotate? cost-efficient fine-tuning of compact models
Fine-tuning large models is highly effective, however, inference can be expensive and
produces carbon emissions. Knowledge distillation has been shown to be a practical …
produces carbon emissions. Knowledge distillation has been shown to be a practical …
2-bit conformer quantization for automatic speech recognition
Large speech models are rapidly gaining traction in research community. As a result, model
compression has become an important topic, so that these models can fit in memory and be …
compression has become an important topic, so that these models can fit in memory and be …