Scaling laws for generative mixed-modal language models
Generative language models define distributions over sequences of tokens that can
represent essentially any combination of data modalities (eg, any permutation of image …
represent essentially any combination of data modalities (eg, any permutation of image …
Reduced precision floating-point optimization for Deep Neural Network On-Device Learning on microcontrollers
Abstract Enabling On-Device Learning (ODL) for Ultra-Low-Power Micro-Controller Units
(MCUs) is a key step for post-deployment adaptation and fine-tuning of Deep Neural …
(MCUs) is a key step for post-deployment adaptation and fine-tuning of Deep Neural …
Autosparse: Towards automated sparse training of deep neural networks
Sparse training is emerging as a promising avenue for reducing the computational cost of
training neural networks. Several recent studies have proposed pruning methods using …
training neural networks. Several recent studies have proposed pruning methods using …
Small reals representations for deep learning at the edge: A comparison
The pervasiveness of deep neural networks (DNNs) in edge devices enforces new
requirements on information representation. Low precision formats from 16 bits down to 1 or …
requirements on information representation. Low precision formats from 16 bits down to 1 or …
Language Adaptation on a Tight Academic Compute Budget: Tokenizer Swapping Works and Pure bfloat16 Is Enough
We investigate continued pretraining of LLMs for language adaptation on a tight academic
budget: a setting in which only a few GPUs can be used in parallel, for a heavily constrained …
budget: a setting in which only a few GPUs can be used in parallel, for a heavily constrained …
Adaptive loss scaling for mixed precision training
Mixed precision training (MPT) is becoming a practical technique to improve the speed and
energy efficiency of training deep neural networks by leveraging the fast hardware support …
energy efficiency of training deep neural networks by leveraging the fast hardware support …
The hidden power of pure 16-bit floating-point neural networks
Lowering the precision of neural networks from the prevalent 32-bit precision has long been
considered harmful to performance, despite the gain in space and time. Many works …
considered harmful to performance, despite the gain in space and time. Many works …
[图书][B] Number Systems for Deep Neural Network Architectures
In this introductory chapter, we provide an overview of the main topics covered in this book
and the motivations to write it. The importance of efficient number systems for Deep Neural …
and the motivations to write it. The importance of efficient number systems for Deep Neural …
MultiPosits: Universal Coding of
P Lindstrom - Conference on Next Generation Arithmetic, 2022 - Springer
Recently proposed real-number representations like Posits and Elias codes provide
attractive alternatives to IEEE floating point for representing real numbers in science and …
attractive alternatives to IEEE floating point for representing real numbers in science and …
On the challenges in programming mixed-precision deep neural networks
Deep Neural Networks (DNNs) are resilient to reduced data precision, which motivates
exploiting low-precision data formats for more efficient computation, especially on custom …
exploiting low-precision data formats for more efficient computation, especially on custom …