This paper explores the environmental impact of the super-linear growth trends for AI from a
holistic perspective, spanning Data, Algorithms, and System Hardware. We characterize the …

There is great interest in how the growth of artificial intelligence and machine learning may
affect global GHG emissions. However, such emissions impacts remain uncertain, owing in …

Transformers are slow and memory-hungry on long sequences, since the time and memory
complexity of self-attention are quadratic in sequence length. Approximate attention …

AI is undergoing a paradigm shift with the rise of models (eg, BERT, DALL-E, GPT-3) that are
trained on broad data at scale and are adaptable to a wide range of downstream tasks. We …

The growing energy and performance costs of deep learning have driven the community to
reduce the size of neural networks by selectively pruning components. Similarly to their …

Large language models have led to state-of-the-art accuracies across several tasks.
However, training these models efficiently is challenging because: a) GPU memory capacity …

Embedding based models have been the state of the art in collaborative filtering for over a
decade. Traditionally, the dot product or higher order equivalents have been used to …

Abstract Machine learning research has long focused on models rather than datasets, and
prominent datasets are used for common ML tasks without regard to the breadth, difficulty …

We present FedScale, a federated learning (FL) benchmarking suite with realistic datasets
and a scalable runtime to enable reproducible FL research. FedScale datasets encompass …

NVIDIA A100 Tensor Core GPU is NVIDIA's latest flagship GPU. It has been designed with
many new innovative features to provide performance and capabilities for HPC, AI, and data …