Getting vit in shape: Scaling laws for compute-optimal model design
IM Alabdulmohsin, X Zhai… - Advances in Neural …, 2024 - proceedings.neurips.cc
Scaling laws have been recently employed to derive compute-optimal model size (number
of parameters) for a given compute duration. We advance and refine such methods to infer …
of parameters) for a given compute duration. We advance and refine such methods to infer …
Compute-efficient deep learning: Algorithmic trends and opportunities
BR Bartoldson, B Kailkhura, D Blalock - Journal of Machine Learning …, 2023 - jmlr.org
Although deep learning has made great progress in recent years, the exploding economic
and environmental costs of training neural networks are becoming unsustainable. To …
and environmental costs of training neural networks are becoming unsustainable. To …
Mimonets: Multiple-input-multiple-output neural networks exploiting computation in superposition
With the advent of deep learning, progressively larger neural networks have been designed
to solve complex tasks. We take advantage of these capacity-rich models to lower the cost of …
to solve complex tasks. We take advantage of these capacity-rich models to lower the cost of …
The Impact of Depth on Compositional Generalization in Transformer Language Models
To process novel sentences, language models (LMs) must generalize compositionally—
combine familiar elements in new ways. What aspects of a model's structure promote …
combine familiar elements in new ways. What aspects of a model's structure promote …
A comparative analysis of deep neural network architectures for sentence classification using genetic algorithm
Because of the number of different architectures, numerous settings of their hyper-
parameters and disparity among their sizes, it is difficult to equitably compare various deep …
parameters and disparity among their sizes, it is difficult to equitably compare various deep …
Evolutionary neural architecture search on transformers for RUL prediction
Remaining useful life (RUL) predictions are a key enabler for predictive maintenance. Data-
driven approaches, typically based on deep neural networks (DNNs), have shown success …
driven approaches, typically based on deep neural networks (DNNs), have shown success …
Simulating weighted automata over sequences and trees with transformers
Transformers are ubiquitous models in the natural language processing (NLP) community
and have shown impressive empirical successes in the past few years. However, little is …
and have shown impressive empirical successes in the past few years. However, little is …
Simulating Weighted Automata over Sequences and Trees with Transformers
Transformers are ubiquitous models in the natural language processing (NLP) community
and have shown impressive empirical successes in the past few years. However, little is …
and have shown impressive empirical successes in the past few years. However, little is …