A kernel-based view of language model fine-tuning
It has become standard to solve NLP tasks by fine-tuning pre-trained language models
(LMs), especially in low-data settings. There is minimal theoretical understanding of …
(LMs), especially in low-data settings. There is minimal theoretical understanding of …
Construction of hierarchical neural architecture search spaces based on context-free grammars
The discovery of neural architectures from simple building blocks is a long-standing goal of
Neural Architecture Search (NAS). Hierarchical search spaces are a promising step towards …
Neural Architecture Search (NAS). Hierarchical search spaces are a promising step towards …
Efficient non-parametric optimizer search for diverse tasks
Efficient and automated design of optimizers plays a crucial role in full-stack AutoML
systems. However, prior methods in optimizer search are often limited by their scalability …
systems. However, prior methods in optimizer search are often limited by their scalability …
Improving training stability for multitask ranking models in recommender systems
Recommender systems play an important role in many content platforms. While most
recommendation research is dedicated to designing better models to improve user …
recommendation research is dedicated to designing better models to improve user …
Towards discovering neural architectures from scratch
The discovery of neural architectures from scratch is the long-standing goal of Neural
Architecture Search (NAS). Searching over a wide spectrum of neural architectures can …
Architecture Search (NAS). Searching over a wide spectrum of neural architectures can …
Structured mutation inspired by evolutionary theory enriches population performance and diversity
Grammar-Guided Genetic Programming (GGGP) employs a variety of insights from
evolutionary theory to autonomously design solutions for a given task. Recent insights from …
evolutionary theory to autonomously design solutions for a given task. Recent insights from …
Biological insights on grammar-structured mutations improve fitness and diversity
Grammar-Guided Genetic Programming (GGGP) employs a variety of concepts from
evolutionary theory to autonomously design solutions for a given task. Recent insights from …
evolutionary theory to autonomously design solutions for a given task. Recent insights from …
Generalisable Agents for Neural Network Optimisation
Optimising deep neural networks is a challenging task due to complex training dynamics,
high computational requirements, and long training times. To address this difficulty, we …
high computational requirements, and long training times. To address this difficulty, we …
Neural Optimizer Equation, Decay Function, and Learning Rate Schedule Joint Evolution
B Morgan, D Hougen - arXiv preprint arXiv:2404.06679, 2024 - arxiv.org
A major contributor to the quality of a deep learning model is the selection of the optimizer.
We propose a new dual-joint search space in the realm of neural optimizer search (NOS) …
We propose a new dual-joint search space in the realm of neural optimizer search (NOS) …
Generalisable Agents for Neural Network Optimisation
Optimising deep neural networks is a challenging task due to complex training dynamics,
high computational requirements, and long training times. To address this difficulty, we …
high computational requirements, and long training times. To address this difficulty, we …