Provable guarantees for neural networks via gradient feature learning
Neural networks have achieved remarkable empirical performance, while the current
theoretical analysis is not adequate for understanding their success, eg, the Neural Tangent …
theoretical analysis is not adequate for understanding their success, eg, the Neural Tangent …
The double-edged sword of implicit bias: Generalization vs. robustness in relu networks
In this work, we study the implications of the implicit bias of gradient flow on generalization
and adversarial robustness in ReLU networks. We focus on a setting where the data …
and adversarial robustness in ReLU networks. We focus on a setting where the data …
From tempered to benign overfitting in relu neural networks
G Kornowski, G Yehudai… - Advances in Neural …, 2024 - proceedings.neurips.cc
Overparameterized neural networks (NNs) are observed to generalize well even when
trained to perfectly fit noisy data. This phenomenon motivated a large body of work on" …
trained to perfectly fit noisy data. This phenomenon motivated a large body of work on" …
Benign overfitting and grokking in relu networks for xor cluster data
Neural networks trained by gradient descent (GD) have exhibited a number of surprising
generalization behaviors. First, they can achieve a perfect fit to noisy training data and still …
generalization behaviors. First, they can achieve a perfect fit to noisy training data and still …
Fourier circuits in neural networks: Unlocking the potential of large language models in mathematical reasoning and modular arithmetic
In the evolving landscape of machine learning, a pivotal challenge lies in deciphering the
internal representations harnessed by neural networks and Transformers. Building on recent …
internal representations harnessed by neural networks and Transformers. Building on recent …
Learning a neuron by a shallow relu network: Dynamics and implicit bias for correlated inputs
We prove that, for the fundamental regression task of learning a single neuron, training a
one-hidden layer ReLU network of any width by gradient flow from a small initialisation …
one-hidden layer ReLU network of any width by gradient flow from a small initialisation …
Feature emergence via margin maximization: case studies in algebraic tasks
Understanding the internal representations learned by neural networks is a cornerstone
challenge in the science of machine learning. While there have been significant recent …
challenge in the science of machine learning. While there have been significant recent …
Vanishing gradients in reinforcement finetuning of language models
Pretrained language models are commonly aligned with human preferences and
downstream tasks via reinforcement finetuning (RFT), which entails maximizing a (possibly …
downstream tasks via reinforcement finetuning (RFT), which entails maximizing a (possibly …
Precise asymptotic generalization for multiclass classification with overparameterized linear models
We study the asymptotic generalization of an overparameterized linear model for multiclass
classification under the Gaussian covariates bi-level model introduced in Subramanian et …
classification under the Gaussian covariates bi-level model introduced in Subramanian et …
Noisy interpolation learning with shallow univariate relu networks
We study the asymptotic overfitting behavior of interpolation with minimum norm ($\ell_2 $ of
the weights) two-layer ReLU networks for noisy univariate regression. We show that …
the weights) two-layer ReLU networks for noisy univariate regression. We show that …