On the implicit bias in deep-learning algorithms
G Vardi - Communications of the ACM, 2023 - dl.acm.org
On the Implicit Bias in Deep-Learning Algorithms Page 1 DEEP LEARNING HAS been highly
successful in recent years and has led to dramatic improvements in multiple domains …
successful in recent years and has led to dramatic improvements in multiple domains …
Provable guarantees for neural networks via gradient feature learning
Neural networks have achieved remarkable empirical performance, while the current
theoretical analysis is not adequate for understanding their success, eg, the Neural Tangent …
theoretical analysis is not adequate for understanding their success, eg, the Neural Tangent …
Transformers learn through gradual rank increase
We identify incremental learning dynamics in transformers, where the difference between
trained and initial weights progressively increases in rank. We rigorously prove this occurs …
trained and initial weights progressively increases in rank. We rigorously prove this occurs …
Implicit bias of gradient descent for two-layer reLU and leaky reLU networks on nearly-orthogonal data
The implicit bias towards solutions with favorable properties is believed to be a key reason
why neural networks trained by gradient-based optimization can generalize well. While the …
why neural networks trained by gradient-based optimization can generalize well. While the …
Benign overfitting in linear classifiers and leaky relu networks from kkt conditions for margin maximization
Linear classifiers and leaky ReLU networks trained by gradient flow on the logistic loss have
an implicit bias towards solutions which satisfy the Karush–Kuhn–Tucker (KKT) conditions …
an implicit bias towards solutions which satisfy the Karush–Kuhn–Tucker (KKT) conditions …
The double-edged sword of implicit bias: Generalization vs. robustness in relu networks
In this work, we study the implications of the implicit bias of gradient flow on generalization
and adversarial robustness in ReLU networks. We focus on a setting where the data …
and adversarial robustness in ReLU networks. We focus on a setting where the data …
Benign overfitting and grokking in relu networks for xor cluster data
Neural networks trained by gradient descent (GD) have exhibited a number of surprising
generalization behaviors. First, they can achieve a perfect fit to noisy training data and still …
generalization behaviors. First, they can achieve a perfect fit to noisy training data and still …
Fourier circuits in neural networks: Unlocking the potential of large language models in mathematical reasoning and modular arithmetic
In the evolving landscape of machine learning, a pivotal challenge lies in deciphering the
internal representations harnessed by neural networks and Transformers. Building on recent …
internal representations harnessed by neural networks and Transformers. Building on recent …
Random feature amplification: Feature learning and generalization in neural networks
In this work, we provide a characterization of the feature-learning process in two-layer ReLU
networks trained by gradient descent on the logistic loss following random initialization. We …
networks trained by gradient descent on the logistic loss following random initialization. We …
Neural Redshift: Random Networks are not Random Functions
Our understanding of the generalization capabilities of neural networks NNs is still
incomplete. Prevailing explanations are based on implicit biases of gradient descent GD but …
incomplete. Prevailing explanations are based on implicit biases of gradient descent GD but …