On the implicit bias in deep-learning algorithms
G Vardi - Communications of the ACM, 2023 - dl.acm.org
On the Implicit Bias in Deep-Learning Algorithms Page 1 DEEP LEARNING HAS been highly
successful in recent years and has led to dramatic improvements in multiple domains …
successful in recent years and has led to dramatic improvements in multiple domains …
Implicit regularization towards rank minimization in relu networks
We study the conjectured relationship between the implicit regularization in neural networks,
trained with gradient-based methods, and rank minimization of their weight matrices …
trained with gradient-based methods, and rank minimization of their weight matrices …
Implicit bias of large depth networks: a notion of rank for nonlinear functions
A Jacot - The Eleventh International Conference on Learning …, 2023 - openreview.net
We show that the representation cost of fully connected neural networks with homogeneous
nonlinearities-which describes the implicit bias in function space of networks with $ L_2 …
nonlinearities-which describes the implicit bias in function space of networks with $ L_2 …
Bottleneck structure in learned features: Low-dimension vs regularity tradeoff
A Jacot - Advances in Neural Information Processing …, 2023 - proceedings.neurips.cc
Previous work has shown that DNNs withlarge depth $ L $ and $ L_ {2} $-regularization are
biased towards learninglow-dimensional representations of the inputs, which can be …
biased towards learninglow-dimensional representations of the inputs, which can be …
[PDF][PDF] Smoothing the edges: a general framework for smooth optimization in sparse regularization using Hadamard overparametrization
This paper presents a framework for smooth optimization of objectives with ℓq and ℓp, q
regularization for (structured) sparsity. Finding solutions to these non-smooth and possibly …
regularization for (structured) sparsity. Finding solutions to these non-smooth and possibly …
Linear neural network layers promote learning single-and multiple-index models
This paper explores the implicit bias of overparameterized neural networks of depth greater
than two layers. Our framework considers a family of networks of varying depths that all have …
than two layers. Our framework considers a family of networks of varying depths that all have …
Feature Learning in -regularized DNNs: Attraction/Repulsion and Sparsity
We study the loss surface of DNNs with $ L_ {2} $ regularization. Weshow that the loss in
terms of the parameters can be reformulatedinto a loss in terms of the layerwise activations …
terms of the parameters can be reformulatedinto a loss in terms of the layerwise activations …
Path regularization: A convexity and sparsity inducing regularization for parallel relu networks
Understanding the fundamental principles behind the success of deep neural networks is
one of the most important open questions in the current literature. To this end, we study the …
one of the most important open questions in the current literature. To this end, we study the …
Inductive bias of multi-channel linear convolutional networks with bounded weight norm
M Jagadeesan, I Razenshteyn… - … on Learning Theory, 2022 - proceedings.mlr.press
We provide a function space characterization of the inductive bias resulting from minimizing
the $\ell_2 $ norm of the weights in multi-channel convolutional neural networks with linear …
the $\ell_2 $ norm of the weights in multi-channel convolutional neural networks with linear …
Linear Recursive Feature Machines provably recover low-rank matrices
A fundamental problem in machine learning is to understand how neural networks make
accurate predictions, while seemingly bypassing the curse of dimensionality. A possible …
accurate predictions, while seemingly bypassing the curse of dimensionality. A possible …