Recently, over-parameterized deep networks, with increasingly more network parameters
than training samples, have dominated the performances of modern machine learning …

This paper studies the curious phenomenon for machine learning models with Transformer
architectures that their activation maps are sparse. By activation map we refer to the …

Recently, there has been significant progress in understanding the convergence and
generalization properties of gradient-based methods for training overparameterized learning …

In the pursuit of explaining implicit regularization in deep learning, prominent focus was
given to matrix and tensor factorizations, which correspond to simplified neural networks. It …

Diagonal linear networks (DLNs) are a toy simpli_cation of artificial neural networks; they
consist in a quadratic reparametrization of linear regression inducing a sparse implicit …

This paper presents a framework for smooth optimization of objectives with ℓq and ℓp, q
regularization for (structured) sparsity. Finding solutions to these non-smooth and possibly …

Understanding the implicit regularization imposed by neural network architectures and
gradient based optimization methods is a key challenge in deep learning and AI. In this work …

This work characterizes the effect of depth on the optimization landscape of linear
regression, showing that, despite their nonconvexity, deeper models have more desirable …

The connection between Residual Neural Networks (ResNets) and continuous-time control
systems (known as NeurODEs) has led to a mathematical analysis of neural networks, which …

We study the implicit regularization of gradient descent towards structured sparsity via a
novel neural reparameterization, which we call a diagonally grouped linear neural network …