Abstract Normalization layers (eg, Batch Normalization, Layer Normalization) were
introduced to help with optimization difficulties in very deep nets, but they clearly also help …

As part of the effort to understand implicit bias of gradient descent in overparametrized
models, several results have shown how the training trajectory on the overparametrized …

Diagonal linear networks (DLNs) are a toy simpli_cation of artificial neural networks; they
consist in a quadratic reparametrization of linear regression inducing a sparse implicit …

In the pursuit of explaining implicit regularization in deep learning, prominent focus was
given to matrix and tensor factorizations, which correspond to simplified neural networks. It …

We study the tensor-on-tensor regression, where the goal is to connect tensor responses to
tensor covariates with a low Tucker rank parameter tensor/matrix without the prior …

The question of what makes a data distribution suitable for deep learning is a fundamental
open problem. Focusing on locally connected neural networks (a prevalent family of …

Gradient descent (GD) is crucial for generalization in machine learning models, as it induces
implicit regularization, promoting compact representations. In this work, we examine the role …

Monotonic linear interpolation (MLI)---on the line connecting a random initialization with the
minimizer it converges to, the loss and accuracy are monotonic---is a phenomenon that is …

This work analyzes the solution trajectory of gradient-based algorithms via a novel basis
function decomposition. We show that, although solution trajectories of gradient-based …

Under a simplified data model, this paper provides a theoretical analysis of learning from
data that have an underlying low-rank tensor structure in both supervised and unsupervised …