Neural networks can learn representations with gradient descent
Significant theoretical work has established that in specific regimes, neural networks trained
by gradient descent behave like kernel methods. However, in practice, it is known that …
by gradient descent behave like kernel methods. However, in practice, it is known that …
Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction
D Stöger, M Soltanolkotabi - Advances in Neural …, 2021 - proceedings.neurips.cc
Recently there has been significant theoretical progress on understanding the convergence
and generalization of gradient-based methods on nonconvex losses with overparameterized …
and generalization of gradient-based methods on nonconvex losses with overparameterized …
Local signal adaptivity: Provable feature learning in neural networks beyond kernels
Neural networks have been shown to outperform kernel methods in practice (including
neural tangent kernels). Most theoretical explanations of this performance gap focus on …
neural tangent kernels). Most theoretical explanations of this performance gap focus on …
Understanding deflation process in over-parametrized tensor decomposition
In this paper we study the training dynamics for gradient flow on over-parametrized tensor
decomposition problems. Empirically, such training process often first fits larger components …
decomposition problems. Empirically, such training process often first fits larger components …
What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglement.
N De La Vega, N Razin… - Advances in Neural …, 2024 - proceedings.neurips.cc
The question of what makes a data distribution suitable for deep learning is a fundamental
open problem. Focusing on locally connected neural networks (a prevalent family of …
open problem. Focusing on locally connected neural networks (a prevalent family of …
Optimal gradient-based algorithms for non-concave bandit optimization
Bandit problems with linear or concave reward have been extensively studied, but relatively
few works have studied bandits with non-concave reward. This work considers a large family …
few works have studied bandits with non-concave reward. This work considers a large family …
Behind the scenes of gradient descent: A trajectory analysis via basis function decomposition
This work analyzes the solution trajectory of gradient-based algorithms via a novel basis
function decomposition. We show that, although solution trajectories of gradient-based …
function decomposition. We show that, although solution trajectories of gradient-based …
Going beyond linear rl: Sample efficient neural function approximation
Abstract Deep Reinforcement Learning (RL) powered by neural net approximation of the Q
function has had enormous empirical success. While the theory of RL has traditionally …
function has had enormous empirical success. While the theory of RL has traditionally …
Implicit regularization for group sparsity
We study the implicit regularization of gradient descent towards structured sparsity via a
novel neural reparameterization, which we call a diagonally grouped linear neural network …
novel neural reparameterization, which we call a diagonally grouped linear neural network …
Implicit Regularization for Tubal Tensor Factorizations via Gradient Descent
We provide a rigorous analysis of implicit regularization in an overparametrized tensor
factorization problem beyond the lazy training regime. For matrix factorization problems, this …
factorization problem beyond the lazy training regime. For matrix factorization problems, this …