A theory of non-linear feature learning with one gradient step in two-layer neural networks
Feature learning is thought to be one of the fundamental reasons for the success of deep
neural networks. It is rigorously known that in two-layer fully-connected neural networks …
neural networks. It is rigorously known that in two-layer fully-connected neural networks …
Provable multi-task representation learning by two-layer relu neural networks
Feature learning, ie extracting meaningful representations of data, is quintessential to the
practical success of neural networks trained with gradient descent, yet it is notoriously …
practical success of neural networks trained with gradient descent, yet it is notoriously …
Learning hierarchical polynomials with three-layer neural networks
We study the problem of learning hierarchical polynomials over the standard Gaussian
distribution with three-layer neural networks. We specifically consider target functions of the …
distribution with three-layer neural networks. We specifically consider target functions of the …
Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit
We study the problem of gradient descent learning of a single-index target function $
f_*(\boldsymbol {x})=\textstyle\sigma_*\left (\langle\boldsymbol {x},\boldsymbol …
f_*(\boldsymbol {x})=\textstyle\sigma_*\left (\langle\boldsymbol {x},\boldsymbol …
Gradient descent induces alignment between weights and the empirical NTK for deep non-linear networks
Understanding the mechanisms through which neural networks extract statistics from input-
label pairs is one of the most important unsolved problems in supervised learning. Prior …
label pairs is one of the most important unsolved problems in supervised learning. Prior …
A novel domain adaptation method with physical constraints for shale gas production forecasting
L Gou, Z Yang, C Min, D Yi, X Li, B Kong - Applied Energy, 2024 - Elsevier
Effective forecasting of shale gas production is essential for optimizing exploration strategies
and guiding subsequent fracturing. However, in the new development of shale gas blocks …
and guiding subsequent fracturing. However, in the new development of shale gas blocks …
Feature learning as alignment: a structural property of gradient descent in non-linear neural networks
D Beaglehole, I Mitliagkas… - Transactions on Machine …, 2024 - openreview.net
Understanding the mechanisms through which neural networks extract statistics from input-
label pairs through feature learning is one of the most important unsolved problems in …
label pairs through feature learning is one of the most important unsolved problems in …
How Does Gradient Descent Learn Features--A Local Analysis for Regularized Two-Layer Neural Networks
The ability of learning useful features is one of the major advantages of neural networks.
Although recent works show that neural network can operate in a neural tangent kernel …
Although recent works show that neural network can operate in a neural tangent kernel …
Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning
Transformer-based large language models (LLMs) have displayed remarkable creative
prowess and emergence capabilities. Existing empirical studies have revealed a strong …
prowess and emergence capabilities. Existing empirical studies have revealed a strong …
Understanding Optimal Feature Transfer via a Fine-Grained Bias-Variance Analysis
In the transfer learning paradigm models learn useful representations (or features) during a
data-rich pretraining stage, and then use the pretrained representation to improve model …
data-rich pretraining stage, and then use the pretrained representation to improve model …