Bome! bilevel optimization made easy: A simple first-order approach
Bilevel optimization (BO) is useful for solving a variety of important machine learning
problems including but not limited to hyperparameter optimization, meta-learning, continual …
problems including but not limited to hyperparameter optimization, meta-learning, continual …
Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data
Benign overfitting, the phenomenon where interpolating models generalize well in the
presence of noisy data, was first observed in neural network models trained with gradient …
presence of noisy data, was first observed in neural network models trained with gradient …
Provable guarantees for neural networks via gradient feature learning
Neural networks have achieved remarkable empirical performance, while the current
theoretical analysis is not adequate for understanding their success, eg, the Neural Tangent …
theoretical analysis is not adequate for understanding their success, eg, the Neural Tangent …
Benign overfitting in two-layer ReLU convolutional neural networks
Modern deep learning models with great expressive power can be trained to overfit the
training data but still generalize well. This phenomenon is referred to as benign overfitting …
training data but still generalize well. This phenomenon is referred to as benign overfitting …
Implicit bias in leaky relu networks trained on high-dimensional data
The implicit biases of gradient-based optimization algorithms are conjectured to be a major
factor in the success of modern deep learning. In this work, we investigate the implicit bias of …
factor in the success of modern deep learning. In this work, we investigate the implicit bias of …
On penalty methods for nonconvex bilevel optimization and first-order stochastic approximation
In this work, we study first-order algorithms for solving Bilevel Optimization (BO) where the
objective functions are smooth but possibly nonconvex in both levels and the variables are …
objective functions are smooth but possibly nonconvex in both levels and the variables are …
On momentum-based gradient methods for bilevel optimization with nonconvex lower-level
F Huang - arXiv preprint arXiv:2303.03944, 2023 - arxiv.org
Bilevel optimization is a popular two-level hierarchical optimization, which has been widely
applied to many machine learning tasks such as hyperparameter learning, meta learning …
applied to many machine learning tasks such as hyperparameter learning, meta learning …
Self-training converts weak learners to strong learners in mixture models
We consider a binary classification problem when the data comes from a mixture of two
rotationally symmetric distributions satisfying concentration and anti-concentration …
rotationally symmetric distributions satisfying concentration and anti-concentration …
On feature learning in neural networks with global convergence guarantees
We study the optimization of wide neural networks (NNs) via gradient flow (GF) in setups that
allow feature learning while admitting non-asymptotic global convergence guarantees. First …
allow feature learning while admitting non-asymptotic global convergence guarantees. First …
Enhanced adaptive gradient algorithms for nonconvex-PL minimax optimization
F Huang - arXiv preprint arXiv:2303.03984, 2023 - arxiv.org
In the paper, we study a class of nonconvex nonconcave minimax optimization problems (ie,
$\min_x\max_y f (x, y) $), where $ f (x, y) $ is possible nonconvex in $ x $, and it is …
$\min_x\max_y f (x, y) $), where $ f (x, y) $ is possible nonconvex in $ x $, and it is …