Implicit regularization of dropout
It is important to understand how dropout, a popular regularization method, aids in achieving
a good generalization solution during neural network training. In this work, we present a …
a good generalization solution during neural network training. In this work, we present a …
Initialization is Critical to Whether Transformers Fit Composite Functions by Inference or Memorizing
Transformers have shown impressive capabilities across various tasks, but their
performance on compositional problems remains a topic of debate. In this work, we …
performance on compositional problems remains a topic of debate. In this work, we …
Understanding the initial condensation of convolutional neural networks
Previous research has shown that fully-connected networks with small initialization and
gradient-based training methods exhibit a phenomenon known as condensation during …
gradient-based training methods exhibit a phenomenon known as condensation during …
Connectivity Shapes Implicit Regularization in Matrix Factorization Models for Matrix Completion
Matrix factorization models have been extensively studied as a valuable test-bed for
understanding the implicit biases of overparameterized models. Although both low nuclear …
understanding the implicit biases of overparameterized models. Although both low nuclear …
Disentangle Sample Size and Initialization Effect on Perfect Generalization for Single-Neuron Target
Overparameterized models like deep neural networks have the intriguing ability to recover
target functions with fewer sampled data points than parameters (see arXiv: 2307.08921). To …
target functions with fewer sampled data points than parameters (see arXiv: 2307.08921). To …
Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing
Transformers have shown impressive capabilities across various tasks, but their
performance on compositional problems remains a topic of debate. In this work, we …
performance on compositional problems remains a topic of debate. In this work, we …