Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning
CH Martin, MW Mahoney - Journal of Machine Learning Research, 2021 - jmlr.org
Random Matrix Theory (RMT) is applied to analyze the weight matrices of Deep Neural
Networks (DNNs), including both production quality, pre-trained models such as AlexNet …
Networks (DNNs), including both production quality, pre-trained models such as AlexNet …
Dense Hebbian neural networks: a replica symmetric picture of supervised learning
E Agliari, L Albanese, F Alemanno… - Physica A: Statistical …, 2023 - Elsevier
We consider dense, associative neural-networks trained by a teacher (ie, with supervision)
and we investigate their computational capabilities analytically, via statistical-mechanics …
and we investigate their computational capabilities analytically, via statistical-mechanics …
Traditional and heavy-tailed self regularization in neural network models
CH Martin, MW Mahoney - arXiv preprint arXiv:1901.08276, 2019 - arxiv.org
Random Matrix Theory (RMT) is applied to analyze the weight matrices of Deep Neural
Networks (DNNs), including both production quality, pre-trained models such as AlexNet …
Networks (DNNs), including both production quality, pre-trained models such as AlexNet …
Multi-species mean field spin glasses. Rigorous results
We study a multi-species spin glass system where the density of each species is kept fixed
at increasing volumes. The model reduces to the Sherrington–Kirkpatrick one for the single …
at increasing volumes. The model reduces to the Sherrington–Kirkpatrick one for the single …
Phase diagram of restricted Boltzmann machines and generalized Hopfield networks with arbitrary priors
Restricted Boltzmann machines are described by the Gibbs measure of a bipartite spin
glass, which in turn can be seen as a generalized Hopfield network. This equivalence allows …
glass, which in turn can be seen as a generalized Hopfield network. This equivalence allows …
Phase transitions in restricted Boltzmann machines with generic priors
We study generalized restricted Boltzmann machines with generic priors for units and
weights, interpolating between Boolean and Gaussian variables. We present a complete …
weights, interpolating between Boolean and Gaussian variables. We present a complete …
Replica symmetry breaking in dense hebbian neural networks
L Albanese, F Alemanno, A Alessandrelli… - Journal of Statistical …, 2022 - Springer
Understanding the glassy nature of neural networks is pivotal both for theoretical and
computational advances in Machine Learning and Theoretical Artificial Intelligence. Keeping …
computational advances in Machine Learning and Theoretical Artificial Intelligence. Keeping …
The ground state energy and concentration of complexity in spherical bipartite models
P Kivimae - Communications in Mathematical Physics, 2023 - Springer
We establish an asymptotic formula for the ground-state energy of the spherical pure (p, q)-
spin glass model for p, q≥ 96. We achieve this through understanding the concentration of …
spin glass model for p, q≥ 96. We achieve this through understanding the concentration of …
Explorations on high dimensional landscapes
Finding minima of a real valued non-convex function over a high dimensional space is a
major challenge in science. We provide evidence that some such functions that are defined …
major challenge in science. We provide evidence that some such functions that are defined …
[HTML][HTML] Hopfield model with planted patterns: A teacher-student self-supervised learning model
F Alemanno, L Camanzi, G Manzan… - Applied Mathematics and …, 2023 - Elsevier
While Hopfield networks are known as paradigmatic models for memory storage and
retrieval, modern artificial intelligence systems mainly stand on the machine learning …
retrieval, modern artificial intelligence systems mainly stand on the machine learning …