Optimization-based separations for neural networks
Depth separation results propose a possible theoretical explanation for the benefits of deep
neural networks over shallower architectures, establishing that the former possess superior …
neural networks over shallower architectures, establishing that the former possess superior …
Width is less important than depth in relu neural networks
We solve an open question from Lu et al.(2017), by showing that any target network with
inputs in $\mathbb {R}^ d $ can be approximated by a width $ O (d) $ network (independent …
inputs in $\mathbb {R}^ d $ can be approximated by a width $ O (d) $ network (independent …
On the optimal memorization power of relu neural networks
We study the memorization power of feedforward ReLU neural networks. We show that such
networks can memorize any $ N $ points that satisfy a mild separability assumption using …
networks can memorize any $ N $ points that satisfy a mild separability assumption using …
Exponential separations in symmetric neural networks
In this work we demonstrate a novel separation between symmetric neural network
architectures. Specifically, we consider the Relational Network~\parencite …
architectures. Specifically, we consider the Relational Network~\parencite …
Depth Separation in Norm-Bounded Infinite-Width Neural Networks
We study depth separation in infinite-width neural networks, where complexity is controlled
by the overall squared $\ell_2 $-norm of the weights (sum of squares of all weights in the …
by the overall squared $\ell_2 $-norm of the weights (sum of squares of all weights in the …
The necessity of depth for artificial neural networks to approximate certain classes of smooth and bounded functions without the curse of dimensionality
In this article we study high-dimensional approximation capacities of shallow and deep
artificial neural networks (ANNs) with the rectified linear unit (ReLU) activation. In particular …
artificial neural networks (ANNs) with the rectified linear unit (ReLU) activation. In particular …
How Many Neurons Does it Take to Approximate the Maximum?
We study the size of a neural network needed to approximate the maximum function over d
inputs, in the most basic setting of approximating with respect to the L 2 norm, for continuous …
inputs, in the most basic setting of approximating with respect to the L 2 norm, for continuous …
Optimal Bump Functions for Shallow ReLU networks: Weight Decay, Depth Separation, Curse of Dimensionality
S Wojtowytsch - Journal of Machine Learning Research, 2024 - jmlr.org
In this note, we study how neural networks with a single hidden layer and ReLU activation
interpolate data drawn from a radially symmetric distribution with target labels 1 at the origin …
interpolate data drawn from a radially symmetric distribution with target labels 1 at the origin …
Spectral complexity of deep neural networks
It is well-known that randomly initialized, push-forward, fully-connected neural networks
weakly converge to isotropic Gaussian processes, in the limit where the width of all layers …
weakly converge to isotropic Gaussian processes, in the limit where the width of all layers …
Rethink depth separation with intra-layer links
The depth separation theory is nowadays widely accepted as an effective explanation for the
power of depth, which consists of two parts: i) there exists a function representable by a …
power of depth, which consists of two parts: i) there exists a function representable by a …