We show that taking the width and depth to infinity in a deep neural network with skip
connections, when branches are scaled by $1/\sqrt {depth} $, result in the same covariance …

In this paper, we study the infinite-depth limit of finite-width residual neural networks with
random Gaussian weights. With proper scaling, we show that by fixing the width and taking …

Adversarial training has shown promise in building robust models against adversarial
examples. A major drawback of adversarial training is the computational overhead …

Dropout is an effective method of mitigating over-fitting while training deep neural networks
(DNNs). This method consists of switching off (dropping) some of the neurons of the DNN …

Decentralized training of large language models has emerged as an effective way to
democratize this technology. However, the potential threats associated with this approach …

Deep learning-based, data-driven models are gaining prevalence in climate research,
particularly for global weather prediction. However, training the global weather data at high …

This paper is the second in the series Commutative Scaling of Width and Depth (WD) about
commutativity of infinite width and depth limits in deep neural networks. Our aim is to …

Data pruning consists of identifying a subset of the training set that can be used for training
instead of the full dataset. This pruned dataset is often chosen to satisfy some desirable …

Abstract Modern Deep Neural Networks (DNNs) exhibit impressive generalization properties
on a variety of tasks without explicit regularization, suggesting the existence of hidden …

Residual connections are ubiquitous in deep learning, since besides residual networks and
their variants, they are also present in Transformer architectures. The dynamic view of …