A comprehensive survey on regularization strategies in machine learning
Y Tian, Y Zhang - Information Fusion, 2022 - Elsevier
In machine learning, the model is not as complicated as possible. Good generalization
ability means that the model not only performs well on the training data set, but also can …
ability means that the model not only performs well on the training data set, but also can …
Normalization techniques in training dnns: Methodology, analysis and application
Normalization techniques are essential for accelerating the training and improving the
generalization of deep neural networks (DNNs), and have successfully been used in various …
generalization of deep neural networks (DNNs), and have successfully been used in various …
Micro-batch training with batch-channel normalization and weight standardization
Batch Normalization (BN) has become an out-of-box technique to improve deep network
training. However, its effectiveness is limited for micro-batch training, ie, each GPU typically …
training. However, its effectiveness is limited for micro-batch training, ie, each GPU typically …
Representative batch normalization with feature calibration
Abstract Batch Normalization (BatchNorm) has become the default component in modern
neural networks to stabilize training. In BatchNorm, centering and scaling operations, along …
neural networks to stabilize training. In BatchNorm, centering and scaling operations, along …
Revisiting internal covariate shift for batch normalization
Despite the success of batch normalization (BatchNorm) and a plethora of its variants, the
exact reasons for its success are still shady. The original BatchNorm article explained it as a …
exact reasons for its success are still shady. The original BatchNorm article explained it as a …
Switchable normalization for learning-to-normalize deep representation
We address a learning-to-normalize problem by proposing Switchable Normalization (SN),
which learns to select different normalizers for different normalization layers of a deep neural …
which learns to select different normalizers for different normalization layers of a deep neural …
Beyond batchnorm: Towards a unified understanding of normalization in deep learning
Inspired by BatchNorm, there has been an explosion of normalization layers in deep
learning. Recent works have identified a multitude of beneficial properties in BatchNorm to …
learning. Recent works have identified a multitude of beneficial properties in BatchNorm to …
Differentiable dynamic quantization with mixed precision and adaptive resolution
Abstract Model quantization is challenging due to many tedious hyper-parameters such as
precision (bitwidth), dynamic range (minimum and maximum discrete values) and stepsize …
precision (bitwidth), dynamic range (minimum and maximum discrete values) and stepsize …
Differentiable learning-to-group channels via groupable convolutional neural networks
Group convolution, which divides the channels of ConvNets into groups, has achieved
impressive improvement over the regular convolution operation. However, existing models …
impressive improvement over the regular convolution operation. However, existing models …
Proxy-normalizing activations to match batch normalization while removing batch dependence
We investigate the reasons for the performance degradation incurred with batch-
independent normalization. We find that the prototypical techniques of layer normalization …
independent normalization. We find that the prototypical techniques of layer normalization …