Normalization techniques are essential for accelerating the training and improving the generalization of deep neural networks (DNNs), and have successfully been used in various …
Batch Normalization (BN) has become an out-of-box technique to improve deep network training. However, its effectiveness is limited for micro-batch training, ie, each GPU typically …
SH Gao, Q Han, D Li, MM Cheng… - Proceedings of the …, 2021 - openaccess.thecvf.com
Abstract Batch Normalization (BatchNorm) has become the default component in modern neural networks to stabilize training. In BatchNorm, centering and scaling operations, along …
Despite the success of batch normalization (BatchNorm) and a plethora of its variants, the exact reasons for its success are still shady. The original BatchNorm article explained it as a …
We address a learning-to-normalize problem by proposing Switchable Normalization (SN), which learns to select different normalizers for different normalization layers of a deep neural …
Inspired by BatchNorm, there has been an explosion of normalization layers in deep learning. Recent works have identified a multitude of beneficial properties in BatchNorm to …
Z Zhang, W Shao, J Gu, X Wang… - … Conference on Machine …, 2021 - proceedings.mlr.press
Abstract Model quantization is challenging due to many tedious hyper-parameters such as precision (bitwidth), dynamic range (minimum and maximum discrete values) and stepsize …
Group convolution, which divides the channels of ConvNets into groups, has achieved impressive improvement over the regular convolution operation. However, existing models …
A Labatie, D Masters… - Advances in Neural …, 2021 - proceedings.neurips.cc
We investigate the reasons for the performance degradation incurred with batch- independent normalization. We find that the prototypical techniques of layer normalization …