Revisiting Internal Covariate Shift for Batch Normalization.

Despite the success of batch normalization (BatchNorm) and a plethora of its variants, the exact reasons for its success are still shady. The original BatchNorm article explained it as a mechanism that reduces the internal covariate shift (ICS), i.e.