Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute a mean and variance which are then used to normalize the summed input to that neuron on each training case. This significantly reduces the training time in feed-forward neural networks. However, the effect of batch normalization is dependent on the mini-batch size and it is not obvious how to apply it to recurrent neural networks. In this paper, we transpose batch normalization into layer normalization by computing the mean and variance used for normalization from all of the summed inputs to the neurons in a layer on a single training case. Like batch normalization, we also give each neuron its own adaptive bias and gain which are applied after the normalization but before the non-linearity. Unlike batch normalization, layer normalization performs exactly the same computation at training and test times. It is also straightforward to apply to recurrent neural networks by computing the normalization statistics separately at each time step. Layer normalization is very effective at stabilizing the hidden state dynamics in recurrent networks. Empirically, we show that layer normalization can substantially reduce the training time compared with previously published techniques.
Layer Normalization (LN) offers a novel approach to normalizing neuron activities within a layer individually, enhancing the training of deep neural networks, particularly RNNs, by reducing training time and improving generalization.
Unlike Batch Normalization that normalizes across mini-batches, LN computes normalization statistics (mean and variance) for each training case, simplifying the process and making it consistent across training and testing.
Empirical studies validate LN's effectiveness in reducing training time and boosting performance in various tasks, highlighting its advantage over Batch Normalization especially in scenarios where batch statistics are impractical.
The paper discusses Layer Normalization's theoretical properties, such as invariance under certain transformations, its simplicity and independence from mini-batch size, making it a versatile choice for many network architectures.
In recent developments within deep learning, particularly in training state-of-the-art deep neural networks, the computational expense has been a significant concern. A crucial stride towards alleviating this challenge has been found in normalization techniques. Layer Normalization, distinct from its predecessor Batch Normalization, proposes a novel approach by normalizing the activities of neurons within a layer across a single training case rather than across different cases in a mini-batch. This paradigm shift not only simplifies the normalization process but also extends its benefits to training dynamics across both feed-forward networks and recurrent neural networks (RNNs).
Layer Normalization (LN) computes normalization statistics—mean and variance—based on the summed inputs to the neurons within a layer for each training case individually. This is a departure from Batch Normalization which leverages the distribution of these statistics across the mini-batch. The method essentially focuses on stabilizing the hidden state dynamics in RNNs, which has shown to significantly reduce training time and improve generalization performance. A pivotal advantage of LN is its consistency in computation at both training and test times, offering a straightforward and effective approach to normalization without introducing dependencies between training cases.
The empirical studies conducted to assess the efficiency of Layer Normalization presented compelling evidence in its favor. Notably, in RNNs, LN demonstrated substantial reductions in training time alongside enhancements in generalization performance compared to existing techniques. Such improvements were systematically validated across various tasks, including image-caption ranking, question-answering, and handwriting sequence generation, to name a few. Furthermore, attention was given to the inherent advantages of LN over Batch Normalization in contexts where batch statistics are either impractical or inefficient, such as in online learning tasks or models with considerable distributional shift over time.
From a theoretical standpoint, the paper explores the geometrical and invariance properties conveyed by Layer Normalization compared to other normalization strategies. Such analysis brings to light the distinct capability of LN in maintaining invariances under weights and data transformations, which are crucial for the stability and efficiency of learning in neural networks. The discussed invariance under re-scaling and re-centering of weights, as well as the robustness to input re-scaling, provides a solid foundation for the predicted benefits and lays ground for further exploration in this direction.
The introduction of Layer Normalization marks a significant step towards more efficient and stable training of deep neural networks, especially in the realm of RNNs where it addresses and mitigates the challenges associated with internal covariate shifts. Its simplicity, coupled with the removal of dependency on mini-batch size, not only makes LN a versatile choice for a wide array of network architectures but also opens avenues for advancements in optimizing training procedures in deep learning. Future work is poised to explore the integration of LN in convolutional neural networks (CNNs) and delve deeper into understanding the broader implications of normalization techniques on the dynamics of deep learning models.
Acknowledgments for the research were directed towards grants from NSERC, CFI, and Google, underscoring the collaborative effort and support in pushing the boundaries of AI and neural network training methodologies.