【论文笔记】没有归一化层的transformer：Transformers without Normalization

### Layer Normalization in Transformer Models Explained In the context of deep learning, normalization techniques play a crucial role in stabilizing and accelerating training processes. For Transformer models specifically, layer normalization (LayerNorm) has been an integral component contributing to their effectiveness. #### Definition and Purpose Layer normalization operates across features within each individual example rather than over mini-batches as batch normalization does. This approach ensures that during inference or when using very small batches, performance remains stable because it relies on per-example statistics instead of batch-level ones[^1]. The primary function of LayerNorm is to reduce internal covariate shift by normalizing activations at every hidden layer. By doing so, this technique facilitates smoother gradients flow through layers which can lead to faster convergence rates during optimization procedures. #### Implementation Details For implementing layer normalization inside a Transformer architecture like SimpleTransformer mentioned earlier: At each position \(i\) along sequence length dimension, the mean \(\mu_i\) and variance \(\sigma^2_i\) are computed based only upon values from current token's embedding vector. Then apply standard score transformation followed by learnable affine parameters gamma (\(\gamma\)) & beta (\(\beta\)): \[y = \frac{x-\mu}{\sqrt{\sigma^{2}+\epsilon}}*\gamma + \beta\] Here’s how one might implement such functionality in Python with PyTorch library: ```python import torch.nn as nn class LayerNormalization(nn.Module): "Construct a layernorm module." def __init__(self, features, eps=1e-6): super(LayerNormalization, self).__init__() self.a_2 = nn.Parameter(torch.ones(features)) self.b_2 = nn.Parameter(torch.zeros(features)) self.eps = eps def forward(self, x): mean = x.mean(-1, keepdim=True) std = x.std(-1, keepdim=True) return self.a_2 * (x - mean) / (std + self.eps) + self.b_2 ``` This code snippet defines a custom `LayerNormalization` class inheriting from PyTorch's Module base class. It initializes two trainable parameters (`a_2`, `b_2`) corresponding respectively to scaling factor γ and shifting term β used after applying zero-mean unit-variance normalization operation element-wise over input tensor 'x'. #### Benefits Within Transformers Within Transformer architectures, incorporating layer normalization before residual connections helps mitigate issues related to vanishing/exploding gradient problems often encountered while stacking multiple attention sub-layers sequentially together without proper regularization mechanisms applied throughout network topology design phase[^2]. --related questions-- 1. How does Batch Normalization differ fundamentally from Layer Normalization? 2. Can you explain why Layer Normalization contributes positively towards reducing Internal Covariate Shift? 3. What specific challenges do Deep Neural Networks face due to Vanishing/Exploding Gradients problem? 4. In what ways have advancements beyond basic Layer Norm impacted modern NLP tasks leveraging large-scale pre-trained language models?