BIDIRECTIONAL RECURRENT NEURAL NETWORKS

Bidirectional Recurrent Neural Networks (BiRNNs) are a type of neural network architecture
designed to capture information from both past and future contexts in sequential data. This is
particularly useful in tasks where the output at a given time step depends on the entire input
sequence, rather than just the past inputs. Below is a summary of the key points discussed in
the text:

1. Causal Structure of Standard RNNs:

 Traditional RNNs process sequences in a causal manner, meaning the state at

time tt depends only on the past inputs x(1),x(2),…,x(t−1)x(1),x(2),…,x(t−1) and the
current input x(t)x(t).

 This structure is limiting in applications where future context is also important for
making predictions at time tt.

2. Need for Bidirectional RNNs:

 In many applications, such as speech recognition, handwriting recognition,
and bioinformatics, the correct interpretation of the current input may depend on
both past and future inputs.

 For example, in speech recognition, the interpretation of a phoneme may depend on

the next few phonemes due to co-articulation, or even on future words due to
linguistic dependencies.

3. Architecture of Bidirectional RNNs:

 A Bidirectional RNN consists of two separate RNNs:

o Forward RNN: Processes the sequence from the start to the end (left to right).

o Backward RNN: Processes the sequence from the end to the start (right to
left).

 At each time step tt, the output o(t)o(t) is computed based on the hidden states of both
the forward RNN h(t)h(t) and the backward RNN g(t)g(t).

 This allows the network to capture information from both the past and the future,
making it more effective for tasks requiring full-sequence context.
4. Advantages of Bidirectional RNNs:

 No Fixed-Size Window: Unlike feedforward or convolutional networks, BiRNNs do

not require a fixed-size window to capture context. They can naturally incorporate
information from the entire sequence.
 Sensitivity to Local Context: The output at time tt is most sensitive to the input
values around tt, but it can also leverage information from distant past and future
inputs if necessary.

5. Extension to 2D Inputs:

 The concept of bidirectional processing can be extended to two-dimensional inputs,

such as images.

 In this case, four RNNs can be used, each processing the input in one of the four
directions: up, down, left, and right.

 At each point (i,j)(i,j) on a 2D grid, the output Oi,jOi,j can capture both local and
long-range dependencies, similar to how BiRNNs work in 1D sequences.
6. Comparison with Convolutional Networks:

 Cost: RNNs applied to images are typically more computationally expensive than
convolutional networks.

 Long-Range Interactions: RNNs allow for long-range lateral interactions between

features in the same feature map, which can be beneficial in certain tasks.

 Convolutional Form: The forward propagation equations for RNNs on images can
be written in a form that resembles a convolution, where the bottom-up input is
computed first, followed by recurrent propagation across the feature map to
incorporate lateral interactions.

7. Applications:

 Bidirectional RNNs have been highly successful in applications such as:

o Handwriting Recognition (Graves et al., 2008; Graves and Schmidhuber,

2009)

o Speech Recognition (Graves and Schmidhuber, 2005; Graves et al., 2013)

o Bioinformatics (Baldi et al., 1999)

In summary, Bidirectional RNNs are a powerful extension of traditional RNNs that allow for
the incorporation of both past and future context in sequential data, making them highly
effective for tasks where the entire input sequence is relevant to the output.

ENCODER-DECODER SEQUENCE-TO-SEQUENCE ARCHITECTURES

Encoder-Decoder Sequence-to-Sequence Architectures, which are a type of Recurrent

Neural Network (RNN) architecture designed to map input sequences to output sequences of
potentially different lengths. This is particularly useful in applications like speech
recognition, machine translation, and question answering, where the input and output
sequences often vary in length.

Key Concepts:
1. Context Representation:

o The input to the RNN is referred to as the "context".

o The goal is to produce a representation CC of this context, which could be a

vector or a sequence of vectors summarizing the input
sequence X=(x(1),…,x(nx)).
2. Encoder-Decoder Architecture:

o Encoder (Input RNN): Processes the input sequence and emits the
context CC, typically as a function of its final hidden state.
 o Decoder (Output RNN): Generates the output
sequence Y=(y(1),…,y(ny)) conditioned on the context CC.

o The lengths of the input sequence nx and output sequence ny can vary, unlike
previous architectures where nx=ny=t.

3. Training:

o The encoder and decoder RNNs are trained jointly to maximize the average
log probability of the output sequence given the input sequence,
i.e., logP(y(1),…,y(ny)∣x(1),…,x(nx))

4. Context as Input to Decoder:

o If the context CC is a vector, the decoder RNN functions as a vector-to-

sequence RNN.
 o The input can be provided as the initial state of the RNN or connected to the
hidden units at each time step, or a combination of both.

5. Flexibility in Architecture:

o There is no constraint that the encoder and decoder must have the same size of
hidden layers.

6. Limitations and Enhancements:

o A limitation arises when the context CC output by the encoder is too small to
summarize a long sequence effectively.

o Bahdanau et al. addressed this by making CC a variable-length sequence and

introducing an attention mechanism that learns to associate elements of the
sequence CC with elements of the output sequence.

Applications:

 Machine Translation: The encoder-decoder architecture has been successfully used

for state-of-the-art translation systems.

 Speech Recognition: Effective in tasks where the input and output sequences differ in
length.

 Question Answering: Useful for generating variable-length answers based on input

questions.

Summary:

The Encoder-Decoder Sequence-to-Sequence Architecture is a powerful framework for

handling tasks where the input and output sequences are of different lengths. By jointly
training an encoder to summarize the input and a decoder to generate the output, this
architecture has achieved significant success in various applications. Enhancements like the
attention mechanism further improve its ability to handle long sequences and complex
dependencies.

DEEP RECURRENT NETWORKS (DRNS)

Deep Recurrent Networks (DRNs), which extend traditional Recurrent Neural Networks
(RNNs) by introducing depth into the transformations involved in the computation. Here's a
detailed explanation of the key points:

1. Basic RNN Computation Blocks:

 The computation in most RNNs can be decomposed into three main blocks:

1. Input to Hidden State: Transformation from the input to the hidden state.
 2. Previous Hidden State to Next Hidden State: Transformation from the
previous hidden state to the next hidden state.

3. Hidden State to Output: Transformation from the hidden state to the output.

 In a standard RNN (like the one in Figure 10.3), each of these blocks is associated
with a single weight matrix, representing a shallow transformation (a single layer
within a deep Multi-Layer Perceptron (MLP)).

2. Introducing Depth:

 Experimental Evidence: Research (e.g., Graves et al., 2013; Pascanu et al., 2014a)
suggests that introducing depth into these transformations can be beneficial. This is in
line with the idea that deeper networks can perform more complex mappings.

 Historical Context: Earlier work (e.g., Schmidhuber, 1992; El Hihi and Bengio,
1996; Jaeger, 2007a) also explored deep RNNs, indicating a long-standing interest in
enhancing RNNs with depth.

3. Deep RNN Architecture:

 Hierarchical State Decomposition: Graves et al. (2013) demonstrated the benefits of
decomposing the state of an RNN into multiple layers (as shown in Figure 10.13).
Lower layers transform the raw input into a more suitable representation, which is
then used by higher layers.

 Separate MLPs for Each Block: Pascanu et al. (2014a) proposed using separate
MLPs (possibly deep) for each of the three transformation blocks (input-to-hidden,
hidden-to-hidden, and hidden-to-output). This approach is illustrated in Figure 10.13.
4. Challenges and Solutions:

 Optimization Difficulty: While adding depth increases the network's capacity, it can
also make optimization more challenging. Deeper networks are generally harder to
train due to longer paths between variables in different time steps.

 Skip Connections: To mitigate this issue, skip connections can be introduced in the
hidden-to-hidden path (as shown in Figure 10.13c). These connections create shorter
paths between variables in different time steps, facilitating easier optimization.

5. Impact of Depth:

 Increased Capacity: Adding depth to each transformation block allows the network
to capture more complex patterns and dependencies in the data.

 Trade-offs: While deeper networks can model more complex functions, they require
careful design to ensure that they remain trainable. Skip connections are one way to
balance depth and trainability.

Summary:
Deep Recurrent Networks enhance traditional RNNs by introducing depth into the
transformations involved in input-to-hidden, hidden-to-hidden, and hidden-to-output
computations. This depth allows the network to perform more complex mappings, but it also
introduces challenges in optimization. Techniques like skip connections can help mitigate
these challenges, making deep RNNs more effective and easier to train. The experimental
evidence supports the idea that deeper architectures are beneficial for tasks requiring complex
sequence modeling.

RECURSIVE NEURAL NETWORKS (RECURSIVENNS)

Recursive Neural Networks (RecursiveNNs) are a class of neural networks designed to

process hierarchical or tree-structured data, making them particularly useful for tasks
involving structured inputs like natural language sentences, image parsing, or other data with
inherent tree-like relationships. Unlike Recurrent Neural Networks (RNNs), which process
sequences in a linear chain, RecursiveNNs operate on tree structures, allowing them to
capture hierarchical dependencies more effectively.

Key Concepts:

1. Tree-Structured Computation:

o RecursiveNNs process data by applying the same set of weights recursively

over a tree structure. Each node in the tree represents a transformation of its
child nodes, allowing the network to capture hierarchical patterns.
 o For example, in natural language processing, a sentence can be represented as
a parse tree, where each node corresponds to a phrase or word. The recursive
network processes the tree from the leaves (words) up to the root (the full
sentence), combining information at each level.

2. Advantages Over RNNs:

o Reduced Depth: For a sequence of length τ, the depth of computation in a

recursive network can be reduced from τ(in RNNs) to O(logτ) in a balanced
tree. This reduction helps mitigate issues like vanishing gradients and
improves the handling of long-term dependencies.

o Hierarchical Representation: RecursiveNNs naturally capture hierarchical

relationships, making them suitable for tasks like parsing, where the structure
of the input (e.g., a sentence) is inherently hierarchical.

3. Tree Structure:

o The structure of the tree is crucial for the performance of recursive networks.
In some cases, the tree structure is fixed (e.g., a balanced binary tree), while in
 others, it is determined by external methods (e.g., a parse tree from a natural
language parser).

o An open research question is how to automatically infer the optimal tree

structure from the data. Some approaches suggest learning the tree structure
jointly with the network parameters, allowing the model to adapt to the input.

4. Variants and Extensions:

o Tensor-Based Recursive Networks: Socher et al. (2013a) proposed using

tensor operations and bilinear forms in recursive networks to model
relationships between concepts more effectively. This approach is particularly
useful for tasks requiring the modeling of interactions between entities, such
as in knowledge graphs or relational data.

o Node-Specific Computations: In some recursive networks, the computation

at each node can vary. For example, Frasconi et al. (1998) proposed
associating inputs and targets with individual nodes, allowing for more
flexible and expressive models.

5. Applications:

o Natural Language Processing (NLP): RecursiveNNs have been used for

tasks like sentiment analysis, parsing, and sentence classification. For
example, Socher et al. (2011a, 2013a) applied recursive networks to parse
trees for sentiment analysis, where the network learns to combine word and
phrase embeddings hierarchically.

o Computer Vision: In tasks like scene parsing, recursive networks can model
the hierarchical structure of objects and their relationships within an image.

o Bioinformatics: RecursiveNNs can be used to model hierarchical

relationships in biological data, such as protein structures or gene expression
networks.

6. Challenges:

o Tree Structure Design: Choosing or learning the appropriate tree structure for
a given task remains a challenge. While fixed structures like balanced trees are
simple, they may not always capture the optimal hierarchy for the data.

o Optimization: Training recursive networks can be more complex than training

RNNs due to the tree-structured computation. Techniques like gradient
clipping and careful initialization are often necessary to ensure stable training.
Summary:
Recursive Neural Networks extend the capabilities of traditional RNNs by processing data
in a tree-structured manner, making them well-suited for tasks involving hierarchical or
structured data. They offer advantages like reduced computational depth and the ability to
capture hierarchical relationships, which are particularly useful in NLP, computer vision, and
bioinformatics. However, challenges like tree structure design and optimization complexity
remain active areas of research. Variants like tensor-based recursive networks and node-
specific computations further enhance their flexibility and applicability to complex tasks

LEAKY UNITS

Leaky Units are a strategy used in Recurrent Neural Networks (RNNs) to handle long-term
dependencies by allowing information to persist over time. They achieve this by
incorporating linear self-connections with weights close to one, enabling the network to
retain information from the past for extended periods. Here’s a detailed explanation of leaky
units:
Key Concepts:

1. Linear Self-Connections:

o Leaky units have linear self-connections with weights near one, allowing
them to accumulate and retain information over time.

o The update rule for a leaky unit can be expressed as:

2. Time Constants:

o The parameter αα controls the time constant of the leaky unit:

 When αα is close to 1, the unit retains information from the past for a
long time, acting like a long-term memory.

 When αα is close to 0, the unit rapidly discards past information,

focusing on recent inputs.

o This flexibility allows leaky units to operate at different time scales, making
them useful for capturing both short-term and long-term dependencies.

3. Adaptive Time Scales:

o The time constants of leaky units can be either fixed or learned:

  Fixed Time Constants: The values of αα are set manually or sampled
from a distribution during initialization and remain constant during
training.

 Learned Time Constants: The values of αα are treated as learnable

parameters, allowing the network to adaptively determine the optimal
time scales for different tasks.

4. Advantages:

o Long-Term Dependencies: Leaky units help mitigate the vanishing gradient

problem by providing paths where gradients do not vanish as quickly, enabling
the network to learn long-term dependencies.

o Smooth Information Flow: Unlike skip connections, which introduce

discrete jumps in time, leaky units allow for a smooth and continuous flow
of information across time steps.

5. Applications:

o Leaky units have been successfully used in various architectures,

including Echo State Networks (ESNs) and other RNN variants.

o They are particularly effective in tasks requiring the modeling of long-term

dependencies, such as time-series prediction, speech recognition,
and natural language processing.

Summary:

Leaky Units are a powerful mechanism for enabling RNNs to capture long-term
dependencies by incorporating linear self-connections with weights near one. They allow the
network to retain information over extended periods, with the flexibility to operate at
different time scales. By adjusting the time constants (either fixed or learned), leaky units
provide a smooth and adaptive way to manage information flow across time steps, making
them a valuable tool for tasks involving sequential data.

LONG SHORT-TERM MEMORY (LSTM)

Long Short-Term Memory (LSTM) networks are a specialized type of Recurrent Neural
Network (RNN) designed to address the problem of learning long-term dependencies. LSTMs
introduce self-loops and gating mechanisms to control the flow of information, allowing
gradients to flow for long durations without vanishing or exploding. Here’s a detailed
explanation of LSTMs:

Key Concepts:

1. Core Idea:
 o LSTMs introduce self-loops to create paths where gradients can flow for long
durations, enabling the network to learn long-term dependencies.

o The self-loop weight is conditioned on the context rather than being fixed,
allowing the time scale of integration to change dynamically based on the input
sequence.

2. LSTM Cell:

o An LSTM cell has an internal state s(t)s(t) that is updated over time.

o The cell state is controlled by gating units:

 Forget Gate (f(t)): Determines how much of the previous

state s(t−1)s(t−1) to retain.

 Input Gate (g(t)): Controls how much of the new input is added to the
state.

 Output Gate (q(t)): Controls how much of the cell state is output to the
hidden state.

3. Gating Mechanisms:
4. Advantages:
o Long-Term Dependencies: LSTMs are designed to capture long-term
dependencies, making them effective for tasks where the context from earlier
time steps is crucial.
o Dynamic Time Scales: The gating mechanisms allow the network to
dynamically adjust the time scale of integration based on the input sequence.
5. Applications:
o LSTMs have been successfully applied in various domains, including:
 Unconstrained Handwriting Recognition (Graves et al., 2009)
 Speech Recognition (Graves et al., 2013; Graves and Jaitly, 2014)
 Machine Translation (Sutskever et al., 2014)
 Image Captioning (Kiros et al., 2014b; Vinyals et al., 2014b; Xu et
al., 2015)
 Parsing (Vinyals et al., 2014a)
6. Variants and Alternatives:
 o Several variants of LSTMs have been proposed to improve performance or
adapt to specific tasks. These include:
 Gated Recurrent Units (GRUs): A simplified version of LSTMs with
fewer parameters.
 Peephole Connections: Additional connections that allow gates to
inspect the cell state directly.
 Depth Gated LSTMs: Incorporating deeper architectures for more
complex tasks.

Summary:

LSTM networks are a powerful extension of traditional RNNs, designed to handle long-term
dependencies through the use of self-loops and gating mechanisms. By dynamically adjusting
the time scale of integration and controlling the flow of information, LSTMs have achieved
state-of-the-art performance in various sequence processing tasks. Their ability to capture
long-term dependencies makes them particularly effective in applications like speech
recognition, machine translation, and image captioning. Variants and alternatives continue to
be explored to further enhance their capabilities.