Scribd
Upload
26 views12 pages

RNN Part1

Uploaded by

oaboalwafa75
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
26 views12 pages

RNN Part1

Uploaded by

oaboalwafa75
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 12

RECURRENT NEURAL NETWORK

Dr.Tamam Alsarhan

The University of Jordan


OUTLINE
 Introduction
 What is RNN?

 LSTM as RNN’s variant

 GRU as RNN’s variant


WHAT IS RECURRENT NEURAL NETWORKS
(RNN)?
 Recurrent Neural Network (RNN) is a class of neural networks that is powerful for
modeling sequence data such as time series or natural language. Basically, the main
idea behind this architecture is to use sequential information.
 The concept of RNN was brought up in 1986 [1].
 FeedForward Neural Network is a bad choice when making predictions on sequential
data. Why?

 Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. nature,
323(6088), 533-536.
RECURRENT NEURAL NETWORK
AN UNROLLED RECURRENT NEURAL
NETWORK
CONT…
 RNNs can use their internal state (memory) to process sequences of
inputs.
 First, it takes the X(0) from the sequence of input and then it outputs h(0)
which together with X(1) is the input for the next step. So, the h(0) and
X(1) is the input for the next step. Similarly, h(1) from the next is the
input with X(2) for the next step and so on. This way, it keeps
remembering the context while training.
WHY LONG SHORT TERM MEMORY
(LSTM)?
 If we are trying to predict the last word in “the clouds are in the sky,” we
don’t need any further context.
 On the contrary, predicting the last word in the text “I grew up in France
… I speak fluent French.” needs context from further back!

 The Problem of Long-Term Dependencies in traditional RNN is really


complex and LSTM is proposed to solve it.
WHAT IS LONG SHORT TERM MEMORY
(LSTM)?
 Long Short-Term Memory (LSTM) networks are a modified version of
recurrent neural networks, which makes it easier to remember past data
in memory. The vanishing gradient problem of RNN is resolved here.
LSTM is well-suited to classify, process, and predict time series given time
lags of unknown duration. It trains the model by using back-propagation.
 In an LSTM network, three gates are present:
it decides what component
is to be updated.
C’t provides change contents

Updating the cell state

Decide what part of the cell


state to output
REFRENCE
 ● http://colah.github.io/posts/2015-08-Understanding-
LSTMs/
 ● http://www.wildml.com/ ●
http://nikhilbuduma.com/2015/01/11/a-deep-dive-into-
recurrent-neural-netwo rks/
 ● http://deeplearning.net/tutorial/lstm.html ●
https://theclevermachine.files.wordpress.com/2014/09/act-
funs.png
 ● http://blog.terminal.com/demistifying-long-short-term-
memory-lstm-recurrent -neural-networks/
 ● A Critical Review of Recurrent Neural Networks for
Sequence Learning, Zachary C. Lipton, John Berkowitz
 ● Long Short-Term Memory, Hochreiter, Sepp and
Schmidhuber, Jurgen, 1997
 ● Gers, F. A.; Schmidhuber, J. & Cummins, F. A. (2000),
'Learning to Forget: Continual Prediction with LSTM.',
Neural Computation 12 (10) , 2451-2471

You might also like