Download to read offline
![History of Neural Network and Deep learning
● Neural Network and Perceptron learning algorithm: [McCulloch and Pitts
(1943), Rosenblatt (1957)]
● Backpropagation: Rumelhart, Hinton and Williams, 1986
○ Theoretically, a neural network can have any number of hidden layers.
○ But, in practice, it rarely had more than one layer hidden layers.
■ Computational issue: Limited computing power
■ Algorithmical issues: Vanishing gradient and Exploding gradient.
● Beginning of Deep learning: Late 1990’s and early 2000’s
○ Solutions:
■ Computational issue: Advance computing powers such as GPUs, TPUs
■ Algorithmical issues
● Pre-training (e.g., AutoEncoder, RBM)
● Better architectures (e.g., LSTM)
● Better activation functions (e.g., Relu) 4](https://image.slidesharecdn.com/cnn-rnn-lstm-attention-240530082850-929c9d52/75/Convolutional-and-Recurrent-Neural-Networks-4-2048.jpg)
![U
U
W
X0
X1
O0
O1
O1
= f(W.O0
+ U.X1
)
= f([W, U] . [O0
, x1
])
t
0
1
Network @ t = 1
31](https://image.slidesharecdn.com/cnn-rnn-lstm-attention-240530082850-929c9d52/75/Convolutional-and-Recurrent-Neural-Networks-31-2048.jpg)
![U
U
W
X0
X1
U
W
X2
O2
O2
= f(W.O1
+ U.X2
)
= f([W, U] . [O1
, x2
])
t
0
1
2
O0
O1
Network @ t = 2
32](https://image.slidesharecdn.com/cnn-rnn-lstm-attention-240530082850-929c9d52/75/Convolutional-and-Recurrent-Neural-Networks-32-2048.jpg)
![● Cue word for the prediction
○ Example 1: sky → clouds [3 units apart]
○ Example 2: hindi → India [9 units apart]
● As the sequence length increases, it becomes hard for RNNs to learn
“long-term dependencies.”
○ Vanishing gradients: If weights are small, gradient shrinks exponentially. Network stops
learning.
○ Exploding gradients: If weights are large, gradient grows exponentially. Weights fluctuate
and become unstable.
Vanishing/Exploding gradients
46](https://image.slidesharecdn.com/cnn-rnn-lstm-attention-240530082850-929c9d52/75/Convolutional-and-Recurrent-Neural-Networks-46-2048.jpg)
![Gated Recurrent Unit (GRU) [Cho et al. (2014)]
● A variant of simple RNN (Vanilla RNN)
● Similar to LSTM
○ Whatever LSTM can do GRU can also do.
● Differences
○ Cell state and hidden are merged together
○ Two gates
■ Reset gate - similar to forget
■ Update gate - similar to input gate
○ No output gate
○ Cell/Hidden state is completely exposed to subsequent units.
● GRU needs fewer parameters to learn and is relatively efficient w.r.t.
computation.
53](https://image.slidesharecdn.com/cnn-rnn-lstm-attention-240530082850-929c9d52/75/Convolutional-and-Recurrent-Neural-Networks-53-2048.jpg)
![Attention Mechanism
1. Bi-RNN Encoder
2. Attention
3. RNN Decoder
4. Output Embeddings
5. Output probabilities
78
[Garc´ıa-Mart´ınez et al., 2016]](https://image.slidesharecdn.com/cnn-rnn-lstm-attention-240530082850-929c9d52/75/Convolutional-and-Recurrent-Neural-Networks-78-2048.jpg)
![• Word2vec [Mikolov et al., 2013]
– Contextual model
– Two variants
• Skip-gram
• Continuous Bag-of-word
• GloVe [Pennington et al., 2014]
– Co-occurrence matrix
– Matrix Factorization
• FastText [Bojanowski et al., 2016]
– Similar to word2vec
– Works on sub-word level
• Bidirectional Encoder Representations from Transformers (BERT) [Devlin et al., 2018]
– Based on Transformer model
• Embeddings from Language Models (ELMo) [Peters et al., 2018]
– Contextual
• The representation for each word depends on the entire context in which it is used.](https://image.slidesharecdn.com/cnn-rnn-lstm-attention-240530082850-929c9d52/75/Convolutional-and-Recurrent-Neural-Networks-82-2048.jpg)
Convolutional and Recurrent Neural Networks
- 1.
- 2. Outline ● Deep Learning ●AutoEncoder ● Convolutional Neural Network (CNN) ● Recurrent Neural Network (RNN) ● Long Short Term Memory (LSTM) ● Gated Recurrent Unit (GRU) ● Attention Mechanism ● Few NLP Applications 2
- 3. Few key termsto start with ● Neurons ● Layers ○ Input, Output and Hidden ● Activation functions ○ Sigmoid, Tanh, Relu ● Softmax ● Weight matrices ○ Input → Hidden, Hidden → Hidden, Hidden → Output ● Backpropagation ○ Optimizers ■ Gradient Descent (GD), Stochastic Gradient Descent (SGD), Adam etc. ○ Error (Loss) functions ■ Mean-Squared Error, Cross-Entropy etc. ○ Gradient of error ○ Passes: Forward pass and Backward pass 3
- 4. History of NeuralNetwork and Deep learning ● Neural Network and Perceptron learning algorithm: [McCulloch and Pitts (1943), Rosenblatt (1957)] ● Backpropagation: Rumelhart, Hinton and Williams, 1986 ○ Theoretically, a neural network can have any number of hidden layers. ○ But, in practice, it rarely had more than one layer hidden layers. ■ Computational issue: Limited computing power ■ Algorithmical issues: Vanishing gradient and Exploding gradient. ● Beginning of Deep learning: Late 1990’s and early 2000’s ○ Solutions: ■ Computational issue: Advance computing powers such as GPUs, TPUs ■ Algorithmical issues ● Pre-training (e.g., AutoEncoder, RBM) ● Better architectures (e.g., LSTM) ● Better activation functions (e.g., Relu) 4
- 5. Deep Learning vsMachine Learning Paradigm ● The main advantage of deep learning based approaches is the trainable features, i.e., it extracts relevant features, on its own, during training. ● Requires minimal human intervention. 5
- 6. Why Deep Learning? ●Recall, artificial neural network tries to mimic the functionality of a brain. ● In brain, computations happen in layers. ● View of representation ○ As we go up in the network, we get high-level representations ⇒ Assists in performing more complex tasks. 6
- 7. Why Deep Architectureswere hard to train? ● General weight-updation rule ● For lower-layers in deep architecture ○ δj will vanish, if it is less than 1 ○ δj will explode, if it is more than 1 7
- 8.
- 9.
- 10. AutoEncoder: Layer 1 10 z= f(x), where z ≈ x z1 z2 z3 z4 z5
- 11.
- 12.
- 13.
- 14. Deep Learning Architectures ●Convolutional neural network (CNN) ○ Aims to extract the local spatial features ● Recurrent neural network (RNN) ○ Exploits the sequential information of a sentence (sentence is a sequence of words). 14
- 15. Convolutional Neural Network LeCunnand Bengio (1995) 15
- 16. Convolutional Neural Networks(CNN) ● A CNN consists of a series (≥ 1) of convolution layer and pooling layer. ● Convolutional operation extracts the feature representations from the input data. ○ Shares the convolution filters over different spatial locations, in a quest of extracting location-invariant features in the input. ○ Shape and weights of the convolution filter determine the features to be extracted from the input data. ○ In general, multiple filters of different shapes are used to ensure the diversity in the extracted features. ● Pooling operation extracts the most relevant features from the convoluted features. Similar to downsampling in image-processing. 16
- 17.
- 18. Recurrent Neural Network(RNN) 18
- 19. Recurrent Neural Network(RNN) ● A neural network with feedback connections ● Enable networks to do temporal processing ● Good at learning sequences ● Acts as memory unit 19
- 20. RNN - Example1 Part-of-speech tagging: ● Given a sentence X, tag each word its corresponding grammatical class. 20
- 21. RNN - Example2 ● ● ● ○ ○ ○ ○ 21
- 22.
- 23. How to trainRNNs? ● Typical FFN ○ Backpropagation algorithm ● RNNs ○ A variant of backpropagation algorithm namely Back-Propagation Through Time (BPTT). 23
- 24. BackPropagation Through Time(BPTT) Error for an instance = Sum of errors at each time step of the instance Gradient of error 24
- 25. BackPropagation Through Time(BPTT) For V For W (Similarly for U) 25
- 26. Visualization of RNNthrough Feed-Forward Neural Network 26
- 27. Problem, Data andNetwork Architecture ● Problem: ○ I/p sequence (X) : X0 , X1 , …, XT ○ O/p sequence (O) : O0 , O1 , …, OT ● Representation of data: ○ I/p dimension : 4 ■ X0 → 0 1 1 0 ○ O/p dimension : 3 ■ O0 → 0 0 1 ● Network Architecture ○ Number of neurons at I/p layer : 4 ○ Number of neurons at O/p layer : 3 ○ Do we need hidden layers? ■ If yes, number of neurons at each hidden layers 27
- 28. U X0 O0 t 0 Network @ t= 0 28
- 29.
- 30.
- 31. U U W X0 X1 O0 O1 O1 = f(W.O0 + U.X1 ) =f([W, U] . [O0 , x1 ]) t 0 1 Network @ t = 1 31
- 32. U U W X0 X1 U W X2 O2 O2 = f(W.O1 + U.X2 ) =f([W, U] . [O1 , x2 ]) t 0 1 2 O0 O1 Network @ t = 2 32
- 33.
- 34. U U W X0 X1 U W X2 W View 1 O1 O0 O2 O-1 =0 Different viewsof the network 34
- 35. U U W X0 X1 U W X2 W O1 O0 O2 W W W U U U X 0 X 1 X 2 O-1 =0 View1 View 2 Different views O1 O0 O2 O-1 =0 35
- 36. O0 O1 O2 W W W U UU X0 X1 X2 O-1 Ot Ot-1 U Xt View 3 View 4 W U U W X0 X1 U W X2 W O1 O0 O2 W W W U U U X 0 X 1 X 2 O-1 =0 View 1 View 2 Different views O1 O0 O2 O-1 =0 36
- 37. When to useRNNs 37
- 38. Usage ● Depends onthe problems that we aim to solve. ● Typically good for sequence processings. ● Some sort of memorization is required. 38
- 39. Bit reverse problem ●Problem definition: ○ Problem 1: Reverse a binary digit. ■ 0 → 1 and 1 → 0 ○ Problem 2: Reverse a sequence of binary digits. ■ 0 1 0 1 0 0 1 → 1 0 1 0 1 1 0 ■ Sequence: Fixed or Variable length ○ Problem 3: Reverse a sequence of bits over time. ■ 0 1 0 1 0 0 1 → 1 0 1 0 1 1 0 ○ Problem 4: Reverse a bit if the current i/p and previous o/p are same. Input sequence 1 1 0 0 1 0 0 0 1 1 Output sequence 1 0 1 0 1 0 1 0 1 0 39
- 40. Data Let ● Problem 1 ○I/p dimension: 1 bit O/p dimension: 1 bit ● Problem 2 ○ Fixed ■ I/p dimension: 10 bit O/p dimension: 10 bit ○ Variable: Pad each sequence upto max sequence length: 10 ■ Padding value: -1 ■ I/p dimension: 10 bit O/p dimension: 10 bit ● Problem 3 & 4 ○ Dimension of each element of I/p (X) : 1 bit ○ Dimension of each element of O/p (O) : 1 bit ○ Sequence length : 10 40
- 41. Network Architecture Problem 1: ●I/p neurons = 1 ● O/p neurons = 1 W W W U U Ot Ot-1 U Xt O-1 O10 W U X10 …. No. of I/p neurons = I/p dimension No. of O/p neurons = O/p dimension Problem 2: Fixed & Variable ● I/p neurons = 10 ● O/p neurons = 10 W O U X U X0 O0 O1 O10 X1 X10 Problem 3: ● I/p neurons = 1 ● O/p neurons = 1 ● Seq len = 10 U Xt = X10 , … , X1 , X0 Ot = O10 , … , O1 , O0 Problem 4: ● I/p neurons = 1 ● O/p neurons = 1 ● Seq len = 10 …. U X0 O0 O1 O10 X1 X10 …. 41 X0 O0 O1 X1
- 42. Different configurations ofRNNs Image Captioning Sentiment Analysis Machine Translation Language modelling 42
- 43.
- 44. Language modelling: Example- 1 • 44
- 45. Language modelling: Example- 2 • 45
- 46. ● Cue wordfor the prediction ○ Example 1: sky → clouds [3 units apart] ○ Example 2: hindi → India [9 units apart] ● As the sequence length increases, it becomes hard for RNNs to learn “long-term dependencies.” ○ Vanishing gradients: If weights are small, gradient shrinks exponentially. Network stops learning. ○ Exploding gradients: If weights are large, gradient grows exponentially. Weights fluctuate and become unstable. Vanishing/Exploding gradients 46
- 47. RNN extensions ● Bi-directionalRNN ● Deep (Bi-directional) RNN 47
- 48. Long Short TermMemory (LSTM) Hochreiter & Schmidhuber (1997) 48
- 49. LSTM ● A variantof simple RNN (Vanilla RNN) ● Capable of learning long dependencies. ● Regulates information flow from recurrent units. 49
- 50. Vanilla RNN vsLSTM 50
- 51. An LSTM cell 51 ●Cell state ct (blue arrow), hidden state ht (green arrow) and input xt (red arrow) ● Three gates ○ Forget (Red-dotted box) ○ Input (Green-dotted box) ○ Output (Blue-dotted box)
- 52. Gated Recurrent Units(GRU) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, Yoshua Bengio (2014) 52
- 53. Gated Recurrent Unit(GRU) [Cho et al. (2014)] ● A variant of simple RNN (Vanilla RNN) ● Similar to LSTM ○ Whatever LSTM can do GRU can also do. ● Differences ○ Cell state and hidden are merged together ○ Two gates ■ Reset gate - similar to forget ■ Update gate - similar to input gate ○ No output gate ○ Cell/Hidden state is completely exposed to subsequent units. ● GRU needs fewer parameters to learn and is relatively efficient w.r.t. computation. 53
- 54.
- 55. Application of DLmethods for NLP tasks 55
- 56. NLP hierarchy ● Likedeep learning, NLP happens in layers! ● Each task receives features from its previous (lower-level) task, and process them to produce its own output and so on. 56
- 57. NLP problems Problems Paradigm POSTagging Sequence Labelling Named Entity Recognition Sentiment Analysis Classification Machine Translation Sequence Transformation Question Answering Summarization 57
- 58.
- 59. RNN/LSTM/GRU for SequenceLabelling Part-of-speech tagging: ● Given a sentence X, tag each word its corresponding grammatical class. 59
- 60. ● Sentence matrix ●Pad sentence to ensure the sequence length ○ Pad length = filter_size - 1 ○ Evenly distribute padding at the start and end of the sequence. ● Apply Convolution filters ● Classification CNN for Sequence Labelling 60
- 61.
- 62. RNN/LSTM/GRU for SentenceClassification Sentiment Classification: ● Given a sentence X, identify the expressed sentiment. 62
- 63. Zhang, Y., Wallace,B. ; A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification; In Proceedings of the 8th International Joint Conference on Natural Language Processing (IJCNLP-2017); pages 253-263; Taipie, Taiwan; 2017. 1. Sentence matrix a. embeddings of words 2. Convolution filters a. Total 6 filters; Two each of size 2, 3 & 4. b. 1 feature maps for each filter 3. Pooling a. 1-max pooling 4. Concatenate the max-pooled vector 5. Classification a. Softmax CNN for Sentence Classification 63
- 64. Sequence to sequencetransformation with Attention Mechanism 64
- 65. Decoder Encoder Sequence labeling v/sSequence transformation PRP VBZ NNP I love mangoes I love mangoes PRP VBZ NNP • Sentence embeddings 65
- 66. Why sequence transformationis required? ● For many application length of I/p and O/p are not necessarily same. E.g. Machine Translation, Summarization, Question Answering etc. ● For many application length of O/p is not known. ● Non-monotone mapping: Reordering of words. ● Applications for which sequence transformation is not require ○ PoS tagging, ○ Named Entity Recognition ○ .... 66
- 67. Encode-Decode paradigm Decoder Encoder Ram eatsmango राम आम खाता <eos> है <eos> ● English-Hindi Machine Translation ○ Source sentence: 3 words ○ Target sentence: 4 words ○ Second word of the source sentence maps to 3rd & 4th words of the target sentence. ○ Third word of the source sentence maps to 2nd word of the target sentence 67
- 68. Problems with Encode-Decodeparadigm ● Encoding transforms the entire sentence into a single vector. ● Decoding process uses this sentence representation for predicting the output. ○ Quality of prediction depends upon the quality of sentence embeddings. ● After few time steps decoding process may not properly use the sentence representation due to long-term dependency. 68
- 69. Solutions ● To improvethe quality of predictions we can ○ Improve the quality of sentence embeddings ‘OR’ ○ Present the source sentence representation for prediction at each time step. ‘OR’ ○ Present the RELEVANT source sentence representation for prediction at each time step. ■ Encode - Attend - Decode (Attention mechanism) 69
- 70. Attention Mechanism ● Representthe source sentence by the set of output vectors from the encoder. ● Each output vector (OV) at time t is a contextual representation of the input at time t. Ram eats mango <eos> OV1 OV2 OV3 OV4 70
- 71. Attention Mechanism ● Eachof these output vectors (OVs) may not be equally relevant during decoding process at time t. ● Weighted average of the output vectors can resolve the relevancy. ○ Assign more weights to an output vector that needs more attention during decoding at time t. ● The weighted average context vector (CV) will be the input to decoder along with the sentence representation. ○ CVi = ∑j aij . OVj where aij is the attn-wt of the jth OV 71
- 72. Attention Mechanism Ram eatsmango <eos> Attention Decoder Encoder CV at1 at2 at3 at4 Decoder takes two inputs: ● Sentence vector ● Attention vector 72
- 73. Attention Mechanism राम Ram eatsmango <eos> CV at1 at2 at3 at4 t=1 73
- 74. Attention Mechanism राम आम Rameats mango <eos> CV at1 at2 at3 at4 t=2 74
- 75. Attention Mechanism राम आमखाता Ram eats mango <eos> CV at1 at2 at3 at4 t=3 75
- 76. Attention Mechanism राम आमखाता है Ram eats mango <eos> CV at1 at2 at3 at4 t=4 76
- 77. Attention Mechanism राम आमखाता है <eos> Ram eats mango <eos> CV at1 at2 at3 at4 t=5 77
- 78. Attention Mechanism 1. Bi-RNNEncoder 2. Attention 3. RNN Decoder 4. Output Embeddings 5. Output probabilities 78 [Garc´ıa-Mart´ınez et al., 2016]
- 79. Attention - Types Givenan input sequence (x1 , x2 , … , xN ) and an output sequence (y1 , y2 , … , yM ) ● Encoder-Decoder Attention ○ yj | x1 , x2 , … , xN ● Decoder Attention ○ yj | y1 , y2 , … , yj-1 ● Encoder Attention (Self) ○ xi | x1 , x2 , … , xN 79
- 80.
- 81.
- 82. • Word2vec [Mikolovet al., 2013] – Contextual model – Two variants • Skip-gram • Continuous Bag-of-word • GloVe [Pennington et al., 2014] – Co-occurrence matrix – Matrix Factorization • FastText [Bojanowski et al., 2016] – Similar to word2vec – Works on sub-word level • Bidirectional Encoder Representations from Transformers (BERT) [Devlin et al., 2018] – Based on Transformer model • Embeddings from Language Models (ELMo) [Peters et al., 2018] – Contextual • The representation for each word depends on the entire context in which it is used.
- 83. Few good reads.. ●Denny Britz; Recurrent Neural Networks Tutorial, Part 1-4 http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introducti on-to-rnns/ ● Andrej Karpathy; The Unreasonable Effectiveness of Recurrent Neural Networks http://karpathy.github.io/2015/05/21/rnn-effectiveness/ ● Chris Olah; Understanding LSTM Networks http://colah.github.io/posts/2015-08-Understanding-LSTMs/ 83
- 84.
- 85. Workshop on AI forComputational Social Systems (ACSS) Sunday, 9th Feb 2020 (http://lcs2.iiitd.edu.in/acss2020/) Organizer Laboratory for Computational Social Systems (LCS2) @ IIIT Delhi. 85 Registration Fee Rs. 200/-
- 86.