Vivekanand Education Society’s

Institute of Technology,
Chembur, Mumbai

ARTIFICIAL NEURAL NETWORK AND FUZZY LOGIC

FA C U LT Y N A M E : A S H W I N I S AWA N T
A S S I S TA N T P R O F E S S O R
D E PA RT M E N T O F E L E C T R O N I C S & T E L E C O M M U N I C AT I O N E N G I N E E R I N G
 SUPERVSED LEARNING
Faculty Name: Ashwini Sawant

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 2

 Architecture of a typical artificial neural network

Input Signals

Output Signals
Middle Layer
Input Layer Output Layer

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 3

 Analogy between biological and artificial
neural networks

Biological Neural Network Artificial Neural Network

Soma Neuron
Dendrite Input
Axon Output
Synapse Weight

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 4

 The neuron as a simple computing element
Diagram of a neuron

Input Signals Weights Output Signals

x1
Y
w1
x2
w2
Neuron Y Y

wn Y
xn

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 5

 ◼ The neuron computes the weighted sum of the input
signals and compares the result with a threshold
value, q. If the net input is less than the threshold,
the neuron output is –1. But if the net input is greater
than or equal to the threshold, the neuron becomes
activated and its output attains a value +1.
◼ The neuron uses the following transfer or activation
function:
n +1, if X  q
X =  xi wi Y=
i =1 −1, if X  q
◼ This type of activation function is called a sign
function.
21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 6
 Activation functions of a neuron

Step function Sign function Sigmoid function Linear function

Y Y Y Y
+1 +1 1 1

0 X 0 X 0 X 0 X
-1 -1 -1 -1

1, if X  0 sign +1, if X  0 sigmoid 1

step
Y = Y = Y = Y linear= X
0, if X  0 −1, if X  0 1 + e− X

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 7

 Can a single neuron learn a task?

◼ In 1958, Frank Rosenblatt introduced a training

algorithm that provided the first procedure for
training a simple ANN: a perceptron.
◼ The perceptron is the simplest form of a neural
network. It consists of a single neuron with
adjustable synaptic weights and a hard limiter.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 8

 Single-layer two-input perceptron

Inputs
x1 Linear Hard
w1 Combiner Limiter
Output
Y
w2
q
x2
Threshold

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 9

 The Perceptron
◼ The operation of Rosenblatt’s perceptron is based
on the McCulloch and Pitts neuron model. The
model consists of a linear combiner followed by a
hard limiter.
◼ The weighted sum of the inputs is applied to the
hard limiter, which produces an output equal to +1
if its input is positive and −1 if it is negative.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 10

 ◼ The aim of the perceptron is to classify inputs, x1,
x2, . . ., xn, into one of two classes, say A1
and A2.
◼ In the case of an elementary perceptron, the n-
dimensional space is divided by a hyperplane into
two decision regions. The hyperplane is defined by
the linearly separable function:
n
 xi wi − q = 0
i =1

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 11

 Linear separability in the perceptrons
x2 x2

Class A1

1
2
1
x1
Class A2 x1

x1w1 + x2w2 −q = 0 x1w1 + x2w2 + x3w3 −q = 0

x3
(a) Two-input perceptron. (b) Three-input perceptron.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 12

 How does the perceptron learn its classification
tasks?
This is done by making small adjustments in the
weights to reduce the difference between the actual
and desired outputs of the perceptron. The initial
weights are randomly assigned, usually in the range
[−0.5, 0.5], and then updated to obtain the output
consistent with the training examples.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 13

 ◼ If at iteration p, the actual output is Y(p) and the
desired output is Yd (p), then the error is given by:

e( p) = Yd ( p) −Y( p) where p = 1, 2, 3, . . .

Iteration p here refers to the pth training example

presented to the perceptron.
◼ If the error, e(p), is positive, we need to increase
perceptron output Y(p), but if it is negative, we
need to decrease Y(p).

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 14

 The perceptron learning rule
wi ( p + 1) = wi ( p) + a . xi ( p) . e( p)

where p = 1, 2, 3, . . .
a is the learning rate, a positive constant less than
unity.
The perceptron learning rule was first proposed by
Rosenblatt in 1960. Using this rule we can derive the
perceptron training algorithm for classification tasks.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 15

 Perceptron’s training algorithm
Step 1: Initialisation
Set initial weights w1, w2,…, wn and threshold q
to random numbers in the range [−0.5, 0.5].
If the error, e(p), is positive, we need to increase
perceptron output Y(p), but if it is negative, we
need to decrease Y(p).

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 16

 Perceptron’s training algorithm (continued)

Step 2: Activation
Activate the perceptron by applying inputs x1(p),
x2(p),…, xn(p) and desired output Yd (p).
Calculate the actual output at iteration p = 1
 n 
Y ( p ) = step   x i ( p ) w i ( p ) − q 
 i = 1 
where n is the number of the perceptron inputs,
and step is a step activation function.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 17

 Perceptron’s training algorithm (continued)
Step 3: Weight training
Update the weights of the perceptron
wi ( p + 1) = wi ( p) + Dwi ( p)
where Dwi(p) is the weight correction at iteration p.
The weight correction is computed by the delta rule:

Dwi ( p) = a  xi ( p) . e( p)
Step 4: Iteration
Increase iteration p by one, go back to Step 2 and
repeat the process until convergence.
21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 18
 Example of perceptron learning: the logical operation AND
Inputs Desired Initial Actual Error Final
Epoch output weights output weights
x1 x2 Yd w1 w2 Y e w1 w2
1 0 0 0 0.3 − 0.1 0 0 0.3 − 0.1
0 1 0 0.3 − 0.1 0 0 0.3 − 0.1
1 0 0 0.3 − 0.1 1 −1 0.2 − 0.1
1 1 1 0.2 − 0.1 0 1 0.3 0.0
2 0 0 0 0.3 0.0 0 0 0.3 0.0
0 1 0 0.3 0.0 0 0 0.3 0.0
1 0 0 0.3 0.0 1 −1 0.2 0.0
1 1 1 0.2 0.0 1 0 0.2 0.0
3 0 0 0 0.2 0.0 0 0 0.2 0.0
0 1 0 0.2 0.0 0 0 0.2 0.0
1 0 0 0.2 0.0 1 −1 0.1 0.0
1 1 1 0.1 0.0 0 1 0.2 0.1
4 0 0 0 0.2 0.1 0 0 0.2 0.1
0 1 0 0.2 0.1 0 0 0.2 0.1
1 0 0 0.2 0.1 1 −1 0.1 0.1
1 1 1 0.1 0.1 1 0 0.1 0.1
5 0 0 0 0.1 0.1 0 0 0.1 0.1
0 1 0 0.1 0.1 0 0 0.1 0.1
1 0 0 0.1 0.1 0 0 0.1 0.1
1 1 1 0.1 0.1 1 0 0.1 0.1
Threshold: q = 0.2; learning rate: = 0.1

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 19

 Two-dimensional plots of basic logical operations
x2 x2 x2

1 1 1

x1 x1 x1
0 1 0 1 0 1

(a) AND (x1  x2) (b) OR (x 1  x 2 ) (c) Exclus iv e-OR

(x 1  x2 )

A perceptron can learn the operations AND and OR,

but not Exclusive-OR.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 20

 Multilayer neural networks
◼ A multilayer perceptron is a feedforward neural network
with one or more hidden layers.
◼ The network consists of an input layer of source neurons,
at least one middle or hidden layer of computational
neurons, and an output layer of computational neurons.
◼ The input signals are propagated in a forward direction on
a layer-by-layer basis.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 21

 Multilayer perceptron with two hidden layers

Output Signals
Input Signals

First Second
Input hidden hidden Output
layer layer layer layer

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 22

 What does the middle layer hide?
◼ A hidden layer “hides” its desired output. Neurons in the
hidden layer cannot be observed through the input/output
behaviour of the network. There is no obvious way to know
what the desired output of the hidden layer should be.
◼ Commercial ANNs incorporate three and sometimes four
layers, including one or two hidden layers. Each layer can
contain from 10 to 1000 neurons. Experimental neural
networks may have five or even six layers, including three or
four hidden layers, and utilise millions of neurons.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 23

 Back-propagation neural network
◼ Learning in a multilayer network proceeds the same way
as for a perceptron.
◼ A training set of input patterns is presented to the
network.
◼ The network computes its output pattern, and if there is
an error − or in other words a difference between actual
and desired output patterns − the weights are adjusted to
reduce this error.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 24

 ◼ In a back-propagation neural network, the learning
algorithm has two phases.
◼ First, a training input pattern is presented to the network
input layer. The network propagates the input pattern
from layer to layer until the output pattern is generated by
the output layer.
◼ If this pattern is different from the desired output, an error
is calculated and then propagated backwards through the
network from the output layer to the input layer. The
weights are modified as the error is propagated.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 25

 Three-layer back-propagation neural network
Input signals
1
x1
1 y1
1
2
x2 2 y2
2

i wij j wjk
xi k yk

m
n l yl
xn
Input Hidden Output
layer layer layer

Error signals

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 26

 The back-propagation training algorithm
Step 1: Initialisation
Set all the weights and threshold levels of the
network to random numbers uniformly
distributed inside a small range:

 2.4 2.4 
 − , + 
 Fi Fi 
where Fi is the total number of inputs of neuron i
in the network. The weight initialisation is done
on a neuron-by-neuron basis.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 27

 Step 2: Activation
Activate the back-propagation neural network by
applying inputs x1(p), x2(p),…, xn(p) and desired
outputs yd,1(p), yd,2(p),…, yd,n(p).
(a) Calculate the actual outputs of the neurons in
the hidden layer:
 n 
y j ( p ) = sigmoid   xi ( p)  wij ( p ) − q j 
 i =1 
where n is the number of inputs of neuron j in the
hidden layer, and sigmoid is the sigmoid activation
function.
21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 28
 Step 2 : Activation (continued)
(b) Calculate the actual outputs of the neurons in
the output layer:
m 
yk ( p) = sigmoid   x jk ( p)  w jk ( p) − q k 
 j =1 
where m is the number of inputs of neuron k in the
output layer.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 29

Step 3: Weight training
Update the weights in the back-propagation network propagating
backward the errors associated with output neurons.
(a) Calculate the error gradient for the neurons in the output layer:

k ( p) = yk ( p)  1 − yk ( p)  ek ( p)
ek ( p) = yd ,k ( p) − yk ( p)
where
Calculate the weight corrections:
Dw jk ( p) =  y j ( p)  k ( p)
Update the weights at the output neurons:
w jk ( p + 1) = w jk ( p) + Dw jk ( p)
21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 30
 Step 3: Weight training (continued)
(b) Calculate the error gradient for the neurons in
the hidden layer:
l
j ( p) = y j ( p)  [1 − y j ( p)]   k ( p) w jk ( p)
k =1
Calculate the weight corrections:
Dwij ( p) =  xi ( p)  j ( p)
Update the weights at the hidden neurons:
wij ( p + 1) = wij ( p) + Dwij ( p)

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 31

 Step 4: Iteration
Increase iteration p by one, go back to Step 2 and
repeat the process until the selected error criterion
is satisfied.
As an example, we may consider the three-layer
back-propagation network. Suppose that the
network is required to perform logical operation
Exclusive-OR. Recall that a single-layer perceptron
could not do this operation. Now we will apply the
three-layer net.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 32

 Three-layer network for solving the Exclusive-
OR operation
−1

q3
w13 −1
x1 1 3 w35
w23 q5

5 y5
w24
x2 2 4 w45
w24
Input q4 Output
layer layer
−1
Hiddenlayer
21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 33
 ◼ The effect of the threshold applied to a neuron in the
hidden or output layer is represented by its weight, q,
connected to a fixed input equal to −1.
◼ The initial weights and threshold levels are set
randomly as follows:
w13 = 0.5, w14 = 0.9, w23 = 0.4, w24 = 1.0, w35 = −1.2,
w45 = 1.1, q3 = 0.8, q4 = −0.1 and q5 = 0.3.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 34

 ◼ We consider a training set where inputs x1 and x2 are
equal to 1 and desired output yd,5 is 0. The actual
outputs of neurons 3 and 4 in the hidden layer are
calculated as
y3 = sigmoid ( x1w13 + x2w23 − q3 ) = 1/ 1+ e−(10.5+10.4−10.8) = 0.5250

y4 = sigmoid ( x1w14 + x2w24 − q4 ) = 1/ 1 + e− (10.9+11.0+10.1) = 0.8808

◼ Now the actual output of neuron 5 in the output layer

is determined as:
y5 = sigmoid( y3w35 + y4w45 − q5) = 1/ 1+ e−(−0.52501.2+0.88081.1−10.3) = 0.5097

◼ Thus, the following error is obtained:

e = yd ,5 − y5 = 0 − 0.5097 = −0.5097

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 35

 ◼ The next step is weight training. To update the
weights and threshold levels in our network, we
propagate the error, e, from the output layer
backward to the input layer.
◼ First, we calculate the error gradient for neuron 5 in
the output layer:
5 = y5 (1− y5) e = 0.5097 (1− 0.5097) ( −0.5097)= −0.1274

◼ Then we determine the weight corrections assuming

that the learning rate parameter, a, is equal to 0.1:
Dw35 =  y3  5 = 0.1 0.5250 (−0.1274) = −0.0067
Dw45 =  y4  5 = 0.1 0.8808 (−0.1274) = −0.0112
Dq5 =  ( −1)  5 = 0.1 (−1)  (−0.1274) = −0.0127
21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 36
 ◼ Next we calculate the error gradients for neurons 3
and 4 in the hidden layer:
3 = y3(1 − y3)  5  w35 = 0.5250 (1 − 0.5250) ( − 0.1274) ( − 1.2) = 0.0381

4 = y4 (1− y4 )  5  w45 = 0.8808 (1 − 0.8808) ( − 0.1274)  1.1 = −0.0147

◼ We then determine the weight corrections:

Dw13 =  x1  3 = 0.1 1 0.0381= 0.0038
Dw23 =  x2  3 = 0.11 0.0381= 0.0038
D q3 =  ( −1)  3 = 0.1 ( −1)  0.0381= −0.0038
Dw14 =  x1  4 = 0.11 ( − 0.0147) = −0.0015
Dw24 =  x2  4 = 0.11 ( −0.0147) = −0.0015
Dq 4 =  ( −1)  4 = 0.1 ( −1)  ( −0.0147) = 0.0015

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 37

 ◼ At last, we update all weights and threshold:
w13 = w13 + D w13 = 0.5 + 0.0038 = 0.5038
w14 = w14 + Dw14 = 0.9 − 0.0015 = 0.8985
w23 = w23 + Dw23 = 0.4 + 0.0038 = 0.4038
w24 = w24 + D w24 = 1.0 − 0.0015 = 0.9985
w35 = w35 + D w35 = − 1.2 − 0.0067 = − 1.2067
w45 = w45 + D w45 = 1.1 − 0.0112 = 1.0888
q 3 = q 3 + D q 3 = 0.8 − 0.0038 = 0.7962
q 4 = q 4 + D q 4 = − 0.1 + 0.0015 = − 0.0985

q 5 = q 5 + D q 5 = 0.3 + 0.0127 = 0.3127

◼ The training process is repeated until the sum of

squared errors is less than 0.001.
21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 38
 Learning curve for operation Exclusive-OR
Sum-Squared Network Error for 224 Epochs
1
10

10 0

Sum-Squared Error
10 -1

10 -2

10 -3

10 -4
0 50 100 150 200
Epoch

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 39

 Final results of three-layer network learning

Inputs Desired Actual Sum of

output output squared
x1 x2 yd y5 e errors
1 1 0 0.0155 0.0010
0 1 1 0.9849
1 0 1 0.9849
0 0 0 0.0175

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 40

 Network represented by McCulloch-Pitts model for
solving the Exclusive-OR operation
−1

+1.5
−1
+1.0
x1 1 3 −2.0 +0.5
+1.0
5 y5
+1.0
x2 2 +1.0
4
+1.0
+0.5

−1
21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 41
 Decision boundaries
x2 x2 x2
x1 + x2 – 1.5 = 0 x1 + x2 – 0.5 = 0

1 1 1

x1 x1 x1
0 1 0 1 0 1

(a) (b) (c)

(a) Decision boundary constructed by hidden neuron 3;

(b) Decision boundary constructed by hidden neuron 4;
(c) Decision boundaries constructed by the complete
three-layer network
21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 42
 Accelerated learning in multilayer
neural networks
◼ A multilayer network learns much faster when the
sigmoidal activation function is represented by a
hyperbolic tangent:
tan h 2a
Y = − a
1+ e−bX
where a and b are constants.
Suitable values for a and b are:
a = 1.716 and b = 0.667

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 43

 ◼ We also can accelerate training by including a
momentum term in the delta rule:

Dw jk ( p) =  Dw jk ( p − 1) +  y j ( p)  k ( p)

where b is a positive number (0  b  1) called the

momentum constant. Typically, the momentum
constant is set to 0.95.

This equation is called the generalised delta rule.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 44

 Learning with momentum for operation Exclusive-OR
Training for 126 Epochs
2
10
10 1
10 0
10 -1
10 -2
10 -3
10 -4
0 20 40 60 80 100 120
Epoch

1.5

1
Learning Rate

0.5

-0.5

-1
0 20 40 60 80 100 120 140
Epoch

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 45

 Learning with adaptive learning rate
To accelerate the convergence and yet avoid the
danger of instability, we can apply two heuristics:
Heuristic 1
If the change of the sum of squared errors has the same
algebraic sign for several consequent epochs, then the
learning rate parameter, a, should be increased.
Heuristic 2
If the algebraic sign of the change of the sum of
squared errors alternates for several consequent epochs,
then the learning rate parameter, a, should be
decreased.
21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 46
 ◼ Adapting the learning rate requires some changes in the
back-propagation algorithm.
◼ If the sum of squared errors at the current epoch exceeds
the previous value by more than a predefined ratio
(typically 1.04), the learning rate parameter is decreased
(typically by multiplying by 0.7) and new weights and
thresholds are calculated.
◼ If the error is less than the previous one, the learning rate
is increased (typically by multiplying by 1.05).

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 47

 Learning with adaptive learning rate
Training for 103 Epochs
2
10
10 1

Sum-Squared Erro
10 0
10 -1
10 -2
10 -3
10 -4
0 10 20 30 40 50 60 70 80 90 100
Epoch
1

0.8
Learning Rate

0.6

0.4

0.2

0
0 20 40 60 80 100 120
Epoch

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 48

 Learning with momentum and adaptive learning rate
Trainingfor 85 Epochs
2
10

Sum-Squared Erro
10 1
10 0
10 -1
10 -2
10 -3
10 -4
0 10 20 30 40 50 60 70 80
Epoch
2.5

2
Learning Rate

1.5

0.5

0
0 10 20 30 40 50 60 70 80 90
Epoch

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 49

 The Hopfield Network
◼ Neural networks were designed on analogy with the brain. The
brain’s memory, however, works by association. For example,
we can recognise a familiar face even in an unfamiliar
environment within 100-200 ms. We can also recall a
complete sensory experience, including sounds and scenes,
when we hear only a few bars of music. The brain routinely
associates one thing with another.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 50

 ◼ Multilayer neural networks trained with the back-
propagation algorithm are used for pattern recognition
problems. However, to emulate the human memory’s
associative characteristics we need a different type of
network: a recurrent neural network.
◼ A recurrent neural network has feedback loops from its
outputs to its inputs. The presence of such loops has a
profound impact on the learning capability of the
network.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 51

 ◼ The stability of recurrent networks intrigued several
researchers in the 1960s and 1970s. However, none was
able to predict which network would be stable, and
some researchers were pessimistic about finding a
solution at all. The problem was solved only in 1982,
when John Hopfield formulated the physical principle
of storing information in a dynamically stable network.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 52

 Single-layer n-neuron Hopfield network

x1 1 y1

x2 2 y2

xi i yi

xn n yn

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 53

 ◼ The Hopfield network uses McCulloch and Pitts
neurons with the sign activation function as its
computing element:

+1, if X  0
sign 
Y = −1, if X  
 Y, if X = 


21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 54

 ◼ The current state of the Hopfield network is
determined by the current outputs of all neurons, y1,
y2, . . ., yn.
Thus, for a single-layer n-neuron network, the state
can be defined by the state vector as:

 y1 
 y2  
Y = 
 
 
 y n 

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 55

 ◼ In the Hopfield network, synaptic weights between
neurons are usually represented in matrix form as
follows:
M
W =  YmYm
T
−M I
m=1

where M is the number of states to be memorised

by the network, Ym is the n-dimensional binary
vector, I is n ´ n identity matrix, and superscript T
denotes matrix transposition.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 56

 Possible states for the three-neuron Hopfield
network
y2

(−1,1, −1) (1, 1, −1)

(−1, 1, 1) (1, 1, 1)
y1
0

(−1,−1,−1) (1,−1,−1)

(−1,−1, 1) (1,−1, 1)
y3
21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 57
 ◼ The stable state-vertex is determined by the weight
matrix W, the current input vector X, and the
threshold matrix q. If the input vector is partially
incorrect or incomplete, the initial state will converge
into the stable state-vertex after a few iterations.
◼ Suppose, for instance, that our network is required to
memorise two opposite states, (1, 1, 1) and (−1, −1, −1).
Thus,
1 −1
Y1 = 1 Y2 = −1 or Y1T = 1 1 1 Y2T = − 1 − 1 − 1
 
1 −1

where Y1 and Y2 are the three-dimensional vectors.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 58
 ◼ The 3 ´ 3 identity matrix I is
1 0 0
I = 0 1 0
0 0 1
◼ Thus, we can now determine the weight matrix as
follows:
1 −1 1 0 0 0 2 2
W = 1 1 1 1 + −1 −1 −1 −1 − 2 0 1 0 = 2 0 2
1 −1 0 0 1 2 2 0
◼ Next, the network is tested by the sequence of input
vectors, X1 and X2, which are equal to the output (or
target) vectors Y1 and Y2, respectively.
21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 59
 ◼ First, we activate the Hopfield network by applying
the input vector X. Then, we calculate the actual
output vector Y, and finally, we compare the result
with the initial input vector X.

0 2 2 1 0  1

       
Y1 = sign 2 0 2 1 − 0  = 1
2 0 1 0  1
 2

0 2 2 −1 0 −1

        
Y2 = sign2 0 2 −1 − 0 = −1
2 2 0 −1 0 −1


21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 60

 ◼ The remaining six states are all unstable. However, stable
states (also called fundamental memories) are capable of
attracting states that are close to them.
◼ The fundamental memory (1, 1, 1) attracts unstable states
(−1, 1, 1), (1, −1, 1) and (1, 1, −1). Each of these unstable
states represents a single error, compared to the
fundamental memory (1, 1, 1).
◼ The fundamental memory (−1, −1, −1) attracts unstable
states (−1, −1, 1), (−1, 1, −1) and (1, −1, −1).
◼ Thus, the Hopfield network can act as an error correction
network.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 61

 Storage capacity of the Hopfield network
◼ Storage capacity is or the largest number of
fundamental memories that can be stored and
retrieved correctly.
◼ The maximum number of fundamental memories
Mmax that can be stored in the n-neuron recurrent
network is limited by
M max = 0.15n

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 62

 Bidirectional associative memory (BAM)
◼ The Hopfield network represents an autoassociative type
of memory − it can retrieve a corrupted or incomplete
memory but cannot associate this memory with another
different memory.
◼ Human memory is essentially associative. One thing may
remind us of another, and that of another, and so on. We use
a chain of mental associations to recover a lost memory.
If we forget where we left an umbrella, we try to recall
where we last had it, what we were doing, and who we
were talking to. We attempt to establish a chain of
associations, and thereby to restore a lost memory.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 63

 ◼ To associate one memory with another, we need a
recurrent neural network capable of accepting an input
pattern on one set of neurons and producing a related, but
different, output pattern on another set of neurons.
◼ Bidirectional associative memory (BAM), first
proposed by Bart Kosko, is a heteroassociative network.
It associates patterns from one set, set A, to patterns from
another set, set B, and vice versa. Like a Hopfield
network, the BAM can generalise and also produce
correct outputs despite corrupted or incomplete inputs.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 64

 BAM operation
x1(p) 1 x1(p+1) 1

1 y1(p) 1 y1(p)
x2 (p) 2 x2(p+1) 2
2 y2(p) 2 y2(p)

xi (p)
j yj(p) j yj(p)
i xi(p+1) i

m ym(p) m ym(p)
xn(p) n xn(p+1) n
Input Output Input Output
layer layer layer layer
(a) Forward direction. (b) Backward direction.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 65

 The basic idea behind the BAM is to store
pattern pairs so that when n-dimensional vector
X from set A is presented as input, the BAM
recalls m-dimensional vector Y from set B, but
when Y is presented as input, the BAM recalls X.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 66

 ◼ To develop the BAM, we need to create a correlation
matrix for each pattern pair we want to store. The
correlation matrix is the matrix product of the input
vector X, and the transpose of the output vector YT.
The BAM weight matrix is the sum of all correlation
matrices, that is,
M
W=  m m
X Y T

m=1
where M is the number of pattern pairs to be stored in
the BAM.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 67

 Stability and storage capacity of the BAM
◼ The BAM is unconditionally stable. This means that any set
of associations can be learned without risk of instability.
◼ The maximum number of associations to be stored in the
BAM should not exceed the number of neurons in the
smaller layer.
◼ The more serious problem with the BAM is incorrect
convergence. The BAM may not always produce the
closest association. In fact, a stable association may be
only slightly related to the initial input vector.

21/02/2022 ASHWINI SAWANT, ASSISTANT PROFESSOR, VESIT 68