CHAPTER 2
ARTIFICIAL NEURAL
NETWORKS: AN
INTRODUCTION
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
DEFINITION OF NEURAL NETWORKS
According to the DARPA Neural Network Study (1988,
AFCEA International Press, p. 60):
• ... a neural network is a system composed of many simple
processing elements operating in parallel whose function is
determined by network structure, connection strengths, and
the processing performed at computing elements or nodes.
According to Haykin (1994), p. 2:
A neural network is a massively parallel distributed processor
that has a natural propensity for storing experiential
knowledge and making it available for use. It resembles the
brain in two respects:
•Knowledge is acquired by the network through a learning
process. “Principles of Soft Computing, 2nd Edition”
•Interneuron connection strengths known as bysynaptic weights
S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
BRAIN COMPUTATION
The human brain contains about 10 billion nerve
cells, or neurons. On average, each neuron is
connected to other neurons through approximately
10,000 synapses.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
INTERCONNECTIONS IN BRAIN
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
BIOLOGICAL (MOTOR) NEURON
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
ARTIFICIAL NEURAL NET
Information-processing system.
Neurons process the information.
The signals are transmitted by means of connection links.
The links possess an associated weight.
The output signal is obtained by applying activations to
the net input.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
MOTIVATION FOR NEURAL NET
Scientists are challenged to use machines more
effectively for tasks currently solved by humans.
Symbolic rules don't reflect processes actually used by
humans.
Traditional computing excels in many areas, but not in
others.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
The major areas being:
Massive parallelism
Distributed representation and computation
Learning ability
Generalization ability
Adaptivity
Inherent contextual information processing
Fault tolerance
Low energy consumption.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
ARTIFICIAL NEURAL NET
W1
X1 Y
W2
X2
The figure shows a simple artificial neural net with two input
neurons (X1, X2) and one output neuron (Y). The inter
connected weights are given by W1 and W2.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
ASSOCIATION OF BIOLOGICAL NET
WITH ARTIFICIAL NET
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
PROCESSING OF AN ARTIFICIAL NET
The neuron is the basic information processing unit of a NN. It
consists of:
1. A set of links, describing the neuron inputs, with weights
W1, W2, …, Wm.
2. An adder function (linear combiner) for computing the
weighted sum of the inputs (real numbers):
m
u W j X j
j1
3. Activation function for limiting the amplitude of the neuron
output. y (u b)
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
BIAS OF AN ARTIFICIAL NEURON
The bias value is added to the weighted sum
∑wixi so that we can transform it from the origin.
Yin = ∑wixi + b, where b is the bias
x1-x2= -1
x2 x1-x2=0
x1-x2= 1
x1
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
MULTI LAYER ARTIFICIAL NEURAL NET
INPUT: records without class attribute with normalized
attributes values.
INPUT VECTOR: X = { x1, x2, …, xn} where n is the number
of (non-class) attributes.
INPUT LAYER: there are as many nodes as non-class
attributes, i.e. as the length of the input vector.
HIDDEN LAYER: the number of nodes in the hidden layer
and the number of hidden layers depends on implementation.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
OPERATION OF A NEURAL NET
- Bias
x0 w0j
x1 w1j
f
Output y
xn wnj
Input Weight Weighted Activation
vector x vector w sum function
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
WEIGHT AND BIAS UPDATION
Per Sample Updating
•updating weights and biases after the presentation of each
sample.
Per Training Set Updating (Epoch or Iteration)
•weight and bias increments could be accumulated in
variables and the weights and biases updated after all the
samples of the training set have been presented.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
STOPPING CONDITION
All change in weights (wij) in the previous epoch are
below some threshold, or
The percentage of samples misclassified in the previous
epoch is below some threshold, or
A pre-specified number of epochs has expired.
In practice, several hundreds of thousands of epochs may
be required before the weights will converge.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
BUILDING BLOCKS OF ARTIFICIAL NEURAL
NET
Network Architecture (Connection between Neurons)
Setting the Weights (Training)
Activation Function
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
LAYER PROPERTIES
Input Layer: Each input unit may be designated by an
attribute value possessed by the instance.
Hidden Layer: Not directly observable, provides
nonlinearities for the network.
Output Layer: Encodes possible values.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
TRAINING PROCESS
Supervised Training - Providing the network with a
series of sample inputs and comparing the output with the
expected responses.
Unsupervised Training - Most similar input vector is
assigned to the same output unit.
Reinforcement Training - Right answer is not provided
but indication of whether ‘right’ or ‘wrong’ is provided.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
ACTIVATION FUNCTION
ACTIVATION LEVEL – DISCRETE OR CONTINUOUS
HARD LIMIT FUCNTION (DISCRETE)
• Binary Activation function
• Bipolar activation function
• Identity function
SIGMOIDAL ACTIVATION FUNCTION (CONTINUOUS)
• Binary Sigmoidal activation function
• Bipolar Sigmoidal activation function
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
ACTIVATION FUNCTION
Activation
functions:
(A) Identity
(B) Binary step
(C) Bipolar step
(D) Binary sigmoidal
(E) Bipolar sigmoidal
(F) Ramp
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
CONSTRUCTING ANN
Determine the network properties:
• Network topology
• Types of connectivity
• Order of connections
• Weight range
Determine the node properties:
• Activation range
Determine the system dynamics
• Weight initialization scheme
• Activation – calculating formula
• Learning rule
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
PROBLEM SOLVING
Select a suitable NN model based on the nature of the
problem.
Construct a NN according to the characteristics of the
application domain.
Train the neural network with the learning procedure of
the selected model.
Use the trained network for making inference or solving
problems.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
NEURAL NETWORKS
Neural Network learns by adjusting the weights so as to
be able to correctly classify the training data and hence,
after testing phase, to classify unknown data.
Neural Network needs long time for training.
Neural Network has a high tolerance to noisy and
incomplete data.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
SALIENT FEATURES OF ANN
Adaptive learning
Self-organization
Real-time operation
Fault tolerance via redundant information coding
Massive parallelism
Learning and generalizing ability
Distributed representation
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
McCULLOCH–PITTS NEURON
Neurons are sparsely and randomly connected
Firing state is binary (1 = firing, 0 = not firing)
All but one neuron are excitatory (tend to increase voltage
of other cells)
• One inhibitory neuron connects to all other neurons
• It functions to regulate network activity (prevent too
many firings)
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
LINEAR SEPARABILITY
Linear separability is the concept wherein the separation
of the input space into regions is based on whether the
network response is positive or negative.
Consider a network having
positive response in the first
quadrant and negative
response in all other
quadrants (AND function)
with either binary or bipolar
data, then the decision line
is drawn separating the
positive response region
from the negative response
region.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
HEBB NETWORK
Donald Hebb stated in 1949 that in the brain, the learning is
performed by the change in the synaptic gap. Hebb explained
it:
“When an axon of cell A is near enough to excite cell B, and
repeatedly or permanently takes place in firing it, some
growth process or metabolic change takes place in one or
both the cells such that A’s efficiency, as one of the cells
firing B, is increased.”
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
HEBB LEARNING
The weights between neurons whose activities are
positively correlated are increased:
dw ij
~ correlation ( x i , x j )
dt
Associative memory is produced automatically
The Hebb rule can be used for pattern association,
pattern categorization, pattern classification and over a
range of other areas.
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
FEW APPLICATIONS OF NEURAL
NETWORKS
“Principles of Soft Computing, 2nd Edition”
by S.N. Sivanandam & SN Deepa
Copyright 2011 Wiley India Pvt. Ltd. All rights reserved.
