Artificial Intelligence

Artificial Neural Networks

- Introduction -

Dr. Jwan .S. Raafat

 INTRODUCTION
• Improvements in various calculation and control
application in systems under variable conditions, and
need for their precise and fast operation and also
complexity of the systems mathematical model in
nature requires new and complex calculator and
controller system which have specific characteristics
such as: adaptively, learning ability, non-requirement to
precise mathematical model of system, robustness
against external noise, small and tolerable internal error.
These properties have made intelligent system as
Artificial Neural Network.
 • An ANNs are systems, which are designed similar to
human neural system and gain parallel processing in a
large number of simple process elements.
• Information fed to them as numerous examples, are
distributed and saved in weights between process
elements (neurons) associatively

• By simulating some of the features of biological

networks of neurons, artificial neural networks are able
to analyze data for patterns, and then make
predications on the basis of these patterns.
Biological Neural Network
 Biological inspiration
Dendrites

Soma (cell body)

Axon
 There is a close analogy between the
structure of a biological neuron and the
artificial neuron.
A biological neuron has three components:

 The Dendrites
They receive (from other neuron) an electrical
impulse transmitted across a synaptic gap by means
of a chemical transmitter which modifies the
incoming signals in a manner similar to the action of
the weights in an ANN
  The Cell Body (Soma)
Which sums the incoming signals and when
sufficient input is received it fires (transmits a
signal) over its axon to other cells. A cell either fires
or dose not at any instant of time (so that transmitted
signals can be treated as binary)

 Axon (Commutation links)

The transmission of the signal from a particular neuron
is accomplished by an action potential resulting from
differential concentrations of ions on either side of the
axon sheath .
Biological nerve cell
 What is a Neural Network?

 An Artificial Neural Network (ANN) is an

information processing paradigm that is inspired by the
way biological nervous systems, such as the brain,
process information.

 It is composed of a large number of highly

interconnected processing elements (neurones)
working in unison to solve specific problems.
  These neurons are interacting with each other via
excitatory or inhibitory connections that has a natural
propensity for storing experimental knowledge and
making it available for use

 ANNs, like people, learn by example. An ANN

is configured for a specific application, such as
pattern recognition or data classification, through a
learning process.

 Learning in biological systems involves

adjustments to the synaptic connections that exist
between the neurons.
 The Model of Artificial Neuron:-

Bias
X1 W1k bk

Activation
X2 W2k
Function
YK
Input . 
F (.)
Signals
Output
.
Summing
. Junction
Xn Wnk

Synaptic
Weight
 Sets of inputs are applied, each representing the
output of another neuron. Each input is multiplied by a
corresponding weight, and all of the weighted inputs
are then summed to determine the activation level of
the neurons. If this summation exceeds a certain
threshold, the neuron responds by issuing a new pulse,
which propagates along its output connection.
Otherwise the neuron remains inactive .

 The neuronal model also includes an externally

applied bias, denoted by (bk). The bias (bk) has the
effect of increasing or lowering the net put of the
activation function, depending on whether it is positive
or negative, respectively.
 In Mathematical terms:
n
y k  f (  w jk x j  b k )
j1

Where f (w, x, b) is the activation function.

Several of more common types of activation signal

function are depicted in (Figure .1 below).
 f (x) f (x)

+1 +1
x x
0
-1

Binary
BinaryStep
StepFunction
Function Hard
HardLimiter
LimiterFunction
Function
f (x) f (x)
+1
+1

x
x
0
-1

Log-
Log-Sigmoid
Sigmoidfunction
function Tangent
TangentFunction
Function

Fig(1)
Fig (1)Activation
ActivationSignal
SignalFunctions
Functions
 Some of specific properties of neural networks
responsible for the interest that they arouse today are:

 Parallelism: Parallelism is fundamental in the architecture

of neural networks when the neurons are considered as sets of
elementary units operating simultaneously which allows a
great speed of calculation .

 Simple processors: Each processor (neurons) simply sums

incoming signals and fires when those signals reach a
threshold.
  Adaptive Learning: An ability to learn how to do tasks
based on the data given for training or initial experience.

 Robust performance: ANNs typically are robust as

computing system. They continue to perform well when part of
the network is disabled or presented with noisy data. This is
possible because the “ knowledge “ stored in an ANN is
distributed over many neurons and interconnections, not just a
single or a few units. This aspect of ANN is sometimes called
fault – tolerance .
 Architecture of Neural Networks

The arrangement of neurons into layers and the

connection patterns within and between layers is
called the net architecture.
There are two popular models of neural
networks: the feedback model and feed-forward
model:
 Feed Back Networks
Feed back networks are the type of neural networks
that contain feed back connections, and they are called
“ Recurrent networks “
Recurrent networks can have signals travelling in
both directions by introducing loops in the network.
This allows the network to accommodate unidirectional
data flow, as occurs in biological structures, and
enables networks to distinguish separate input patterns
from the input sequence. Recurrent networks have a
dynamic memory: their outputs at a given instant
reflect the current input as well as previous inputs and
outputs .
  Feed Forward Networks

Feed-forward ANNs allow signals to travel one way

only; from input to output. There is no feedback
(loops) i.e.
the output of any layer does not affect that same layer.
Feed-forward ANNs tend to be straight forward
networks that associate inputs with outputs. They are
extensively used in pattern recognition .

Feed forward networks can classify into :

 1. Single – Layer Feed Forward Networks

A Single layer net has one layer of connection

weights. Often the units can be distinguished as input
unit (which receives signals from the outside world
and distributes them to other processing elements of
the network. For this reason the first layer is not
include in the layer count, since it is not performing
the computation work ) and output unit (from which
the response of the net can be read).
 2. Multilayer Feed Forward Networks
Larger and more complex networks generally
offer greater computational capabilities. Multilayers
feed forward networks (commonly referred to as
multilayer preceptors) are an important class of neural
networks. MLPs have been applied successfully to
solve some difficult and diverse problems in many
disciplines of science and technology .

MLPs distinguishes itself by the presence of one or

more hidden layers, whose computation nodes are
called hidden neurons or hidden units.
 The function of hidden neurons is to intervene
between the external input and the network output in
some useful manner.

Multilayer networks may be formed by simply parallel

a group of single layers, the output of one layer
provides the input to the subsequent layer. Figure (2)
shows such a network.

The main method of training MLPs is the error back

propagation, which was recognized around 1985 by
several groups of scientist’s .
 W111 W211

W221
W121

W131

Input W213 Output

W122
Signals Response
W223

. .
W143
. .
. W224
.
W143

.
. W225
W153
.
W235

Input Layer Hidden Layer Output Layer

Fig(2)
Fig (2)MLFF
MLFFNetwork
Network. .
NEURAL NETWORK LEARNING
The artificial neural networks learning ability is its
most intriguing property. Like the biological system,
networks modify themselves as a result of experience
to produce a more desirable behavior pattern.
Adjusting the weights is commonly called “ training ”
and the network is said to “ learn “.
Training is accomplished by sequentially applying input
vector, while adjusting network weights according to a
pre-determined procedure. During training, the network
weights gradually converge to values such that each
input vector produces the desired output vector.
Learning can be defined as:
  Supervised Learning
A teacher is assumed to be present during the
learning process and each example pattern used
to train the network includes an input pattern
together with the a target or desired output
pattern, the correct answer.
During the learning process, a comparison can be made
between the computed output by the network and the
correct output to determine the error. The error can be
used to change network parameters, which result in an
improvement in performance of network .

 Unsupervised Learning
The network has no feedback on the desired or
correct output. There is no teacher to present target
patterns. Its also referred to as self- organization, its
expected to organize itself into some useful
configuration, and modifies the weights to produce
output vectors that are consistent .
Methods of Learning:-
Hebbian learning rule.
Perceptron layer network.
Widrow-Hoff (Adaline)
Delta learning rule
Winner
EBP
The Error Back Propagation
The error back propagation (EBP) method
is the most effective and most widely used
learning method for the training of
multilayered neural networks (MLP), that
uses differentiable activation functions and
supervised training. One of the most
common activation functions used in EBP
is the sigmoid function and it’s defined as:


1
f ( x) 
1  exp(x )

f ( x )  f ( x )1  f ( x ) 
'

where the constant (  ) determine the steepness of the rise.

Like the delta rule, the EBP is an

optimization procedure based on gradient
descendent that adjusts weights to reduce
the system error. The name back
propagation arises from the method in
which corrections are made to the weights.
 In the learning phase input patterns are
represented to the network in some
sequence.
Each training pattern is propagated forward
layer by layer until an output is computed.
The computed output is then compared to a
desired or target output (supervised
training) and an error value is determined.
The errors are used as input to feed back
connections from which adjustments are
made to the synaptic weights in a back ward
direction.
 Using EBP, the hidden layer weights are
adjusted using the errors from the
subsequent layer.
Hence the EBP starts with computing at the
output layer the error term, then this term is
used to correct the weights of the last
hidden layer and the output layer. Like
wise, an error value computed from the last
hidden layer outputs is used to adjust the
weights connections in the next to the last
hidden layer and so on until the weight
connections to the first hidden layer are
adjusted.
 Therefore the EBP is an iterative gradient
algorithm designed to minimize the mean
square error between the output of the
multilayer neural network and the desired
output .
EBP algorithm has been enhanced by
adding several choices of design; such as
number of layers, variable learning rate and
momentum term, with the addition of error
approximation for non linear system. Those
enhancements with no doubt have increased
the efficiency and the convergence speed of
the algorithm .
EBP Algorithm
Initialize network weight values .
Repeat the following steps until some
criterion is reached: (for each training
pair).
Sums weighted input and apply activation
function to compute output of hidden
layer.
h j  f (  ix i w ij )
 Sums weighted output of hidden layer and
apply activation function to compute output
of output layer.
y k  f (  j h j w jk )
Compute back propagation error.
 k  (d k  y k )f (  j h j w jk )
'

Calculate weight correction term.

w jk (n )   k h j  w jk (n  1)
 Sums delta input for each hidden unit and
calculate error term.
 j   k  k w jk f (  ix i w ij )
'

Calculate weight correction term.

w ij (n )   j x i  w ij (n  1)
Update weights.
w jk (n)  w jk (n  1)  w jk (n )

w ij (n )  w ij (n  1)  w ij (n )
   (d
2
SSE  p
k y ) p
k
p k

p : is the number of input patterns.

k : is the number of output neuron.

 if SSE 10-4

Stop.

These steps are illustrated in Figure (3)

 Inputs Forward Pass Back Pass

X 1 X 2 X 3 Xi
X i w ij (n )  w ij (n  1)   w ij (n )

w ij ( n )   j x i
 w ij (n  1 )
w ij

 j
 f ' (  ix i w ij
) k
( k w jk
)

h1 h2 hj h j
 f (  ix i w ij )

w jk
(n )  w jk
(n  1)   w jk
(n )

w jk
w jk ( n )   k h j
 w jk (n  1)

y k
 f ( j h jw jk )
  (d  y ) f ' (  jh j w )
y
k k k jk

1 y k

Outputs

Fig (3) EBP in MLP .

APPLICATION OF ANNs
An ANN is configured for a specific
application, such as pattern recognition (e.g.
speech and visual image recognition),
associative recall, classification, data
compression, modeling, and adaptive control
.
In recent years there have been a lot of
interesting applying ANN in power
engineering field. These applications are:
 Neural network is used to estimate the
critical fault clearing time (CCT), which is
constitutes a complex function of pre- fault
system condition, fault type and location,
and protective relaying strategy. Feed
forward net with error back- propagation
learning scheme is described and was
implemented .
Neural network was applied for fault
diagnosis. The inputs of the neural network
are relays, signals from circuit breaker, load
flow information and alarm records.
 The output of neural network is the bits of a binary
string that encodes the fault.

A Neural network is developed to estimate the

short term and long term load forecasting that
consists of recognizing load patterns in
historical load series. The general idea in an
intelligent load forecasting system is to train an
ANN with past load data and other related data
such as temperature, forecasting period and
type of the day. The output layer contains a
single element, which corresponds to the
forecasted value .
 Fast-decoupled load flow method based on
MLP model with real and reactive load
demand at load buses as inputs. The output
nodes provided the corresponding values of
voltage magnitudes and angles at all the P-Q
buses .

Artificial Neural Network employs to control

single synchronous generator operating on to
an infinite bus bar, during the transient
period caused by a disturbance. On-line feed
forward operations of the trained ANNs are
simulated .
 MLP intelligent estimates the states of the
test distribution system of two series
measured data and is a good tool for
distribution management and control system.

artificial neural network is used to control the

multi-tap capacitors installed in radial
distribution system for varying the load
profile so that the system losses are to be
minimized.
 MLP network model has been proposed to
reconfigure the network for loss minimization
in distribution systems. The model is designed
to predict the switching status of dynamic
switches in the optimal configuration for
various load conditions. Training sets for the
ANN are generated by varying the constant P-
Q load models and carrying out the off-line
network reconfiguration simulations .