Introduction to Neural Networks
Artificial neural networks (ANNs)
Other terms/names
• Connectionist
• Parallel distributed processing
• Neural computation
• Adaptive networks..
History of ANNs
• 1943-McCulloch & Pitts are generally
recognised as the designers of the first
neural network
• 1949-First learning rule
• 1969-Minsky & Papert - perceptron limitation
- Death of ANN
• 1980’s - Re-emergence of ANN - multi-layer
networks
ANN
• Artificial neural networks (ANNs) are
computing systems vaguely inspired by
the biological neural networks
• An ANN is based on a collection of connected
units or nodes called artificial neurons which
loosely model the neurons in a biological brain.
ANN
• Each connection, like the synapses in a
biological brain, can transmit a signal
from one artificial neuron to another.
• An artificial neuron that receives a
signal can process it and then signal
additional artificial neurons connected
to it.
Brain and Machine
• The Brain
– Pattern Recognition
– Association
– Complexity
– Noise Tolerance
• The Machine
– Calculation
– Precision
– Logic
The contrast in architecture
• The Von Neumann architecture uses a
single processing unit;
– Tens of millions of operations per second
– Absolute arithmetic precision
• The brain uses many slow unreliable
processors acting in parallel
Computers vs. Neural Networks
“Standard” Computers Neural Networks
One CPU Highly parallel
processing
Fast processing units Slow processing units
Reliable units Unreliable units
Static infrastructure Dynamic
Why Artificial Neural Networks?
There are two basic reasons why we are interested in building
artificial neural networks (ANNs):
• Technical viewpoint: Some problems such as character
recognition or the prediction of future states of a system
require massively parallel and adaptive processing.
• Biological viewpoint: ANNs can be used to replicate and
simulate components of the human (or animal) brain, thereby
giving us insight into natural information processing.
Biological Neural Network Vs Artificial Neural Network
Artificial Neural Network
Biological Neural Network
Biological Neural Network Artificial Neural Network
Dendrites Inputs
Cell nucleus Nodes
Synapse Weights
Axon Output
ANN
• The “building blocks” of neural networks are the
neurons.
• In technical systems, we also refer to them as units or
nodes.
• Basically, each neuron
receives input from many other neurons.
changes its internal state (activation) based on the
current input.
sends one output signal to many other neurons,
possibly including its input neurons (recurrent
network).
The Architecture of ANN
ANN
• In common ANN implementations, the
signal at a connection between
artificial neurons is a real number, and
the output of each artificial neuron is
computed by some non-linear function
of the sum of its inputs.
• The connections between artificial
neurons are called 'edges'.
ANN
• Artificial neurons and edges typically
have a weight that adjusts as learning
proceeds.
• The weight increases or decreases the
strength of the signal at a connection.
• Typically, artificial neurons are
aggregated into layers.
ANN
• Different layers may perform different
kinds of transformations on their inputs.
• Signals travel from the first layer (the
input layer), to the last layer (the output
layer), possibly after traversing the layers
multiple times.
ANN
Advantages of ANN
• Parallel processing
• Storing data on the entire network
• Capability to work with incomplete
knowledge
• Memory distribution
• Fault tolerance
Disadvantages of ANN
• Hardware Dependence
• Understanding the network’s operation
• Assured network structure
• Difficulty in presenting the issue to the
network
• The network’s lifetime is unknown
Applications ANN
• Image Processing and Character recognition
• Forecasting
• Stock Market Prediction.
• Social Media.
• Aerospace.
• Defence.
• Healthcare.
• Signature Verification and Handwriting Analysis
Example
A four pixel image
Categorize images solid
vertical
diagonal
horizontal
Categorize images solid
vertical
diagonal
horizontal
Categorize images solid
vertical
diagonal
horizontal
Categorize images solid
vertical
diagonal
horizontal
solid
Categorize images
vertical
diagonal
horizontal
Input neurons
Pixel brightness
-1.0 -.75 -.50 -.25 0.0 +.25 +.50 +.75 +1.0
Input vector
.50
0.0
-.75
.75
A neuron
+
Sum all the inputs
.50 .50
0.00
-.75
+ .75
0.0 .50
+
.50
-.75
.75
Weights
.50 .50 x 1.0
1.0 0.00 x 1.0
-.75 x 1.0
1.0
+ .75 x 1.0
0.0 .50
+
1.0
.50
-.75
1.0
.75
Weights
.50 .50 x -.2
-.2 0.00 x 0.0
-.75 x .8
0.0
+ .75 x -.5
0.0 -1.075
+
.8
-1.075
-.75
-.5
.75
Weights
.50 .50 x -.2
-.2 0.00 x 0.0
-.75 x .8
0.0
+ .75 x -.5
0.0 -1.075
+
.8
-1.075
-.75
-.5
.75
Squash the result
.50
0.0
+
1.075 .746
-.75
.75
Sigmoid squashing function
1.0
.5
-2.0 -1.5 -1.0 -.5
.5 1.0 1.5 2.0
-.5
-1.0
Sigmoid squashing function
1.0
.5
-2.0 -1.5 -1.0 -.5
.5 1.0 1.5 2.0
-.5
-1.0
Your number goes in here
Sigmoid squashing function
1.0
.5
-2.0 -1.5 -1.0 -.5
.5 1.0 1.5 2.0
-.5
-1.0
Sigmoid squashing function
The squashed version
comes out here 1.0
.5
-2.0 -1.5 -1.0 -.5
.5 1.0 1.5 2.0
-.5
-1.0
Sigmoid squashing function
1.0
.5
-2.0 -1.5 -1.0 -.5
.5 1.0 1.5 2.0
-.5
-1.0
Sigmoid squashing function
1.0
.5
-2.0 -1.5 -1.0 -.5
.5 1.0 1.5 2.0
-.5
-1.0
the answer stays between -1 and 1.
1.0
.5
-2.0 -1.5 -1.0 -.5
.5 1.0 1.5 2.0
-.5
-1.0
Squash the result
.50
0.0
+
1.075 .746
-.75
.75
Weighted sum-and-squash neuron
.50
0.0
.746
-.75
.75
Make lots of neurons, identical except for
weights
To keep our picture
clear, weights will either
be
1.0 (white)
-1.0 (black) or
0.0 (missing)
Receptive fields get more complex
Repeat for additional layers
Receptive fields get still more complex
Repeat with a variation
Rectified linear units (ReLUs)
1.0
.5
-2.0 -1.5 -1.0 -.5
.5 1.0 1.5 2.0
-.5
If your number is positive, keep it.
Otherwise you get a zero. -1.0
solid
vertical
diagonal
horizontal
solid
vertical
diagonal
horizontal
solid
vertical
diagonal
horizontal
solid
vertical
diagonal
horizontal
solid
vertical
diagonal
horizontal
solid
vertical
diagonal
horizontal
solid
vertical
diagonal
horizontal
Errors truth
0.
solid
vertical
0.
diagonal
0.
horizontal
1.
Errors truth
0.
answer
.5
solid
vertical
0. .75
diagonal
0. -.25
horizontal
1. -.75
Errors error
.5
truth
0.
answer
.5
solid
vertical
.75 0. .75
diagonal
.25 0. -.25
horizontal
1.75 1. -.75
Errors error
.5
truth
0.
answer
.5
solid
vertical
.75 0. .75
diagonal
.25 0. -.25
horizontal
1.75 1. -.75
total 3.25
Learn all the weights: Gradient
descent
Error at:
original weight
Weight
Learn all the weights: Gradient
descent
Error at:
lower weight
original weight
Weight
Learn all the weights: Gradient
descent
Error at:
lower weight
original weight
higher weight
Weight
Numerically calculating the gradient is expensive
Error at:
lower weight
original weight
higher weight
Weight
Calculate the gradient (slope)
directly
Error at:
original weight
Weight
Slope
Error at:
change in
weight = +1
original weight
Weight
Slope
Error at:
change in
weight = +1
original weight
move along
the curve
Weight
Slope
Error at:
change in
weight = +1
original weight
change in
error = -2
Weight
Slope slope = change in error
change in weight
Error at:
change in
weight = +1
original weight
change in
error = -2
Weight
