Lecture 2: Multilayer Perceptrons
CS460: Deep Learning
What is a Neural network ?
Neural network is a machine that is
designed to model the way in which
the brain performs a particular task
or function of interest.
To achieve good performance, neural
networks employ a massive
interconnection of simple computing
cells referred to as "neurons" or
"processing units."
Inspired by humans
Thebrain is a highly complex,
nonlinear; and parallel computer. It
has the capability to organize its
neurons, so as to perform certain
computations (e.g., pattern
recognition, perception, and motor
control) many times faster than the
fastest digital computer in existence
today.
Biological Neural
Networks
Dendrites
Synapse
Synapse
Axon
Axon
Dendrites Soma
Soma
Modeling the single
neuron
Learning in simple
neurons
If we have two groups of objects, one
group of several written A's, and the
other of B's, we may want our
neuron to tell the A's from the B's, as
in figure.
We want it to output a 1 when an A
is presented and a 0 when it sees a
B.
Biology analogy
Biological Artificial
Soma Node/neuron
Dendrites Input
Axon Output
Synapse Weight
The perceptron
The simplest kind of neural network is a single-layer
perceptron network, which consists of a single layer
of output nodes; the inputs are fed directly to the
outputs via a series of weights. The sum of the
products of the weights and the inputs is calculated in
each node, and if the value is above some threshold
the neuron fires and takes the activated value;
otherwise it takes the deactivated value.
Neurons with this kind of activation function are also
called artificial neurons or linear threshold units.
In the literature the term perceptron often refers to
networks consisting of just one of these units.
The perceptron (cont’d)
theperceptron is an algorithm for learning
a binary classifier called a
threshold function: a function that maps its
input x(a real-valued vector) to an output
value f(X) (a single binary value):
wherew is a vector of real-valued
weights, w . x is the dot product ,
where m is the number of inputs to the
perceptron, and b is the bias.
The perceptron (cont’d)
Perceptrons can be trained by a simple
learning algorithm that is usually called
the delta rule. It calculates the errors
between calculated output and sample
output data, and uses this to create an
adjustment to the weights, thus
implementing a form of gradient descent.
Single-layer perceptrons are only capable
of learning linearly separable patterns
Linearly Separable
XOR Function
Itis impossible for a single-layer
perceptron network to learn an
XOR function
Non-linear
transformations
A single-layer neural network can
compute a continuous output instead
of a step function. A common choice
is the so-called logistic function:
Non-linear
transformations
The logistic function is one of the
family of functions called
sigmoid functions. It has a
continuous derivative, which allows
it to be used in backpropagation.
This function is also preferred
because its derivative is easily
calculated (differentiable) :
Sigmoid function
Multi Layer Perceptron
(MLP)
MLP is a class of a feedforward (Acyclic) artificial neural
network (ANN).
Each neuron in one layer has directed connections to
the neurons of the subsequent layer. In many
applications the units of these networks apply a
sigmoid function as an activation function.
MLPs models are the most basic deep neural network,
which is composed of a series of fully connected layers.
Each new layer is a set of nonlinear functions of a
weighted sum of all outputs (fully connected) from the
prior one.
Multilayer feed-forward networks, given enough hidden
units and enough training samples, can closely
approximate any function.
The Architecture
MLP with one hiddenx layer
1 (PE)
x2 Weighted Transfer
(PE) Sum Function
Y1
x3 (S) (f)
(PE)
(PE) (PE)
Output
(PE)
Layer
Hidden
(PE)
Layer
Input
Layer
MLP processing
(a) Single neuron (b) Multiple neurons
x1 x1 w11 (PE) Y1
w1
w21
(PE) Y
w1 w12
x2 Y X 1W1 X 2W2
x2 w22 (PE) Y2
PE: Processing Element (or neuron)
Y1 X1W11 X 2W21
w23
Y2 X1W12 X2W22
Y3 X 2W 23 (PE) Y3
MLP processing (cont’d)
Summation function: Y = 3(0.2) + 1(0.4) + 2(0.1) = 1.2
X1 = 3 Transfer function: YT = 1/(1 + e-1.2) = 0.77
W2 = 0.4 Processing Y = 1.2
X2 = 1 YT = 0.77
element (PE)
X3 = 2
Designing the MLP
Before training can begin, the user must decide on
the network topology by specifying:
the number of units in the input layer,
the number of hidden layers (if more than one), the
number of units in each hidden layer, and
the number of units in the output layer.
Normalizing the input values (between 0.0 and 1.0)
for each attribute measured in the training tuples
will help speed up the learning phase and prevent
the exploding gradient problem.
Discrete-valued attributes may be encoded such
that there is one input unit per domain value.
Choice of the transfer function
Transformation (Transfer)
Function
Linear function
Sigmoid (logical activation) function [0
1]
Tangent Hyperbolic function [-1 1]
MLP: Design issues
Neural networks can be used for both
classification (to predict the class label
of a given tuple) and numeric prediction
(to predict a continuous-valued output).
For classification, one output unit may
be used to represent two classes (where
the value 1 represents one class, and
the value 0 represents the other).
If there are more than two classes, then
one output unit per class is used.
MLP: Design issues
There are no clear rules as to the “best”
number of hidden layer units.
Network design is a trial-and-error process and
may affect the accuracy of the resulting trained
network.
The initial values of the weights may also affect
the resulting accuracy.
Once a network has been trained and its
accuracy is not considered acceptable, it is
common to repeat the training process with
a different network topology or
a different set of initial weights.
The XOR function -
revisted
MLP Box Office prediction
example
The Learning algorithm
Itadjusts the weights of the
machine, in order to minimize the
average squared error.
Learning in MLP
The learning algorithm procedure
Initialize weights with random values and set
other network parameters
Read in the inputs and the desired outputs
Compute the actual output (by working
forward through the layers)
Compute the error (difference between the
actual and desired output)
Change the weights by working backward
through the hidden layers
Repeat steps 2-5 until weights stabilize
Learning in MLP (cont’d)
Backpropagation learns by iteratively
processing a data set of training tuples,
comparing the network’s prediction for each
tuple with the actual known target value.
The target value may be the known class label
of the training tuple (for classification
problems) or a continuous value (for numeric
prediction).
For each training tuple, the weights are
modified so as to minimize the mean-squared
error between the network’s prediction and the
actual target value.
Learning in MLP (cont’d)
These modifications are made in the
“backwards” direction (i.e., from the
output layer) through each hidden
layer down to the first hidden layer
(hence the name backpropagation).
Although it is not guaranteed, in
general the weights will eventually
converge, and the learning process
stops.
MLPs Bottlenecks
1. Dimensionality issue
Rule of thumb: The number of
training samples should be at least 5
to 10 times the number of weights in
the network.
Otherwise,the network is prone to
overfitting
2. Overfitting
2. Overfitting (cont’d)
3. The black-box syndrome
A common criticism for ANN: The lack of
transparency/explainability
Answer: sensitivity analysis
Conducted on a trained ANN
The inputs are perturbed while the
relative change on the output is
measured/recorded
Results illustrate the relative importance
of input variables
sensitivity analysis
4. Vanishing gradient
problem
In machine learning, the vanishing gradient problem is
encountered when training artificial neural networks with
gradient-based learning methods and backpropagation. In such
methods, during each iteration of training each of the neural
network's weights receives an update proportional to the
partial derivative of the error function with respect to the current
weight. The problem is that in some cases, the gradient will be
vanishingly small, effectively preventing the weight from changing
its value. In the worst case, this may completely stop the neural
network from further training. As one example of the problem cause,
traditional activation functions such as the hyperbolic tangent
function have gradients in the range (0,1], and backpropagation
computes gradients by the chain rule. This has the effect of
multiplying n of these small numbers to compute gradients of the
early layers in an n-layer network, meaning that the gradient (error
signal) decreases exponentially with n while the early layers train
very slowly.
Building Neural
Networks
Architecture of a neural network is driven
by the task it is intended to address
Classification, regression, clustering,
general optimization, association, ….
Most popular architecture: Feedforward
multi-layered perceptron with
backpropagation learning algorithm
Used for both classification and regression
type problems
Others – Recurrent, self-organizing feature
maps, Hopfield networks, …
Development of NNs
Backpropagation
Multi-layer networks use a variety of learning techniques, the most
popular being back-propagation.
The output values are compared with the correct answer to compute the
value of some predefined error-function. By various techniques, the error
is then fed back through the network.
The algorithm adjusts the weights of each connection in order to reduce
the value of the error function by some small amount.
After repeating this process for a sufficiently large number of training
cycles, the network will usually converge to some state where the error
of the calculations is small.
In this case, one would say that the network has learned a certain target
function. To adjust weights properly, one applies a general method for
non-linear optimization that is called gradient descent. For this, the
network calculates the derivative of the error function with respect to the
network weights, and changes the weights such that the error decreases
(thus going downhill on the surface of the error function).
For this reason, back-propagation can only be applied on networks with
differentiable activation functions.
The steps Of The
Backpropagation
Initialize the weights:
The weights in the network are
initialized to small random numbers
(e.g., ranging from−1.0 to 1.0, or−0.5
to 0.5).
Each unit has a bias associated with it,
as explained later.
The biases are similarly initialized to
small random numbers.
Each training tuple, X, is processed by
the following steps.
Propagate the inputs
forward:
First,the training tuple is fed to the
network’s input layer.
The inputs pass through the input units,
unchanged.
That is, for an input unit, j, its output, Oj, is
equal to its input value, Ij.
Next, the net input and output of each unit in
the hidden and output layers are computed.
The net input to a unit in the hidden or
output layers is computed as a linear
combination of its inputs.
The steps Of The
Backpropagation
Propagate the inputs forward:
Each hidden layer or output layer unit has a
number of inputs to it that are, in fact, the
outputs of the units connected to it in the
previous layer.
Propagate the inputs
forward
To compute the net input to the unit, each input
connected to the unit is multiplied by its
corresponding weight, and this is summed.
Given a unit, j in a hidden or output layer, the net
input, Ij, to unit j is
where wij is the weight of the connection from unit i
in the previous layer to unit j; Oi is the output of
unit i from the previous layer; and θj is the bias of
unit j.
The bias acts as a threshold in that it serves to vary
the activity of the unit.
Propagate the inputs
forward
Each unit in the hidden and output layers takes its net
input and then applies an activation function to it.
The function symbolizes the activation of the neuron
represented by the unit.
The logistic, or sigmoid, function is used. Given the
net input Ij to unit j, then Oj, the output of unit j, is
computed as
Thelogistic function is nonlinear and differentiable,
allowing the backpropagation algorithm to model
classification problems that are linearly inseparable.
Propagate the inputs
forward
We compute the output values, Oj, for
each hidden layer, up to and including
the output layer, which gives the
network’s prediction.
In practice, it is a good idea to cache (i.e.,
save) the intermediate output values at
each unit as they are required again later
when back propagating the error.
This trick can substantially reduce the
amount of computation required.
Back propagate the
error
Theerror is propagated backward by updating
the weights and biases to reflect the error of
the network’s prediction. For a unit j in the
output layer, the error Errj is computed by
where Oj is the actual output of unit j, and Tj is
the known target value of the given training
tuple.
Note that Oj(1−Oj) is the derivative of the
logistic function.
Back propagate the
error
To compute the error of a hidden layer unit j,
the weighted sum of the errors of the units
connected to unit j in the next layer are
considered.
The error of a hidden layer unit j is
where wjk is the weight of the connection from
unit j to a unit k in the next higher layer, and
Errk is the error of unit k.
Back propagate the
error
The weights and biases are updated to reflect the
propagated errors.
Weights are updated by the following equations,
where delta(wij) is the change in weight wij:
The variable l is the learning rate, a constant typically
having a value between 0.0 and 1.0.
The learning rate helps avoid getting stuck at a local
minimum in decision space. If the learning rate is too
small, then learning will occur at a very slow pace. If
learning rate is too large, then oscillation.
Back propagate the
error
Biasesare updated by the following equations, where
delta(θj) is the change in bias θj:
The updating of the weights and biases after the
presentation of each tuple, referred to case updating.
Alternatively, the weight and bias increments could
be accumulated in variables, so that the weights and
biases are updated after all the tuples in the training
set have been presented. (called epoch updating)
Batch/mini-batch updating : weight and bias are
updated after several samples
one iteration through the training set is an epoch.
Terminating condition
Training stops when:
All delta(wij) in the previous epoch are so small
as to be below some specified threshold, or
The percentage of tuples misclassified in the
previous epoch is below some threshold, or
A pre-specified number of epochs has expired.
Inpractice, several hundreds of thousands
of epochs may be required before the
weights will converge.
