Mechatronics Engineering and Automation Program

CSE488: Computational Intelligence

Lab#07: Introduction to Neural Networks and Tensorflow

Introduction to Neural Networks and TensorFlow

August 19, 2020

1 Introduction to Neural Networks

1.1 Neurons
The human brain consists of neurons, which are the basic building blocks of the nervous system. A neuron
consists of a cell body, or soma, a single axon, and dendrites. Neurons are connected to one another by
the dendrites and axon terminals. A signal from one neuron is passed to the axon terminal and dendrites
of another connected neuron, which receives it and passes it through the soma, axon, and terminal, and
so on.

1.2 Artificial Neuronal Networks

An artificial neural network tries to mimic the brain at its most basic level, i.e., that of the neuron. An
artificial neuron has a similar structure to that of a human neuron and comprises the following sections:

1
CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

• Input: This layer is similar to dendrites and takes input from other networks/neurons.
• Summation: This layer functions like the soma of neurons. It aggregates the input signal received.
• Activation: This layer is also similar to a soma, and it takes the aggregated information and fires a
signal only if the aggregated input crosses a certain threshold value. Otherwise, it does not fire.
• Output: This layer is similar to axon terminals in that it might be connected to other neu-
rons/networks or act as a final output layer (for predictions).

1.3 Forward and Backward Propagation

In a fully connected neural network, when the inputs pass through the neurons (hidden layer to output
layer), and the final value is calculated at the output layer, we say that the inputs have forward prop-
agated. Consider, for example, a fully connected neural network with two inputs, X1 and X2, and one
hidden layer with three neurons and an output layer with a single output Yp (numeric value)

The inputs will be fed to each of the hidden layer neurons, by multiplying each input value with a weight
(W) and summing them with a bias value (b). So, the equations at the neurons’ hidden layer will be as

2
CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

follows:
H1 = W1 ∗ X1 + W4 ∗ X2 + b1
H2 = W2 ∗ X1 + W5 ∗ X2 + b2
H3 = W3 ∗ X1 + W6 ∗ X2 + b3

The values H1 , H2 , and H3 will be passed to the output layer, with weights W7 , W8 , and W9 , respectively.
The output layer will produce the final predicted value of Y

Yp = W7 ∗ H1 + W8 ∗ H2 + W9 ∗ H3

As the input data (X1 and X2) in this network flows in a forward direction to produce the final outcome,
Yp, it is said to be a feed forward network, or, because the data is propagating in a forward manner, a
forward propagation. Now, suppose the actual value of the output is known (denoted by Y). In this case,
we can calculate the difference between the actual value and the predicted value, i.e., L = (Y − Yp)2 ,
where L is the loss value. To minimize the loss value, we will try to optimize the weights accordingly, by
taking a derivate of the loss function to the previous weights,

For example, if we have to find the rate of change of loss function as compared to W7 , we would take a
derivate of the Loss function to that of W7 (d L /dW7 ), and so on. As we can see from the preceding diagram,
the process of taking the derivates is moving in a backward direction, that is, a backward propagation is
occurring. There are multiple optimizers available to perform backward propagation, such as stochastic
gradient descent (SGD), AdaGrad, among others.

3
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

1.4 TensorFlow: Data Flow Graph

TensorFlow is based entirely on the structuring and use of graphs and on the flow of data through it,
exploiting them in such a way as to make mathematical calculations. The graph created internally to
the TensorFlow runtime system is called Data Flow Graph and it is structured in runtime according to
the mathematical model that is the basis of the calculation you want to perform. In fact, Tensor Flow
allows you to define any mathematical model through a series of instructions implemented in the code.
TensorFlow will take care of translating that model into the Data Flow Graph internally.
The nodes of the Data Flow Graph represent mathematical operations, while the edges of the graph
represent tensors (multidimensional data arrays). The name TensorFlow derives from the fact that these
tensors represent the flow of data through graphs, which can be used to model artificial neural networks.

[1]: import numpy as np

import matplotlib.pyplot as plt
import tensorflow as tf

[2]: import tensorflow.compat.v1 as tf

tf.disable_v2_behavior()

WARNING:tensorflow:From F:\Users\Yehia\anaconda3\envs\py3\lib\site-
packages\tensorflow_core\python\compat\v2_compat.py:88:
disable_resource_variables (from tensorflow.python.ops.variable_scope) is
deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term

4
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

1.5 The Model and Sessions in TensorFlow

Before starting to program it is important to understand the internal operation of TensorFlow, including
how it interprets the commands in Python and how it is executed internally. TensorFlow works through
the concept of model and sessions, which define the structure of a program with a certain sequence of
commands. At the base of any TensorFlow project there is a model that includes a whole series of vari-
ables to be taken into consideration and that will define the system. Variables can be defined directly or
parameterized through mathematical expressions on constants.

[3]: c = tf.constant(2,name='c')
x = tf.Variable(3,name='x')
y = tf.Variable(c*x,name='y')

WARNING:tensorflow:From F:\Users\Yehia\anaconda3\envs\py3\lib\site-
packages\tensorflow_core\python\ops\variables.py:2826:
Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated
and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in
eager and graph (inside tf.defun) contexts.
But now if you try to see the internal value of y (you expect a value of 6) with the print() function, you
will see that it will give you the object and not the value.

[4]: print(x)
print(y)

<tf.Variable 'x:0' shape=() dtype=int32_ref>

<tf.Variable 'y:0' shape=() dtype=int32_ref>
In fact, you have defined variables belonging to the TensorFlow Data Flow Graph, that is a graph with
nodes and connections that represent your mathematical model. You will see later how to access these
values via sessions.
Placeholders allow you to build the graph corresponding to the neural network, and to create operations
inside without absolutely knowing the data to be calculated. In fact, you can build the structure of the
graph (and also the neural network).
In practical cases, given a training set consisting of the value to be analyzed x (a tensor) and an expected
value y (a tensor), you will define two placeholders (x & y), with the tf.placeholder() function.

[5]: X = tf.placeholder("int32")
Y = tf.placeholder("int32")

Once you have defined all the variables involved, i.e., you have defined the mathematical model at the
base of the system, you need to perform the appropriate processing and initialize the whole model with
the tf.global_variables_initializer() function.
[6]: model = tf.global_variables_initializer()

5
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

Now that you have a model initialized and loaded into memory, you need to start doing the calcula-
tions, but to do that you need to communicate with the TensorFlow runtime system. For this purpose a
TensorFlow session is created, during which you can launch a series of commands to interact with the
underlying graph corresponding to the model you have created. You can create a new session with the
tf.Session() constructor. Within a session, you can perform the calculations and receive the values of
the variables obtained as results, i.e., you can check the status of the graph during processing.
You have already seen that the operation of TensorFlow is based on the creation of an internal graph
structure, in which the nodes are able to perform processing on the flow of data inside tensors that follow
the connections of the graph. So when you start a session, in practice you do nothing but instantiate this
graph.
A session has two main methods: - session.extend() allows you to make changes to the graph during
the calculation, such as adding new nodes or connections. - session.run() launches the execution of the
graph and allows you to obtain the results in output.
Since several operations are carried out within the same session, it is preferred to use the construct with:
with all calls to methods inherent to it. In this simple case, you simply want to see the values of the
variables defined in the model and print them on the terminal.

[7]: with tf.Session() as session:

session.run(model)
print(session.run(y))

1.6 Tensors
The basic element of the TensorFlow library is the tensor. In fact, the data that follow the flow within the
Data Flow Graph are tensors

6
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

A tensor is identified by three parameters:

• rank: Dimension of the tensor (a matrix has rank 2, a vector has rank 1)
• shape: Number of rows and columns (e.g. (3.3) is a 3x3 matrix)
• type: Type of tensor elements.
Tensors are nothing more than multidimensional arrays. In previous chapters, you saw how easy it is to
get them thanks to the NumPy library.

[8]: t = np.arange(9).reshape((3,3))
print(t)

[[0 1 2]
[3 4 5]
[6 7 8]]
Now you can convert this multidimensional array into a TensorFlow tensor very easily using the
tf.convert_to_tensor() function, which takes two parameters. The first parameter is the array t that
you want to convert and the second is the type of data you want to convert it to, in this case int32.

[9]: tensor = tf.convert_to_tensor(t, dtype=tf.int32)

with tf.Session() as sess:
print(sess.run(tensor))

[[0 1 2]
[3 4 5]
[6 7 8]]
But tensors can be built directly from TensorFlow, without using the NumPy library. There are a number
of functions that make it possible to enhance the tensors quickly and easily.

[10]: t0 = tf.zeros((3,3),'float64')
t1 = tf.random_uniform((3, 3), minval=0, maxval=1, dtype=tf.float32)
with tf.Session() as session:
print(session.run(t0))
print(session.run(t1))

[[0. 0. 0.]
[0. 0. 0.]
[0. 0. 0.]]
[[0.22495162 0.35007727 0.53897154]
[0.23294413 0.8983315 0.16192901]
[0.3974179 0.2803402 0.08716547]]

[11]: t2 = tf.random_normal((3, 3), mean=0, stddev=3, dtype=tf.float32)

sum = tf.add(t1,t2)
mul = tf.matmul(t1,t2)

7
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

with tf.Session() as session:

print(session.run(sum))
print(session.run(mul))

[[-2.4435725 -5.337214 -2.4221702]

[-3.2562509 -2.0436409 2.5152516]
[ 6.0670414 -2.9810224 11.062997 ]]
[[-3.7746558 1.375079 -0.50169706]
[-1.7163159 0.41153982 -0.49753755]
[-2.9293296 -0.3637664 -0.7236777 ]]

1.6.1 Building Single Layer Perceptron

Data to be analyzed The set of data that you will study is a set of 11 points distributed in a Cartesian
axis divided into two classes of membership. The first six belong to the first class, the other five to the
second. The coordinates (x, y) of the points are contained within a numpy inputX array, while the class
to which they belong is indicated in inputY. This is a list of two-element arrays, with an element for each
class they belong to. The value 1 in the first or second element indicates the class to which it belongs. If
the element has value [1,0], it will belong to the first class. If it has value [0,1], it belongs to the second
class. The fact that they are float values is due to the optimization calculation of deep learning. You will
see later that the test results of the neural networks will be floating numbers, indicating the probability
that an element belongs to the first or second class.
Suppose, for example, that the neural network will give you the result of an element that will have the
following values: [0.910, 0.090] This result will mean that the neural network considers that the element
under analysis belongs to 91% to the first class and to 9% to the second class. You will see this in practice
at the end of the section, but it was important to explain the concept to better understand the purpose of
some values.
[12]: #Training set
inputX = np.array([[1.,3.],[1.,2.],[1.,1.5],[1.5,2.],[2.,3.],[2.5,1.5]
,[2.,1.],[3.,1.],[3.,2.],[3.5,1.],[3.5,3.]])
inputY = [[1.,0.]]*6+ [[0.,1.]]*5
print(inputX.shape)
print(inputY)

(11, 2)
[[1.0, 0.0], [1.0, 0.0], [1.0, 0.0], [1.0, 0.0], [1.0, 0.0], [1.0, 0.0], [0.0,
1.0], [0.0, 1.0], [0.0, 1.0], [0.0, 1.0], [0.0, 1.0]]
To better see how these points are arranged spatially and which classes they belong to, there is no better
approach than to plot everything with matplotlib.

[13]: import matplotlib.pyplot as plt

yc = [0]*6 + [1]*5
print(yc)

8
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

plt.scatter(inputX[:,0],inputX[:,1],c=yc, s=50, alpha=0.9)

plt.show()

[0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1]

The SLP Model Definition If you want to do a deep learning analysis, the first thing to do is define
the neural network model you want to implement. So you will already have in mind the structure to be
implemented, how many neurons and layers and compounds (in this case only one), the weight of the
connections, and the cost function to be applied.
Following the TensorFlow practice, you can start by defining a series of parameters necessary to charac-
terize the execution of the calculations during the learning phase. The learning_rate is a parameter that
regulates the learning speed of each neuron. This parameter is very important and plays a very important
role in regulating the efficiency of a neural network during the learning phase. Another parameter to be
defined is training_epochs. This defines how many epochs (learning cycles) will be applied to the neural
network for the learning phase. During program execution, it will be necessary in some way to monitor
the progress of learning and this can be done by printing values on the terminal. You can decide how
many epochs you will have to display a printout with the results, and insert them into the display_step
parameter. A reasonable value is every 50 or 100 steps.

[14]: learning_rate = 0.01

training_epochs= 2000
display_step = 50

To make the implemented code reusable, it is necessary to add parameters that specify the number of

9
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

elements that make up the training set, and how many batches must be divided. In this case you have a
small training set of only 11 items. So you can use them all in one batch. Finally, you can add two more
parameters that describe the size and number of classes to which the incoming data belongs.

[15]: n_samples = 11
batch_size = 11
total_batch = int(n_samples/batch_size)

n_input = 2 #size data input (# size of each element of x)

n_classes = 2 # n of classes

Now that you have defined the parameters of the method, let’s move on to building the neural network.
First, define the inputs and outputs of the neural network through the use of placeholders.

[16]: # tf Graph input

x = tf.placeholder("float", [None, n_input])
y = tf.placeholder("float", [None, n_classes])

Now that you have defined the placeholders, occupied with the weights and the bias, which, as you saw,
are used to define the connections of the neural network. These tensors W and b are defined as variables
by the constructor Variable() and initialized to all zero values with tf.zeros().

[17]: # Set model weights

W = tf.Variable(tf.zeros([n_input, n_classes]))
b = tf.Variable(tf.zeros([n_classes]))

The variables weight and bias you have just defined will be used to define the evidence x ∗ W + b, which
characterizes the neural network in mathematical form. The tf.matmul() function performs a multipli-
cation between tensors x ∗ W, while the tf.add() function adds to the result the value of bias b.
[18]: evidence = tf.add(tf.matmul(x, W), b)

From the value of the evidence, you can directly calculate the probabilities of the output values with the

10
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

tf.nn.softmax() function.
[19]: y_ = tf.nn.softmax(evidence)

Continuing with the construction of the model, now you must think about establishing the rules for the
minimization of these parameters and you do so by defining the cost (or loss). In this phase you can
choose many functions; one of the most common is the mean squared error loss.

[20]: cost = tf.reduce_sum(tf.pow(y-y_,2))/ (2 * n_samples)

But you can use any other function that you think is more convenient. Once the cost (or loss) function
has been defined, an algorithm must be established to perform the minimization at each learning cycle
(optimization). You can use the tf.train.GradientDescentOptimizer() function as an optimizer that
bases its operation on the Gradient Descent algorithm.

[21]: optimizer =tf.train.GradientDescentOptimizer(learning_rate=learning_rate).

,→minimize(cost)

With the definition of the cost optimization method (minimization), you have completed the definition of
the neural network model. Now you are ready to begin to implement its learning phase.

Learning Phase Before starting, define two lists that will serve as a container for the results obtained
during the learning phase. In avg_set you will enter all the cost values for each epoch (learning cy-
cle), while in epoch_set you will enter the relative epoch number. These data will be useful at the end
to visualize the cost trend during the learning phase of the neural network, which will be very use-
ful for understanding the efficiency of the chosen learning method for the neural network. Then be-
fore starting the session, you need to initialize all the variables with the function you’ve seen before,
tf.global_variables_initializer().
[22]: avg_set = []
epoch_set=[]
init = tf.global_variables_initializer()

[23]: with tf.Session() as sess:

sess.run(init)
for i in range(training_epochs):
sess.run(optimizer, feed_dict = {x: inputX, y: inputY})
if i % display_step == 0:
c = sess.run(cost, feed_dict = {x: inputX, y: inputY})
print("Epoch:", '%04d' % (i), "cost=", "{:.9f}".format(c))
avg_set.append(c)
epoch_set.append(i + 1)
print("Training phase finished")

#Save Model Parameters for next sessions

W = W.assign(sess.run(W))
b = b.assign(sess.run(b))

11
CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

training_cost = sess.run(cost, feed_dict = {x: inputX, y:inputY})

print("Training cost =", training_cost, "\nW=", sess.run(W),"\nb=", sess.run(b))
last_result = sess.run(y_, feed_dict = {x:inputX})
print("Last result =",last_result)

Epoch: 0000 cost= 0.249360308

Epoch: 0050 cost= 0.221041128
Epoch: 0100 cost= 0.198898256
Epoch: 0150 cost= 0.181669697
Epoch: 0200 cost= 0.168204829
.
.
.
Epoch: 1650 cost= 0.094320126
Epoch: 1700 cost= 0.093779817
Epoch: 1750 cost= 0.093265593
Epoch: 1800 cost= 0.092775457
Epoch: 1850 cost= 0.092307687
Epoch: 1900 cost= 0.091860712
Epoch: 1950 cost= 0.091433086
Training phase finished
Training cost = 0.09103152
W= [[-0.7092779 0.70927787]
[ 0.6299925 -0.6299924 ]]
b= [ 0.34513068 -0.34513065]
Last result = [[0.9548542 0.04514585]
[0.85713255 0.14286745]
[0.76163834 0.23836163]
[0.7469474 0.2530526 ]
[0.83659446 0.16340552]
[0.2756484 0.7243516 ]
[0.29175714 0.7082429 ]
[0.09067497 0.909325 ]
[0.26010242 0.73989755]
[0.04676623 0.9532338 ]
[0.3787802 0.6212199 ]]
Now that the learning phase is over, it is useful to print a summary table on the terminal that shows
you the trend of the cost during it. You can do this thanks to the values contained in the avg_set and
epoch_set lists that you filled during the learning process.
As you can see, the cost is gradually improving during the epoch, up to a value of 0.168. Then it is
interesting to see the values of the W weights and the bias of the neural network. These values represent
the parameters of the model, i.e., the neural network instructed to analyze this type of data and to carry
out this type of classification.
These parameters are very important, because once they are obtained and knowing the structure of the
neural network used, it will be possible to reuse them anywhere without repeating the learning phase.

12
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

Do not consider this example that takes only a minute to do the calculation; in real cases it may take days
to do it, and often you have to make many attempts and adjust and calibrate the different parameters
before developing an efficient neural network that is very accurate at class recognition, or at performing
any other task.

[24]: plt.plot(epoch_set,avg_set,'o',label = 'SLP Training phase')

plt.ylabel('cost')
plt.xlabel('epochs')
plt.legend()
plt.show()

Now you can move on to see the results of the classification during the last step of the learning phase.

[25]: yc = last_result[:,1]
plt.scatter(inputX[:,0],inputX[:,1],c=yc, s=50, alpha=1)
plt.show()

13
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

The color ranging from blue (belonging to 100% to the first group) to yellow (belonging to 100% to the
second group). As you can see, the division in the two classes of the points of the training set is quite
optimal, with an uncertainty for the four points on the central diagonal (green).

Test Phase and Accuracy Calculation Now that you have an educated neural network, you can create
the evaluations and calculate the accuracy. First you define a testing set with elements different than the
training set. For convenience, these examples always use 11 elements.

[26]: #Testing set

testX = np.array([[1.,2.25],[1.25,3.],[2,2.5],[2.25,2.75],[2.5,3.],
[2.,0.9],[2.5,1.2],[3.,1.25],[3.,1.5],[3.5,2.],[3.5,2.5]])
testY = [[1.,0.]]*5 + [[0.,1.]]*6
print(testX)
print(testY)

[[1. 2.25]
[1.25 3. ]
[2. 2.5 ]
[2.25 2.75]
[2.5 3. ]
[2. 0.9 ]
[2.5 1.2 ]
[3. 1.25]
[3. 1.5 ]
[3.5 2. ]

14
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

[3.5 2.5 ]]
[[1.0, 0.0], [1.0, 0.0], [1.0, 0.0], [1.0, 0.0], [1.0, 0.0], [0.0, 1.0], [0.0,
1.0], [0.0, 1.0], [0.0, 1.0], [0.0, 1.0], [0.0, 1.0]]

[27]: yc = [0]*5 + [1]*6

print(yc)
plt.scatter(testX[:,0],testX[:,1],c=yc, s=50, alpha=0.9)
plt.show()

[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]

[28]: inputX

[28]: array([[1. , 3. ],
[1. , 2. ],
[1. , 1.5],
[1.5, 2. ],
[2. , 3. ],
[2.5, 1.5],
[2. , 1. ],
[3. , 1. ],
[3. , 2. ],
[3.5, 1. ],
[3.5, 3. ]])

15
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

[29]: with tf.Session() as sess:

print(sess.run(W))

[[-0.7092779 0.70927787]
[ 0.6299925 -0.6299924 ]]

[30]: with tf.Session() as sess:

W = sess.run(W)
b = sess.run(b)
pred = tf.nn.softmax(evidence)
result = sess.run(pred, feed_dict = {x: testX})
correct_prediction = tf.equal(tf.argmax(pred, 1),tf.argmax(testY, 1))
# Calculate accuracy
accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
print("Accuracy:", accuracy.eval({x: testX, y: testY}))

Accuracy: 1.0
Apparently, the neural network was able to correctly classify all 11 samples. It displays the points on the
Cartesian plane with the same system of color gradations ranging from dark blue to yellow.

[31]: yc = result[:,1]
plt.scatter(testX[:,0],testX[:,1],c=yc, s=50, alpha=1)
plt.show()

The results can be considered optimal, given the simplicity of the model used and the small amount of

16
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

data used in the training set. Now you will face the same problem with a more complex neural network,
the Perceptron Multi Layer.

1.6.2 Multi Layer Perceptron (with One Hidden Layer)

In this section, you will deal with the same problem as in the previous section, but using an MLP (Multi
Layer Perceptron) neural network.A neural network MLP differs from a SLP neural network in that it can
have one or more hidden layers. Therefore, you will write parameterized code that allows you to work in
the most general way possible, establishing at the time of definition the number of hidden layers present
in the neural network and how many neurons they are composed of. Define two new parameters that
define the number of neurons present for each hidden layer. The n_hidden_1 parameter will indicate how
many neurons are present in the first hidden layer, while n_hidden_2 will indicate how many neurons
are present in the second hidden layer. To start with a simple case, you will start with an MLP neural
network with only one hidden layer consisting of only two neurons. Let’s comment on the part related to
the second hidden layer.

[32]: # Network Parameters

n_hidden_1 = 2 # 1st layer number of neurons
#n_hidden_2 = 0 # 2nd layer number of neurons
n_input = 2 # size data input
n_classes = 2 # classes

learning_rate = 0.001
training_epochs = 2000
display_step = 50
n_samples = 11
batch_size = 11
total_batch = int(n_samples/batch_size)

# tf Graph input
X = tf.placeholder("float", [None, n_input])
Y = tf.placeholder("float", [None, n_classes])

Now you have to deal with the definition of the various W and bias b weights for the different connections.
The neural network is now much more complex, having several layers to take into account. An efficient
way to parameterize them is to define them as follows, commenting out the weight and bias parameters
for the second hidden layer (only for the MLP with one hidden layer as this example).

[33]: # Store layers weight & bias

weights = {
'h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])),
#'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
'out': tf.Variable(tf.random_normal([n_hidden_1, n_classes]))
}
biases = {'b1': tf.Variable(tf.random_normal([n_hidden_1])),
#'b2': tf.Variable(tf.random_normal([n_hidden_2])),
'out': tf.Variable(tf.random_normal([n_classes]))

17
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

To create a neural network model that takes into account all the parameters you’ve specified dynamically,
you need to define a convenient function, which you’ll call multilayer_perceptron()

[34]: # Create model

def multilayer_perceptron(x):
layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
#layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])
# Output fully connected layer with a neuron for each class
out_layer = tf.matmul(layer_1, weights['out']) + biases['out']
return out_layer

The next steps are done are the previous example:

[35]: evidence = multilayer_perceptron(X)

y_ = tf.nn.softmax(evidence)

# Define cost and optimizer

cost = tf.reduce_sum(tf.pow(Y-y_,2))/ (2 * n_samples)
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
init = tf.global_variables_initializer()

[36]: #Training set

inputX = np.array([[1.,3.],[1.,2.],[1.,1.5],[1.5,2.],[2.,3.],[2.5,1.5]
,[2.,1.],[3.,1.],[3.,2.],[3.5,1.],[3.5,3.]])
inputY = [[1.,0.]]*6+ [[0.,1.]]*5

[37]: #Testing set

testX = np.array([[1.,2.25],[1.25,3.],[2,2.5],[2.25,2.75],[2.5,3.],
[2.,0.9],[2.5,1.2],[3.,1.25],[3.,1.5],[3.5,2.],[3.5,2.5]])
testY = [[1.,0.]]*5 + [[0.,1.]]*6

Training and Testing

[38]: avg_set = []
epoch_set = []
init = tf.global_variables_initializer()

[39]: with tf.Session() as sess:

sess.run(init)

for epoch in range(training_epochs):

avg_cost = 0.
# Loop over all batches
for i in range(total_batch):

18
CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

#batch_x, batch_y = inputdata.next_batch(batch_size) TO BE IMPLEMENTED

batch_x = inputX
batch_y = inputY
_, c = sess.run([optimizer, cost], feed_dict={X: batch_x, Y: batch_y})
# Compute average loss
avg_cost += c / total_batch
if epoch % display_step == 0:
print("Epoch:", '%04d' % (epoch+1), "cost={:.9f}".format(avg_cost))
avg_set.append(avg_cost)
epoch_set.append(epoch + 1)
print("Training phase finished")

last_result = sess.run(y_, feed_dict = {X: inputX})

training_cost = sess.run(cost, feed_dict = {X:inputX, Y: inputY})
print("Training cost =", training_cost)
print("Last result =", last_result)

result = sess.run(y_, feed_dict = {X: testX})

correct_prediction = tf.equal(tf.argmax(y_, 1),tf.argmax(testY, 1))
# Calculate accuracy
accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
print("Accuracy:", accuracy.eval({X: testX, Y: testY}))

Epoch: 0001 cost=0.322610110

Epoch: 0051 cost=0.226225898
Epoch: 0101 cost=0.205549985
Epoch: 0151 cost=0.196315512
Epoch: 0201 cost=0.187209174
Epoch: 0251 cost=0.178294659
.
.
.
Epoch: 1651 cost=0.079582952
Epoch: 1701 cost=0.078789726
Epoch: 1751 cost=0.078031659
Epoch: 1801 cost=0.077309571
Epoch: 1851 cost=0.076624788
Epoch: 1901 cost=0.075978816
Epoch: 1951 cost=0.075373009
Training phase finished
Training cost = 0.074808605
Last result = [[0.9948355 0.00516454]
[0.9659619 0.03403818]
[0.91591376 0.08408617]
[0.89051586 0.10948414]
[0.9405608 0.05943919]

19
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

[0.20411345 0.7958866 ]
[0.25564632 0.74435365]
[0.02743917 0.9725609 ]
[0.16072556 0.8392744 ]
[0.00802145 0.9919785 ]
[0.27143803 0.728562 ]]
Accuracy: 1.0

1.7 High-level implementation using Keras

[40]: import tensorflow as tf

from tensorflow import keras as ks

1.7.1 Sample data

[41]: inputX = np.array([[1.,3.],[1.,2.],[1.,1.5],[1.5,2.],[2.,3.],[2.5,1.5]

,[2.,1.],[3.,1.],[3.,2.],[3.5,1.],[3.5,3.]])
targetO1 = np.hstack((np.ones(6),np.zeros(5)))
targetO2 = np.hstack((np.zeros(6),np.ones(5)))
inputY = np.stack((targetO1, targetO2),axis=1).astype('int32')
print(inputY.shape)

(11, 2)

1.7.2 Model definition

[42]: nn_model = ks.models.Sequential()
nn_model.add(tf.keras.Input(shape=(2,)))
nn_model.add(ks.layers.Dense(2, activation='softmax'))

WARNING:tensorflow:From F:\Users\Yehia\anaconda3\envs\py3\lib\site-
packages\tensorflow_core\python\ops\resource_variable_ops.py:1635: calling
BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops)
with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

[43]: optimizer = 'adam'

loss_function = 'mean_squared_error'
metric = ['accuracy']

[44]: nn_model.compile(optimizer=optimizer, loss=loss_function,metrics=metric)

20
 CSE488: Computational Intelligence Lab07: Neural Networks and Tensorflow

1.7.3 Model training

[48]: nn_model.fit(inputX, inputY, epochs=10 )

Train on 11 samples
Epoch 1/10
11/11 [==============================] - 0s 273us/sample - loss: 0.1198 - acc:
0.8182
Epoch 2/10
11/11 [==============================] - 0s 91us/sample - loss: 0.1197 - acc:
0.8182
.
.
Epoch 8/10
11/11 [==============================] - 0s 182us/sample - loss: 0.1194 - acc:
0.8182
Epoch 9/10
11/11 [==============================] - 0s 182us/sample - loss: 0.1193 - acc:
0.8182
Epoch 10/10
11/11 [==============================] - 0s 182us/sample - loss: 0.1193 - acc:
0.8182

[48]: <tensorflow.python.keras.callbacks.History at 0x1e7542c10c8>

1.7.4 Model Evaluation

[49]: testX = np.array([[1.,2.25],[1.25,3.],[2,2.5],[2.25,2.75],[2.5,3.],
[2.,0.9],[2.5,1.2],[3.,1.25],[3.,1.5],[3.5,2.],[3.5,2.5]])
testY = np.array([[1.,0.]]*5 + [[0.,1.]]*6)

[50]: test_loss, test_accuracy = nn_model.evaluate(testX, testY)

print('Training Data Accuracy {}'.format(round(float(test_accuracy),2)))

11/11 [==============================] - 0s 91us/sample - loss: 0.0699 - acc:

1.0000
Training Data Accuracy 1.0

21