IMPLEMENTING NEURAL NETWORKS IN TENSORFLOW

DR. SANJAY CHATTERJI

 TensorFlow
 Now that we have a better theoretical understanding of deep learning
models, we will spend this chapter implementing some of these
algorithms in software.
 TensorFlow is an open source software library released in 2015 by
Google to make it easier for developers to design, build, and train
deep learning models.
 TensorFlow is a Python library that allows users to express arbitrary
computation as a graph of data flows.
 Nodes in this graph represent mathematical operations
 edges represent data that is communicated from one node to another

 Representing deep neural networks as tensors take advantage of

the speedups afforded by modern hardware.
TensorFlow Compare to Alternatives
 Theano (built by the LISA Lab out of the University of Montreal)
 Ithas an additional “graph compilation” step that takes significant amounts of
time while setting up certain kinds of deep learning architectures.
 TensorFlow has a much cleaner interface as compared to Theano.
 Many classes of models can be expressed in significantly fewer lines without
sacrificing the expressiveness of the framework.
 TensorFlow was built with production use in mind, whereas Theano was
designed by researchers almost purely for research purposes.
 Torch (largely maintained by Facebook AI Research)
 Thisframework is written in Lua which is a scripting language less commonly
used outside the deep learning community.
TensorFlow Basics
 Install
 sudo apt-get install python-pip python-dev
 pip install --upgrade tensorflow
 pip install ipython
 Constant
 import tensorflow as tf
 deep_learning = tf.constant('Deep Learning')
 session = tf.Session()
 session.run(deep_learning)
 a = tf.constant(2)
 b = tf.constant(3)
 multiply = tf.mul(a, b)
 session.run(multiply)
TensorFlow Variables
 TensorFlow variables are in-memory buffers that contain tensors.
 Normal tensors
 instantiated
when a graph is run
 immediately wiped clean afterward
 TensorFlow variables survive across multiple executions of a graph.
 explicitly
initialized before a graph is used for the first time.
 use gradient methods to modify variables after each iteration
 save the values stored in variables to disk.
 weights = tf.Variable(tf.random_normal([300, 200], stddev=0.5), name="weights“, trainable=False)
TensorFlow Operations
 TensorFlow operations represent abstract transformations that are
applied to tensors in the computation graph.
 Operations may also be supplied with an optional name attribute.
 An operation consists of one or more kernels
 represent device-specific implementations.
Placeholder Tensors
 The only missing piece is how we pass the input to our deep model.
 A variable
is insufficient as it is only meant to be initialized once.
 We need a component that we populate when the computation graph is run.
 TensorFlow solves this problem using a construct placeholder.
 Here we define a placeholder where x
 has 784 columns (each data sample has 784 dimensions)
 has undefined #rows (x can be initialized with arbitrary #data samples.)
 ith row of the multiply tensor corresponds to W multiplied with ith data sample.
 placeholders need to be filled every time the computation graph is run.
Sessions in TensorFlow
 TensorFlow program interacts with
computation graph using session.
 Session is responsible for
 building the initial graph
 to initialize all variables appropriately
 to run the computational graph
 let’s consider a simple Python script
 A computational graph is built by the session.
 We then initialize the variables as required
 Finally, we run the subgraph using sess.run
 sess.run interface can also be used to
train networks.
Navigating Variable Scopes and Sharing
Variables
 Building complex models often requires reusing and sharing large
sets of variables that we’ll want to instantiate together in one
place.
 Unfortunately, trying to enforce modularity and readability can
result in unintended results if we aren’t careful.
 Lets build a network setup consists of six variables describing
three layers.
 If we wanted to use this network multiple times, we’d prefer to
encapsulate it into a compact function like my_network.
 Use this network on two different inputs
 Our second call to my_network doesn’t use the same variables as
the first call.
 In many cases, we don’t want to create a copy, but rather reuse
the model and its variables.
 we shouldn’t be using tf.Variable.
 Instead, we should use a more advanced naming scheme that
takes advantage of TensorFlow’s variable scoping.

tf.get_variable(<name>, <shape>, <initializer>)

Checks if a variable with this name exists, retrieves the variable if it does, or creates it using the
shape and initializer if it doesn’t.
tf.variable_scope(<scope_name>)
Manages the namespace and determines the scope in which tf.get_variable operates.
 Enabling sharing
 Unlike tf.Variable, the tf.get_variable command checks that a
variable of the given name hasn’t already been instantiated.
 By default, sharing is not allowed (just to be safe!)
 If we want to enable sharing within a variable scope, we can say
so explicitly:
 Managing Models over the CPU and GPU
"/cpu:0"
The CPU of our machine.
"/gpu:0"
The first GPU of our machine, if it has one.
"/gpu:1"
The second GPU of our machine, if it has one.

To inspect which devices are used by the computational graph, we can initialize
our TensorFlow session with the log_device_placement set to True:
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
Models that span multiple GPUs by
building models in a tower-like fashion
 Using a specific device
 If we desire to use a specific device, we may do so by using with
tf.device to select the appropriate device.
 If the chosen device is not available, however, an error
will be thrown.
 If we would like TensorFlow to find another available device if the
chosen device does not exist, we can pass the
allow_soft_placement flag to the session variable.
Specifying Logistic Regression Model in TensorFlow
 Let’s build a simple model to tackle the MNIST dataset.
 our goal is to identify handwritten digits from 28 x 28 black-and-
white images.
 The first network that we’ll build implements logistic regression.
 Our model uses
 Matrix W represent the weights of the connections in the network
 Vector b corresponding to the biases to estimate whether an input x
belongs to class I
 It doesn’t have any hidden layers
 We have an output softmax of size 10
Pictorial representation of the Logistic
Regression network

 The model makes decent headway toward correctly classi-

fying our dataset, but there’s lots of room for improvement.
Phases of building logistic regression model
 inference: produces a probability distribution over the output
classes given a minibatch.
 loss: computes the value of the error function
 training: computes gradients of the model’s parameters and update
 evaluate: determines the effectiveness of a model
 We can represent logistic regression by taking softmax of the input
multiplied with matrix of weights connecting input and output layer.
 Row of output tensor: probability over output classes for
corresponding sample in the minibatch.
 Then compute the average error per data sample.
 Then compute gradients and modify the parameters of the model.
Logistic Regression model
 Note that when we create the training operation, we also pass in a variable
that represents the number of minibatches that have been processed.
 Each time the training operation is run, this step variable is incremented so
that we can keep track of progress.
 def training(cost, global_step):
 optimizer = tf.train.GradientDescentOptimizer(learning_rate)
 train_op = optimizer.minimize(cost, global_step=global_step)
 return train_op
 Finally, we put together a simple computational subgraph to evaluate the
model on the validation or test set.
 def evaluate(output, y):
 correct_prediction = tf.equal(tf.argmax(output, 1), tf.argmax(y, 1))
 accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
 return accuracy
Logging and Training the Logistic Regression Model
 as we train the model, we log several summary statistics.
 we use the tf.scalar_summary and tf.histogram_summary
commands to log the cost for each minibatch, validation error,
and the distribution of parameters.
 Every epoch, we run the tf.merge_all_summaries in order to
collect all summary statistics we’ve logged and use a
tf.train.SummaryWriter to write the log to disk.
 we also save the model parameters using the tf.train.Saver
model saver.
 By default, the saver maintains the latest five checkpoints, and
we can restore them for future use.
 Some codes we run in the class
import tensorflow as tf
x = tf.constant([1, 2, 3])
x = tf.constant(4)
y = tf.constant([4, 5, 6])
print(x)
z = tf.add(x, y)
x = tf.constant(4, shape=(1,1), dtype=tf.float32)
print(z)
print(x)
z = x+y
x = tf.constant([[4.0]])
print(z)
print(x)
z = x**3
x = tf.ones((3,3))
print(z)
print(x)
z = tf.tensordot(x, y, axes=1)
x = tf.zeros((4,3))
print(z)
print(x)
x = tf.random.normal(shape = (2, 3))
x = tf.eye(4)
y = tf.random.uniform(shape = (3, 2))
print(x)
z = tf.matmul(x, y)
x = tf.random.normal(shape = (3,3), mean = 0, stddev=1)
print(z)
print(x)
z=x@y
x = tf.random.uniform(shape = (3,3), minval=0, maxval=1)
print(z)
print(x)
x = tf.constant([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
x = tf.range(9)
print(x[:]) print(x[1:])
print(x)
print(x[1:3])
x = tf.range(start = 1, limit = 11, delta = 2)
print(x[::3])
print(x)
Thank You