Lecture 11

An Introduction to Deep Learning and Keras

Internet of Things
The slides are made by Andreas Karagounis and John Ramey,
adapted by Tan Le

Instructor: Le Duy Tan, Ph.D.

Email: [email protected]
 source: https://www.alibabacloud.com/blog/Learning-Machine-Learning-Part-3-Application_p68909
source: https://hackernoon.com/deep-learning-resources-e32bd081e84d

source: https://hackernoon.com/challenges-in-deep-learning-57bbf6e73bb source: https://analyticsindiamag.com/artificial-neural-networks-101/

source: http://time.com/4129247/google-self-driving-cars-patent/

source: https://www.zmescience.com/science/news-science/self-driving-car-how-it-sees/

source: http://www.businessinsider.com/how-googles-self-driving-cars-see-the-world-2015-
10
source: https://gfycat.com/gifs/detail/AntiqueFavoriteJackal

source: https://research.googleblog.com/2016/10/supercharging-style-transfer.html
source: https://deepdreamgenerator.com/

source: http://barabeke.it/portfolio/deep-dream-generator

source: https://deepdreamgenerator.com/
 source: https://nicolovaligi.com/deep-learning-models-semantic-segmentation.html

source: https://www.suasnews.com/2017/08/yolo-open-source-real-time-image-recognition/

source:
http://68.media.tumblr.com/b813b4afff4b1bc74dd553158fe4aca5/tumblr_oiuv3zLd
u51qav3uso4_r1_540.gif
What we will accomplish in this hour:

● Load housing data into Python

● Normalize the data
● Create a neural network model with Keras for
predicting house prices (regression problem)
● If we have time we will talk about overfitting
and regularization
 Keras is a high-level neural
network API written in Python
capable of running on top
TensorFlow

Goal: go from idea to result

with the least possible delay
source: https://keras.io/

TensorFlow is an open source

software library for numerical
computation using data flow
graphs
source:
https://upload.wikimedia.org/wikipedia/c
ommons/thumb/1/11/TensorFlowLogo.s
vg/2000px-TensorFlowLogo.svg.png
 Boston Housing dataset
● Samples contain 13
attributes of houses at
different locations around
the Boston suburbs in the
late 1970s
● Targets are the median
price of the house (what
we are trying to predict)

source: https://cdn-images-1.medium.com/max/2000/1*bvXWvdDMk8FTS1WWd9_JEg.jpeg
Our 506 samples
Data we are using to build our model The continuous value that we are
trying to predict
Zero indexed

13 Features: crime, age, number of rooms 1 Label: The price of the home
etc... (median value in $1000’s)
Loading our data and splitting into training, validation and test
(x_train, y_train), (x_val, y_val), (x_test, y_test) = np.load('housing.npy')

Final stage

Tuning parameter stage

● Out of our 506 samples we will allocate: 350 to training, 50 to validating and
106 to testing
● Very important in machine learning that we have separate data which we
don’t learn from so that we can make sure that what we’ve learned actually
generalizes!
Preprocessing stage: Always normalize your data!

source: http://cs231n.github.io/neural-networks-2/

● Normalization of a dataset is a common requirement for many machine learning estimators: they might behave
badly if the individual feature do not more or less look like standard normally distributed data
● Z-score normalization is the most popular method
 Linear Regression
What is simple linear regression? What is multiple linear regression?

● Learning a collection of weights

● Fitting hyperplanes!

source: http://sphweb.bumc.bu.edu/otlt/MPH-
Modules/BS/R/R5_Correlation-Regression/R5_Correlation-
source: https://sebastianraschka.com/faq/docs/closed-form-vs-gd.html Regression_print.html
We want to minimize these vertical offsets
● We want to minimize the following cost function

Learned weights

vertical offsets
What are neurons in a neural network

source: http://cs231n.github.io/neural-networks-1/
Example of an activation function

● Allows for nonlinearities

source: https://ayearofai.com/rohan-4-the-vanishing-gradient-problem-ec68f76ffb9b
Ok let’s define our model in Keras
first_model = Sequential()
first_model.add(Dense(13, input_dim=13, kernel_initializer='random_uniform', activation='relu'))
first_model.add(Dense(1, kernel_initializer='random_uniform'))

The Sequential model is a linear stack of layers

first_model = Sequential()
 Number of features Activation function
Number of neurons
Randomly initializing weights

first_model.add(Dense(13, input_dim=13, kernel_initializer='uniform', activation='relu'))

Number of neurons Randomly initializing weight

first_model.add(Dense(1, kernel_initializer='uniform'))
What does our model even look like?
Input layer

13*13+13=182 weights in total!

14 biases
ReLU is getting applied to our 13
Hidden layer neurons in the hidden layer

Output layer
Defining an optimizer and compiling our model

Learning rate

sgd = SGD(lr=0.03)
first_model.compile(loss='mean_squared_error', optimizer=sgd)
What is SGD?
● SGD (Stochastic Gradient Descent) is a variant of Gradient Descent

Learning rate

Determines the size of each step

source: https://sebastianraschka.com/faq/docs/closed-form-vs-gd.html
Training and testing our model
first_model.fit(x_train, y_train, batch_size=5, validation_data=(x_val, y_val), epochs=30)
test_score = first_model.evaluate(x_test, y_test)

Training feature matrix 350 Number iterations through the

samples entire dataset
Feeding data in by rows of 5

first_model.fit(x_train, y_train, batch_size=5, validation_data=(x_val, y_val), epochs=30)

Training labels 350 samples

Validation labels 50 labels
Validation feature matrix 50
samples
 Test feature matrix 106
samples Test labels 106
samples

test_score = first_model.evaluate(x_test, y_test)

What is actually happening?

source: https://www.codeproject.com/KB/dotnet/predictor/learn.gif
source: http://knowyourmeme.com/memes/we-need-to-go-deeper
Jokes aside this is what we do!
We tune the hyperparameters:

● The learning rate

● The batch size
● The number of epochs
● The number of neurons per layer
● The number of layers (deep learning)
● Different optimizers
● Add regularization (next few slides)
● List continues...
Overfitting
● Our model is failing to generalize well
● Test error is higher than our training error
● As our model becomes more flexible overfitting becomes a larger problem

Test MSE

Training MSE

source: http://www-bcf.usc.edu/~gareth/ISL/
Add regularization: dropout!

● Reduces overfitting by killing off (dropping out)

neurons in the network with a certain probability
Probability with which
we dropout

regularized_model.add(Dropout(0.5))

source: https://medium.com/@amarbudhiraja/https-medium-com-amarbudhiraja-learning-less-to-learn-better-dropout-in-deep-machine-learning-74334da4bfc5
Overfitting in our model
Why is deep learning all the hype?

● Test Deep Learning Model MSE: 0.619443686222

● Test scikit-learn MSE: 1.00261191579

Capturing nonlinearities in our predictor variables!

Where to go from here...
● Machine Learning Crash Course: https://developers.google.com/machine-
learning/crash-course/
● Deep Learning Specialization: https://www.coursera.org/specializations/deep-
learning
● Neural Networks and Deep Learning:
http://neuralnetworksanddeeplearning.com/
● More advanced Deep Learning Book: http://www.deeplearningbook.org/
Your activity
Please work in a group, then make the presentation
slides
- Access: https://keras.io/api/applications/
- Select 01 available model to answer these
questions:
• What is your understanding of this model: what
are they used for, input, output, and more!
• Run an example of this model on Google Colab