Lecture 10:
Convolutional Neural Networks
Zerrin Yumak
Utrecht University
Announcements
• Project proposal deadline is extended!
• New project proposal deadline: 21 Dec Monday 23:59pm
• We will look at your proposals and give feedback by email.
• If you have any questions, you can ask for an appointment after Christmas
holidays (meetings will happen during lecture hours)
• Not mandatory, please ask for an appointment 3-4 days in advance.
• For technical questions, reach out to your TAs and talk to them in computer lab
hours.
In this lecture
• A bit of history
• CNN basics
• Convolutions, strides, pooling
• CNN architectures
• AlexNext
• ZFNet
• VGGNet
• GoogLeNet
• Residual Networks
• Applications of CNN
• Object detection
• Object segmentation
• …
A bit of history
Hubel and Wiesel, Harvard, 1960s © MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
A bit of history
CS231n: Convolutional Neural Networks
A bit of history
CS231n: Convolutional Neural Networks
A bit of history
CS231n: Convolutional Neural Networks
CNNs are widely used now..
CS231n: Convolutional Neural Networks
Images are numbers
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
Fully Connected Neural Network
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
Using Spatial Structure
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
Using Spatial Structure
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
Feature Extraction with Convolution
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
Feature Extraction and Convolution
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
Features of X
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
Filters to Detect X Features
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
The Convolution Operation
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
The Convolution Operation
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
The Convolution Operation
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
The Convolution Operation
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
The Convolution Operation
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
Producing Feature Maps
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
CNNs for Classification
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
Convolutional Layers: Local Connectivity
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
CNNs: Spatial Arrangement of Output Volume
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
Introducing Non-Linearity
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
Pooling
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
Pooling
CS231n: Convolutional Neural Networks
Convolution Layer
CS231n: Convolutional Neural Networks
Convolution Layer
CS231n: Convolutional Neural Networks
Convolution Layer
CS231n: Convolutional Neural Networks
Convolution Layer
CS231n: Convolutional Neural Networks
Convolution Layer
CS231n: Convolutional Neural Networks
Convolution Layer
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
Stride
CS231n: Convolutional Neural Networks
In practice: Common to zero pad the border
CS231n: Convolutional Neural Networks
In practice: Common to zero pad the border
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
[ConvNetJS demo: training on CIFAR-10]
CS231n: Convolutional Neural Networks
CNNs for Classification: Feature Learning
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
CNNs for Classification: Class Probabilities
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
CNNs: Training with Backpropagation
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
ImageNet Dataset
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
ImageNet Challenge
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
ImageNet Challenge: Classification Task
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
ImageNet Challenge: Classification Task
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2017. ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 6 (May 2017), 84-90
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
ImageNet Large Scale Visual Recognition
Challenge (ILSVRC) winners
CS231n: Convolutional Neural Networks
ImageNet Large Scale Visual Recognition
Challenge (ILSVRC) winners
M. D. Zeiler and R. Fergus, Visualizing and Understanding Convolutional Networks, European conference on computer vision, 818-833, 2013
CS231n: Convolutional Neural Networks
ImageNet Large Scale Visual Recognition
Challenge (ILSVRC) winners
CS231n: Convolutional Neural Networks
K. Simonyan and A. Zisserman, Very Deep Convolutional Networks for Large-Scale Image Recognition, Computer Vision and Pattern Recognition, 2014
CS231n: Convolutional Neural Networks
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelv, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, IEEE Conference on Computer Vision and Pattern
Recognition, 2015
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
https://www.youtube.com/watch?v=C86ZXvgpejM
https://www.youtube.com/watch?v=KfV8CJh7hE0
CS231n: Convolutional Neural Networks
ImageNet Large Scale Visual Recognition
Challenge (ILSVRC) winners
CS231n: Convolutional Neural Networks
K. He, X. Zhang, S. Ren, J. Sun, Deep Residual Learning for Image Recognition, IEEE Conference on Computer Vision and Pattern Recognition, 2016
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
CS231n: Convolutional Neural Networks
https://www.youtube.com/watch?v=ZILIbUvp5lk
https://www.youtube.com/watch?v=RYth6EbBUqM
CS231n: Convolutional Neural Networks
Comparing complexity
CS231n: Convolutional Neural Networks
Data
© MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
So far: Image Classification
CS231n: Convolutional Neural Networks
Other Computer Vision Tasks
CS231n: Convolutional Neural Networks
Semantic Segmentation
CS231n: Convolutional Neural Networks
Semantic Segmentation: FCNs
Long, Shelhamer, and Darrell, “Fully Convolutional Networks for Semantic Segmentation”, CVPR 2015
Noh et al, “Learning Deconvolution Network for Semantic Segmentation”, ICCV 2015 © MIT 6.S191: Introduction to Deep Learning
IntroToDeepLearning.com
Classification + Localization
CS231n: Convolutional Neural Networks
Classification + Localization
CS231n: Convolutional Neural Networks
Human Pose Estimation
Johnson and Everingham, "Clustered Pose and Nonlinear Appearance Models for Human Pose Estimation", BMVC 2010
Toshev and Szegedy, “DeepPose: Human Pose Estimation via Deep Neural Networks”, CVPR 2014
CS231n: Convolutional Neural Networks
Human Pose Estimation
CS231n: Convolutional Neural Networks
Object Detection/Instance Segmentation
Girshick et al, “Rich feature hierarchies for accurate object detection and semantic segmentation”, CVPR 2014
Redmon et al, “You Only Look Once: Unified, Real-Time Object Detection”, CVPR 2016
He et al, “Mask R-CNN”, arXiv 2017
© MIT 6.S191: Introduction to Deep Learning
https://www.youtube.com/watch?v=GSwYGkTfOKk IntroToDeepLearning.com
Image Captioning using RNNs
CS231n: Convolutional Neural Networks
Image Captioning using RNNs
A. Karpathy, L. Fei-Fei, Deep Visual-Semantic Alignments for Generating Image Descriptions, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
https://cs.stanford.edu/people/karpathy/deepimagesent/
CS231n: Convolutional Neural Networks
Visualizing Convolutional Networks
http://www.cs.cmu.edu/~aharley/vis/
© MIT 6.S191: Introduction to Deep Learning
For more info: http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture12.pdf IntroToDeepLearning.com
For your projects..
• Use research papers mentioned in the slides and other papers you found
• There are links in the slides and on the project website
• An interesting source is https://paperswithcode.com
• Do your research and investigate what is possible and what you like to do
• Choose a problem
• e.g. object classification, object detection, segmentation, face detection/recognition, pose estimation, image/video
captioning
• You can also choose more advanced problems such as GANs
• Choose a dataset
• e.g. CIFAR, ImageNet, MSCoco
• Modify/reimplement different architectures
• e.g. AlexNet, GoogLeNet, ResNet etc.
• Try different hyperparameters
• Pay attention, the models in these slides are just the beginning. More challenging, state-of-the-art problems and in depth
analysis result in higher grade!
Supplementary material and references
• Deep Learning book, Chapter 9
• CS231N: Convolutional Neural Networks, Stanford University
• http://cs231n.stanford.edu/2017/syllabus
• 6.S191: Introduction to Deep Learning, MIT
• http://introtodeeplearning.com/2018/#schedule
• Coursera Deeplearning.ai on YouTube:
https://www.youtube.com/channel/UCcIXc5mJsHVYTZR1maL5l9w/videos
• (Slides are mainly adopted from the above courses)
