Getting started with Theano

Jon Barker, Solutions Architect

 What is Theano?
Simple first application

Theano for deep learning

Agenda Step-by-step example
Related projects
Example applications

2
What is Theano?
A mathematical symbolic expression compiler

A Python library for symbolic maths - far broader than just Deep Learning
Tightly integrated with the Python ecosystem

Fast C/CUDA back-end and transparent GPU acceleration

3
What is Theano?
Symbolic expression compiler

Variables and expressions are symbolic - more like maths than code

but, symbolic expressions use a familiar NumPy-like syntax

4
What is Theano?
Additional features

Developed and used since January 2008, created at Universit de Montral

Large contributor community

Tools for inspecting and debugging code

Great tutorials and examples - http://deeplearning.net/tutorial/

5
What is Theano?
Theano defines a language, a compiler and a library

Recipe for a Theano application:

Define symbolic expressions

Compile a function that can compute numeric values using those expressions
Execute that function on data

6
 y =ab
Example Theano application
a, b 2 R

7
 y =ab
Example Theano application
a, b 2 R

Initialize symbolic variables

8
 y =ab
Example Theano application
a, b 2 R

Initialize symbolic variables

Define symbolic expression

9
 y =ab
Example Theano application
a, b 2 R

Initialize symbolic variables

Define symbolic expression

Compile a function

10
 y =ab
Example Theano application
a, b 2 R

Initialize symbolic variables

Define symbolic expression

Compile a function

Use on numeric data

11
Theano for deep learning

Symbolic computation for tensors

Sub-modules of tensor operations relevant to DL

Highly expressive
Symbolic differentiation
Transparent GPU acceleration
Easily integrates with Python ecosystem

12
Theano for deep learning
Symbolic computation for tensors

y1 y2
y1 = (w11 x1 + w21 x2 + w31 x3 )
y2 = (w12 x1 + w22 x2 + w32 x3 )
w11 w12 w31 w33

=)
w21 w22

x1 x2 x3 y = (W x)

13
Theano for deep learning
Highly expressive: easily add new activation or loss functions

14
 The update is a function of
Theano for deep learning partial derivative of error w.r.t
parameters
Symbolic differentiation

15
 Example: Linear model

Fit a linear model to this data, i.e. find w 2 R such that y = wx

16
Example credit: Introduction to Deep Learning with Python, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
 Example: Linear model
Initialize symbolic variables

17
Example credit: Introduction to Deep Learning with Python, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
 Example: Linear model
Initialize symbolic variables

Define symbolic model

18
Example credit: Introduction to Deep Learning with Python, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
 Example: Linear model
Initialize symbolic variables

Define symbolic model

Initialize model parameter

19
Example credit: Introduction to Deep Learning with Python, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
 Example: Linear model
Initialize symbolic variables

Define symbolic model

Initialize model parameter

Define symbolic loss

20
Example credit: Introduction to Deep Learning with Python, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
 Example: Linear model
Initialize symbolic variables

Define symbolic model

Initialize model parameter

Define symbolic loss

Determine partial derivative of loss w.r.t parameter

21
Example credit: Introduction to Deep Learning with Python, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
 Example: Linear model
Initialize symbolic variables

Define symbolic model

Initialize model parameter

Define symbolic loss

Determine partial derivative of loss w.r.t parameter
Define how to update parameter based on gradient

22
Example credit: Introduction to Deep Learning with Python, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
 Example: Linear model
Initialize symbolic variables

Define symbolic model

Initialize model parameter

Define symbolic loss

Determine partial derivative of loss w.r.t parameter
Define how to update parameter based on gradient

compile theano function

23
Example credit: Introduction to Deep Learning with Python, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
 Example: Linear model
Initialize symbolic variables

Define symbolic model

Initialize model parameter

Define symbolic loss

Determine partial derivative of loss w.r.t parameter
Define how to update parameter based on gradient

Iterate through data 100 times, compile theano function

updating parameter after each
iteration

24
Example credit: Introduction to Deep Learning with Python, Alec Radford, https://www.youtube.com/watch?v=S75EdAcXHKk
Example: Linear model

Before training After 100 iterations

25
Designing more complex architectures

Layer types are typically defined as Python classes

26
Designing more complex architectures

Networks can be defined symbolically as sequences of class instances

27
Related projects
Built on top of Theano (mostly machine learning)

Blocks
Keras
Lasagne
Morb Typically simplify syntax and interface for artificial
neural network training at the expense of
Pylearn2
expressiveness
PyMC 3
sklearn-theano
theano-rnn
more
28
Related projects
CNN example in Keras

Input Conv MaxPool Dense DropOut Dense

29
Example applications
Music recommendation at Spotify

http://benanne.github.io/2014/08/05/spotify-cnns.html

Audio spectrogram Latent factors

input output

30
Example applications
Classifying galaxies and plankton and winning Kaggle contests
http://benanne.github.io/2014/04/05/galaxy-zoo.html
http://benanne.github.io/2015/03/17/plankton.html

31
Example applications

Many more examples at: http://deeplearning.net/tutorial/

Unsupervised training and auto-encoders

Recurrent Neural Networks

32
Installation

Linux, OS X or Windows
Requirements:

Python >= 2.6

g++, python-dev
NumPy, SciPy
BLAS

33
Installation

Basic instructions:
pip install Theano
easy_install Theano

Advanced instructions (for bleeding edge installs):

git clone git://github.com/Theano/Theano.git
cd Theano
python setup.py develop -user

34
Configuration

Three ways to configure Theano:

~/.theanorc: settings you always want
THEANO_FLAGS: setting for one job
theano.config: mid-code settings changes

35
GPU acceleration

Three ways to invoke:

Add device=gpu to .theanorc
Add device=gpu to THEANO_FLAGS
Set theano.config.device=gpu in code

cuDNN acceleration (including v4) is automatic if installed on system

36
Deep Learning Lab Series Schedule
developer.nvidia.com/deep-learning-courses
Review the other seminars in series

Seminar #2 Introduction to DIGITs

Seminar #3 - Getting Started with the CaffeFramework
Seminar #5 - Getting Started with the Torch Framework

37
Hands-on Lab

1. Create an account at nvidia.qwiklab.com

2. Go to Getting started with Theano lab at bit.ly/dlnvlab4

3. Start the lab and enjoy!

Only requires a supported browser, no NVIDIA GPU necessary!

Lab is free until end of Deep Learning Lab series

38