The Deep Learning Glossary

DRIVERLESS CARS, BETTER PREVENTATIVE HEALTHCARE, AND
EVEN BETTER FASHION RECOMMENDATIONS ARE ALL POSSIBLE
TODAY BECAUSE OF DEEP LEARNING.

OUR FRIENDS AT RE-WORK PUBLISHED AN “A” TO “Z” DEEP
LEARNING GLOSSARY. HERE ARE THE MOST IMPORTANT TERMS
LINKED WITH RESOURCES FOR MORE IN-DEPTH EXPLORATION …

“PROCESSING DEVICES THAT ARE
LOOSELY MODELLED AFTER THE
NEURONAL STRUCTURE OF THE
HUMAN BRAIN.”
SOURCE: UNIVERSITY OF WISCONSIN-MADISON
ARTIFICIAL NEURAL NETWORKS (ANN’S)
NVIDIA Applied Research
Compressing DNA Engine Facial Performance Video Classification

“DESCRIBES A LARGE VOLUME OF DATA
– BOTH STRUCTURED AND
UNSTRUCTURED – THAT INUNDATES A
BUSINESS ON A DAY-TO-DAY BASIS.”
SOURCE: SAS INSIGHTS
BIG DATA
AI-Accelerated Analytics For Industries
Finance Telco IoT

“COMPRISED OF ONE OR MORE
CONVOLUTIONAL LAYERS AND THEN
FOLLOWED BY ONE OR MORE FULLY
CONNECTED LAYERS AS IN A
STANDARD MULTILAYER NEURAL
NETWORK.”
SOURCE: UFLDL
CONVOLUTIONAL NEURAL NETWORKS
Detection & Classification of
Hand Gestures
Resource Efficient Inference

“FORM OF MACHINE LEARNING THAT
ENABLES COMPUTERS TO LEARN FROM
EXPERIENCE AND UNDERSTAND THE
WORLD IN TERMS OF A HIERARCHY OF
CONCEPTS.”
SOURCE: GOODFELLOW, I., BENGIO, T., COURVILLE, A.
DEEP LEARNING
Deep Learning Applications
Transforming Finance Detecting Cancer Deep Learning In Self-Driving Cars

“IS A REPRESENTATION OF INPUT, OR
AN ENCODING. FOR EXAMPLE, A
NEURAL WORD EMBEDDING IS A
VECTOR THAT REPRESENTS THAT
WORD.”
SOURCE: DEEPLEARNING4J
EMBEDDING
Research Publication
Generating An Embedded Neural Network

“ALLOW SIGNALS TO TRAVEL ONE WAY
ONLY; FROM INPUT TO OUTPUT.
THERE IS NO FEEDBACK (LOOPS) I.E.
THE OUTPUT OF ANY LAYER DOES NOT
AFFECT THAT SAME LAYER.”
SOURCE: IMPERIAL COLLEGE
FEEDFORWARD NETWORK
Tutorial
Creating a Feed-Forward Network With DIGITS

“TYPE OF AI ALGORITHMS USED IN
UNSUPERVISED MACHINE LEARNING,
IMPLEMENTED BY A SYSTEM OF TWO
NEURAL NETWORKS COMPETING
AGAINST EACH OTHER IN A ZERO-SUM
GAME FRAMEWORK.”
SOURCE: IAN GOODFELLOW
GENERATIVE ADVERSARIAL NETWORKS
Origin of Generative Adversarial Networks
Ian Goodfellow Podcast GTC 17 Talk – Generative Adversarial Networks

“ARE AN ARCHITECTURE TO LET
INFORMATION FLOW UNHINDERED
ACROSS SEVERAL RNN LAYERS ON SO-
CALLED “INFORMATION HIGHWAYS.”
HIGHWAY NETWORKS
Highway Networks for Visual Questions Answering

“TRAINING DEEP MODELS ARE SUFFICIENTLY DIFFICULT TASKS. MOST
ALGORITHMS ARE STRONGLY AFFECTED BY THE CHOICE OF
INITIALIZATION. THE INITIAL POINT CAN DETERMINE WHETHER THE
ALGORITHM CONVERGES AT ALL, WITH SOME INITIAL POINTS BEING
SO UNSTABLE THAT THE ALGORITHM ENCOUNTERS NUMERICAL
DIFFICULTIES AND FAILS ALTOGETHER.”
SOURCE: IAN GOODFELLOW
INITIALIZATION
Article
Weight Initialization In Deep Neural Networks

“AN ARTIFICIAL NOISE ADDED TO THE
INPUTS DURING TRAINING USED AS
ANOTHER METHOD FOR REGULARIZATION
AND IMPROVING GENERALIZATION OF A
NEURAL NETWORK.”
SOURCE: MICHAEL BRAGISNKEY, CTO AT AIDOC
JITTER
Tutorial
What is Jitter?

“IS A TYPE OF UNSUPERVISED LEARNING, WHICH IS USED WHEN
YOU HAVE UNLABELED DATA (I.E., DATA WITHOUT DEFINED
CATEGORIES OR GROUPS). THE GOALS OF THIS ALGORITHM IS TO
FIND GROUPS IN THE DATA, WITH THE NUMBER OF GROUPS
REPRESENTED BY THE VARIABLE K.”
SOURCE: DATA SCIENCE
K-MEANS ALGORITHM
Tutorial
CUDA Implementation of the K-Means Clustering Algorithm

“FOR EACH PREDICTION, THERE IS AN
ASSOCIATED NUMBER WHICH IS THE
LOSS. FOR A TRUE PREDICTION, THE
LOSS WILL BE SMALL AND FOR A
TOTALLY WRONG PREDICTION THE LOSS
WILL BE HIGH.”
SOURCE: MICHAEL BRAGINSKY, CTO AT AIDOC
LOSS FUNCTION
Loss Function for Image Restoration

“IS A FEEDFORWARD NEURAL NETWORK WITH MULTIPLE FULLY-
CONNECTED LAYERS THAT USE NONLINEAR ACTIVATION FUNCTIONS
TO DEAL WITH DATA WHICH IS NOT LINEARLY SEPARABLE. AN MLP IS
THE MOST BASIC FORM OF A MULTILAYER NEURAL NETWORK, OR A
DEEP NEURAL NETWORKS IF IT HAS MORE THAN 2 LAYERS.”
SOURCE: DEEPLEARNING.NET
MULTILAYER PERCEPTRON (MLP)
Multi-Layer Perceptron On A GPU

“IS THE COMPREHENSION BY COMPUTERS OF THE STRUCTURE AND
MEANING OF HUMAN LANGUAGE (E.G., ENGLISH, SPANISH,
JAPANESE), ALLOWING USERS TO INTERACT WITH THE COMPUTER
USING NATURAL SENTENCES.”
SOURCE: GARTNER RESEARCH
NATURAL LANGUAGE PROCESSING
Blog
NVIDIA Developer – Natural Language Processing

“ONE-SHOT LEARNING IS WHEN AN
ALGORITHM LEARNS FROM ONE OR A FEW
NUMBER OF TRAINING EXAMPLES, CONTRAST
TO THE TRADITIONAL MACHINE-LEARNING
MODELS WHICH USES THOUSANDS EXAMPLES
IN ORDER TO LEARN..”
SOURCE: SUSHOVAN HALDAR
ONE-SHOT LEARNING
One-Shot Imitation Learning with OpenAI & Berkeley

“TYPE OF LAYER COMMONLY FOUND
IN CONVOLUTIONAL NEURAL
NETWORKS, WHICH INTEGRATES
INFORMATION FROM NEURONS WITH
NEARBY RECEPTIVE FIELDS.”
SOURCE: MICHAEL BRAGINSKY, CTA AT AIDOC
POOLING
Blog
ParallelForall – Deep Learning In a Nutshell Core Concepts

“A NOVEL ARTIFICIAL AGENT, TERMED A DEEP Q-NETWORK, THAT
CAN LEARN SUCCESSFUL POLICIES DIRECTLY FROM HIGH-
DIMENSIONAL SENSORY INPUTS USING END-TO-END REINFORCEMENT
LEARNING.”
SOURCE: PETERSEN, S. (2015) HUMAN-LEVEL CONTROL THROUGH DEEP REINFORCEMENT LEARNING.
Q-NETWORKS
Blog
Q-Network Trained to Play Breakout on OpenAI Gym

“A BRANCH OF MACHINE LEARNING THAT
IS GOAL ORIENTATED; THAT IS,
REINFORCEMENT LEARNING ALGORITHMS
HAVE AS THEIR OBJECT TO MAXIMIZE A
REWARD, OFTEN OVER THE COURSE OF
MANY DECISIONS.”
REINFORCEMENT LEARNING
Blog
Deep Learning in a Nutshell: Reinforcement Learning

“IS A FUNCTION USED AS THE OUTPUT LAYER OF A NEURAL NETWORK
THAT CLASSIFIES INPUT. IT CONVERTS VECTORS INTO CLASS
PROBABILITIES. SOFTMAX NORMALIZES THE VECTOR OF SCORES BY
FIRST EXPONENTIATING AND THEN DIVIDING BY A CONSTANT.”
SOFTMAX REGRESSION
Tutorial
Stanford – Softmax Regression

“ALLOWS US TO [TRAIN NEW MODELS] BY LEVERAGING THE ALREADY
EXISTING LABELED DATA OF SOME RELATED TASK OR DOMAIN. WE TRY
TO STORE THIS KNOWLEDGE GAINED IN SOLVING THE SOURCE TASK IN
THE SOURCE DOMAIN AND APPLY IT TO OUR PROBLEM OF INTEREST.”
SOURCE: SEBASTIAN RUDER
TRANSFER LEARNING
Transfer Learning From Deep Features for Remote Sensing and Poverty Mapping

“IS A TYPE OF MACHINE LEARNING
ALGORITHM USED TO DRAW INFERENCES
FROM DATASETS CONSISTING OF INPUT
DATA WITHOUT LABELED RESPONSES.”
SOURCE: MATHWORKS
UNSUPERVISED LEARNING
Blog
Using Unsupervised Learning For Artistic Style

“IS A DIRECTED MODEL THAT USES
LEARNED APPROXIMATE INFERENCE
AND CAN BE TRAINED PURELY WITH
GRADIENT-BASED METHODS.”
SOURCE: GOODFELLOW, IAN
VARIATIONAL AUTOENCODER
Blog
Auto-Encoder Model Querying a Computer To Design Clothing

“TO PENALIZE LARGE WEIGHTS USING
PENALTIES OR CONSTRAINTS ON THEIR
SQUARED VALUES (L2 PENALTY) OR
ABSOLUTE VALUES (L1 PENALTY).”
SOURCE: HINTON, . G. NEURAL NETWORKS FOR MACHINE LEARNING
WEIGHT DECAY
Weight Decay Can Improve Generalization

“THE PROCESS OF INITIALIZING WEIGHTS THAT THE VARIANCE
REMAINS THE SAME FOR “X” AND “Y”. THIS INITIALIZATION
PROCESS IS KNOWN AS XAVIER INITIALIZATION.”
SOURCE: PRATEEK JOSHI
XAVIER INITIALIZATION
Research Publications
Understanding the Difficulty of Training Deep Feedforward Neural Networks

“AS PIONEERS OF DEEP LEARNING, YOSHUA & YANN HAVE MADE
SIGNIFICANT CONTRIBUTIONS TO THE ADVANCEMENTS OF THIS FIELD.
THEY WILL BOTH BE JOINING THE DEEP LEARNING SUMMIT,
MONTREAL IN OCTOBER, SPEAKING ON THE “PANEL OF PIONEERS.”
SOURCE: RE-WORK GLOSSARY
YOSHUA BENGIO &
Their Work
Advancing Robotics, Physics, and Medicine
with AI – Yann LeCunn
YANN LECUN
The Rise of AI Through Deep Learning – Yoshua Bengio

“AN EXTREME FORM OF TRANSFER
LEARNING, WHERE NO LABELED
EXAMPLES ARE GIVEN AT ALL FOR THE
ZERO-SHOT LEARNING TASK.”
SOURCE: GOODFELLOW, IAN
ZERO-SHOT LEARNING
Zero-Shot Learning Through Cross-Modal Transfer

Explore Now
DISCOVER THE LATEST NEWS AND
UPDATES ON DEEP LEARNING AND AI …

The Deep Learning Glossary

More Related Content

What's hot (20)

Similar to The Deep Learning Glossary (20)

More from NVIDIA (20)

Recently uploaded (20)

The Deep Learning Glossary

Editor's Notes