Generative Models

Discriminative/Predictive Models
• we use the training data to find a discriminant function f(x) that maps
each x directly onto a class label, thereby combining the inference
and decision stages into a single learning problem.
Generative Models
• Approaches that explicitly or implicitly model the distribution of
inputs as well as outputs are known as generative models, because by
sampling from them it is possible to generate synthetic data points in
the input space.
• Page 43, Pattern Recognition and Machine Learning, 2006.
Generative Model
• A Generative Model is a powerful way of learning any kind of data
distribution using unsupervised learning.
All types of generative models aim at
• Learning the true data distribution of the training set so as to
generate new data points with some variations.
.
Generative Model
• A generative model describes how a dataset is generated, in terms of
a probabilistic model. By sampling from this model, we are able to
generate new data.

• This is because a model must discriminate examples of input variables

across classes; it must choose or make a decision as to what class a
given example belongs.
Generative Model
Two of the most commonly used and efficient approaches are
• Variational Autoencoders (VAE)
• Generative Adversarial Networks (GAN).
Generative Adversarial Networks
(GAN).
• Generative adversarial networks (GANs) are algorithmic architectures
that use two neural networks in order to generate new, synthetic
instances of data similar to real data. They are used widely in image
generation, video generation and voice generation.
Generative Adversarial Networks
(GAN).
• A generative adversarial network (GAN) has pair of neural networks:

• The generator learns to generate plausible data. The generated instances become negative training examples for the
discriminator.
• The generator tries to generate fake images
• The discriminator learns to distinguish the generator's fake data from real data. The discriminator penalizes the generator
for producing implausible results.
• The discriminator tries to detect the fake images
Generative Adversarial Networks
(GAN).
• Intuitions:
• ●Generator tries the best to cheat the discriminator by generating
more realistic images by improving the generated images.
• In this ways the training procedure continues until the discriminator is
deceived by the generator.
• Discriminator tries the best to distinguish whether the image is
generated by generator or it is real image
Building Simple GAN
Faces look like a diagonal or backward slash
Noisy Image

• These are noisy images not faces.

• Values here are set as random numbers, mostly generated randomly
During training the model will learn that the faces will look like, higher
values will be at diagonal
 Building a Discriminator

• =1*1+0*(-1)+0*(-1)+1*1 =0.25*1+1*(-1)+0.5*(-1)+0.75*1
• =2 =0.5
• We will use a threshold here
If value>1then face
• else not face
• It is just a classifier that learns the weights and relationship between the features and target labels to assign
the real or fake labels
Generator Network
Generator Input and outputs
• Inputs: Point in latent space, e.g. a 100 element vector of Gaussian random numbers.
• Outputs: Two-dimensional square grayscale image of 28×28 pixels with pixel values in [0,1].
• Developing a generator model requires that we transform a vector from the latent space
with, 100 dimensions to a 2D array with 28×28 or 784 values.
• The first step is a Dense layer as the first hidden layer that has enough nodes to represent a
low-resolution version of the output image. Specifically, an image half the size (one quarter
the area) of the output image would be 14×14 or 196 nodes, and an image one quarter the
size (one eighth the area) would be 7×7 or 49 nodes.
• We don’t just want one low-resolution version of the image; we want many parallel versions or
interpretations of the input. This is a pattern in convolutional neural networks where we have many
parallel filters resulting in multiple parallel activation maps, called feature maps, with different
interpretations of the input. We want the same thing in reverse: many parallel versions of our output
with different learned features that can be collapsed in the output layer into a final image. The model
needs space to invent, create, or generate.
Training and Error
• Targer label=1
• Prediction=0.1
• Error=1-0.1 error=-ln(0.1)=2.3
• This is large error

• Targer label=1
• Prediction=0.99
• Error=1-0.1 error=-ln(0.1)=0.1
• This is smallerror
• Targer label=0
• Prediction=0.1
• Error=1-0.1 error=-ln(1-0.1)=0.1
• This is smallerror

• Targer label=1
• Prediction=0.99
• Error=1-0.1 error=-ln(1-0.1)=2.3
• This is largeerror
• If prediction is to be 1
• Then log-loss=-ln(prediction)
• If prediction is to be 0 then
• Log-loss=-ln(1-prediction)
Back Propagation

Error Plot

Prediction
Error
Training of GANs
• Training is carried out in two steps
• Train the Discriminator
The discriminator is trained and the generator network is freezed
• This means the training dataset of the generator is turned as false
• The network only makes forward propagation(no back propagation is carried
out at this stage)
• The discriminator is trained on the real data and it is checked if it can predict
them correctly then it is trained on the fake data genrated by the
discriminator
• Train the Generator
The Generator is trained and the generator network is freezed
• Get the results from the generator and train the generator to produce better
samples
• Then using the error back propagation the generator leans to produce better
samples
The weights in the generator model are updated based on the performance of
the discriminator model.

The training continues until the generator fools the discriminator

Training of GANs (Cont…)
Training of the GAN
Now image generated by the generator is passed to the discriminator
the discriminator gives us the probability
Here we have to define the correct error function
here if the image is generated by the generator the discriminator should say it
fake

Discriminator
Generator

Prediction here should be=0

Error=-ln1-0.68)

Ln(1-D(G(z))
Ln(D(G(z))
 Now generator is required to generate the image that looks real
During training many epochs (many iteration doing the same process) are carried
out to update the weights

Discriminator
Generator Prediction here should be=1
Error=-ln(0.68)

Prediction here should be=0

Error=-ln(1-0.68)

Ln(1-D(G(z))
Ln(D(G(z))
Training consisting of many epochs
• A random vector is fed to the generator
• The generator produces the image
• The produced image is passed to the discriminator to check whether the image is fake or real
• Then use the back propagation to update the weights of the generator and discriminator
• Then we plug into the discriminator the real image and update the
weights of the discriminator using back propogation
Training of Gen. and Disc.
• Input is random numbers vector
• During forward pass we get image from generator which is usually not as the target(in this case not as face)
• Then generated image is passed to the discriminator
• The discriminator recognizes whether it is fake or original by giving the output as probability
• If probability is higher that 0.5 the image is real image otherwise fake image
• After any epochs these weights are obtained
The Generator Network
• Important:
• The generator model is not compiled and does not specify a loss function or optimization
algorithm. This is because the generator is not trained directly.
• The generator model is responsible for creating new, fake but plausible images of data samples.
• It does this by taking a point from the latent space as input and outputting a square image.
• The latent space is an arbitrarily defined vector space of Gaussian-distributed values, e.g. 100
dimensions.
• It has no meaning, but by drawing points from this space randomly and providing them to the
generator model during training,
• The generator model will assign meaning to the latent points and, in turn, the latent space,
until, at the end of training, the latent vector space represents a compressed representation of
the output space, images, that only the generator knows how to turn into plausible images.
• Therefore, the first hidden layer, the Dense, needs enough nodes for multiple low-
resolution versions of our output image, such as 128.
• model.add(Dense(128 * 7 * 7, input_dim=100))
• The activations from these nodes can then be reshaped into something image-like
to pass into a convolutional layer, such as 128 different 7×7 feature maps.
• model.add(Reshape((7, 7, 128)))
• The next major architectural innovation involves upsampling the low-resolution
image to a higher resolution version of the image.
• The output layer of the model is a Conv2D with one filter and a kernel size of 7×7
and ‘same’ padding, designed to create a single feature map and preserve its
dimensions at 28×28 pixels. A sigmoid activation is used to ensure output values
are in the desired range of [0,1].
• The GAN architecture was first described in the 2014 paper by Ian
Goodfellow, et al. titled “Generative Adversarial Networks.”
• A standardized approach called Deep Convolutional Generative
Adversarial Networks, or DCGAN, that led to more stable models was
later formalized by Alec Radford, et al. in the 2015 paper titled “
Unsupervised Representation Learning with Deep Convolutional
Generative Adversarial Networks“.
• Most GANs today are at least loosely based on the DCGAN
architecture …
• — NIPS 2016 Tutorial: Generative Adversarial Networks, 2016.