Module 1

Generative Adversarial Network and Probabilistic Models

1. What are Generative Models?

Generative models are a type of AI that learn to generate new data points similar to a given
set of training data.
●​ Input: A dataset (e.g., images, text, etc.).
●​ Output: New, realistic samples similar to the training data.

Common Applications:

1.​ Image Generation: Creating lifelike images.

2.​ Style Transfer: Changing the style of images (e.g., sketches to realistic photos).
3.​ Data Augmentation: Generating synthetic but realistic data to expand datasets.
4.​ Anomaly Detection: Identifying unusual patterns.

2. Fundamentals of Probabilistic Modeling

Generative models are built on probabilistic modeling, using probability distributions to

represent data.

2.1 Probability Distributions Used in Generative Models

1.​ Gaussian (Normal) Distribution:

●​ Applies to continuous data and has a bell-shaped curve.
●​ Formula:

Pronounced as: f of x equals one divided by the square root of two pi sigma
squared, multiplied by e to the power negative of x minus mu squared over
two sigma squared.
2.​ Bernoulli Distribution
●​ Applies to binary outcomes (e.g., success or failure).
●​ Formula:
 Where x x is either 0 or 1, and p is the probability of success.

3.​ Multivariate Gaussian Distribution:

●​ Extends the Gaussian distribution to multiple dimensions (e.g., when
working with multiple features).

3. Introduction to Generative Adversarial Networks (GANs)

3.1 Overview:

GANs are a type of generative model introduced by Ian Goodfellow in 2014. They consist of
two neural networks:
1.​ Generator: Creates fake data points.
2.​ Discriminator: Classifies data as real or fake.

These two networks are trained together in a process called adversarial training:
●​ The Generator tries to produce data that looks real.
●​ The Discriminator tries to distinguish between real and fake data.

3.2 GAN Training Process:

●​ Step 1: The generator creates fake samples.

●​ Step 2: The discriminator evaluates the samples as “real” or “fake”.
●​ Step 3: Feedback flows back to the generator, allowing it to improve.

This process repeats until the generator is good enough that the discriminator can no longer
distinguish real from fake.

3.3 GAN Architecture:

1.​ Generator Network:

●​ Takes in random noise (z) from a latent space.
●​ Creates new data samples based on the noise input.
2.​ Discriminator Network:
●​ A binary classifier trained to predict whether input data is real or fake.
 ●​ The objective of GAN:

​ ​ Pronounce as: Minimize over G and maximize over D,

expectation of x sampled from real data distribution log of D(x) plus
expectation of z sampled from latent space log of one minus D of G of
z.

3.4 Challenges with GANs:

1.​ Training Instability: The two networks compete, which makes balancing training
difficult.
2.​ Mode Collapse: The generator learns to produce limited variations, causing a lack of
diversity.
3.​ Convergence Issues: GANs often struggle to reach a stable equilibrium during
training.

4. Other Common Generative Models

In addition to GANs, other popular generative models include:

1.​ Variational Autoencoders (VAEs):
●​ Learn to encode data into a latent space and generate new data by sampling
from it.
2.​ Autoregressive Models:
●​ Predict future data points based on past data.
●​ Examples: PixelRNN, PixelCNN.
3.​ Energy-Based Models:
●​ Assign energy scores to data points, with lower scores indicating higher
likelihood.

5. Importance of Generative Models

Generative models are vital to artificial intelligence because they allow machines to exhibit
creativity and produce original content.
Key Advantages:

●​ Data Synthesis: Generate realistic data for training.

●​ Creative Applications: Used in art, video, and music creation.
●​ Learning Representations: Help in understanding the underlying data structure.

Example: How GANs Generate Faces

1.​ The generator creates an image from random noise.

2.​ The discriminator determines whether the image is real or fake.
3.​ Over time, the generator learns to create lifelike images.

This process can produce realistic human faces, landscapes, or even entirely synthetic
datasets.

6. Challenges for Generative Models

1.​ High computational cost due to training complexities.

2.​ Generative models can inadvertently learn biases present in the training dataset.
3.​ Difficulties in balancing quality and diversity of outputs.

7. Significance of Probabilistic Models

Probabilistic models underpin the success of generative models by grounding them in

mathematical principles of probability.
1.​ Hidden Markov Models (HMMs):
●​ Useful for sequential data, like speech or time series.
2.​ Bayesian Networks:
●​ Represent probabilistic dependencies among variables.
●​ Example: Diagnosing diseases based on symptoms.
3.​ Gaussian Mixture Models (GMMs):
●​ Use multiple Gaussian distributions to model complex data.
Summary of Key Formulas:

Module 1.2. Probabilistic Models

1. What are Probabilistic Models?

Probabilistic Models use the principles of probability distributions to model uncertainty in

data. They provide a foundation for representing and reasoning about phenomena that
involve randomness or incomplete data.

Key Advantages:

1.​ Handles uncertainty effectively by modeling data with confidence or randomness.

2.​ Captures relationships between variables using probability.
3.​ Useful in diverse applications, from speech recognition to natural language
processing and computer vision.

2. Types of Probabilistic Models

2.1 Bayesian Networks

Bayesian Networks (also known as Belief Networks) are directed graphical models that
represent a set of variables and their probabilistic dependencies.
●​ Structure: Uses a directed acyclic graph (DAG) where:
●​ Nodes represent random variables.
●​ Edges represent conditional dependencies.

Example:

Diagnosing diseases based on symptoms:​

.

Joint Probability in Bayesian Networks:

For variables A,B,C the joint probability is written as:
P(A,B,C) = P(A) ⋅ P(B∣A) ⋅ P(C∣A)

2.2 Hidden Markov Models (HMMs)

HMMs are used to model sequential or time-series data where the system evolves over time,
but certain variables (states) are hidden.

Example:

Speech recognition — the states are phonemes (hidden), and the observations are sound
wave features.

2.3 Gaussian Mixture Models (GMMs)

GMMs are probabilistic models that represent data as a mixture of multiple Gaussian
distributions. They are often used for clustering tasks.
3. Important Concepts
4. Probabilistic Models in Practice

Applications:

1.​ Speech Recognition: Using HMMs to model phoneme sequences.

2.​ Image Segmentation: Applying GMMs for separating pixel groups in an image.
3.​ Anomaly Detection: Probabilistic models like GMMs or Bayesian networks detect
unusual events.

1.3 Basics of GAN

1. What are GANs?

Generative Adversarial Networks (GANs) are deep learning models introduced by Ian
Goodfellow in 2014. GANs are designed to generate realistic data by training two neural
networks—the Generator and the Discriminator—in an adversarial setting.

Why use GANs?

GANs are capable of:

1.​ Generating realistic data (e.g., images, text, music).
2.​ Addressing tasks like image synthesis, style transfer, and data augmentation.
2. Components of GANs

2.1 Generator (G)

The generator is a neural network that creates fake data from random noise, aiming to trick
the discriminator into classifying fake data as real.

Input to the Generator:

A random noise vector z, typically sampled from a probability distribution (e.g., Gaussian or
uniform).

Output from the Generator:

A fake data sample (e.g., an image, text, or audio).

2.2 Discriminator (D)

The discriminator is a binary classifier that tries to distinguish between real data (from the
dataset) and fake data (from the generator).

Input to the Discriminator:

●​ Real data samples from the dataset.
●​ Fake data samples from the generator.

Output from the Discriminator:

A probability:
●​ 1 if the data is real.
●​ 0 if the data is fake.

3. How GANs Work

3.1 The Adversarial Process:

GANs involve a game between two neural networks:

1.​ The Generator tries to produce realistic fake data.
2.​ The Discriminator tries to reject fake data and correctly identify real data.
3.2 Objective of GANs:

3.3 Steps in GAN Training:

1.​ Initialize weights for both the generator and discriminator.

2.​ Train the Discriminator:
●​ Distinguishes between real and fake samples.
●​ Updates weights to maximize its accuracy.
3.​ Train the Generator:
●​ Adjusts weights to produce samples that fool the discriminator.
4.​ Repeat the process alternately until the generator produces data indistinguishable
from real data.

4. Example: Image Generation using GANs

Let’s assume the task is to generate realistic human faces:

1.​ The Generator:
●​ Takes random noise as input (e.g., a vector of random numbers).
●​ Outputs an image of a human face.
2.​ The Discriminator:
●​ Receives two inputs:
●​ Real human faces (from the dataset).
●​ Fake human faces (from the generator).
●​ Outputs whether the input is real or fake.
3.​ Over several iterations, the Generator improves to produce realistic face images.

5. Challenges in GANs

1.​ Training Instability:

 ●​ Training GANs is like balancing two competing objectives, which can be
unstable.
2.​ Mode Collapse:
●​ The generator may produce limited variations of outputs rather than diverse
and realistic samples.
3.​ Overfitting:
●​ The discriminator may overfit to the training data, reducing its ability to
generalize.

6. Variants of GANs

Several improved versions of GANs address these challenges:

1.​ Conditional GANs (CGANs):
●​ Adds additional information (e.g., labels) to guide the generator and
discriminator.
2.​ Deep Convolutional GANs (DCGANs):
●​ Leverages convolutional neural networks (CNNs) for image generation.
3.​ Wasserstein GANs (WGANs):
●​ Addresses instability by replacing the loss function.

7. Mathematical Details

7.1 Discriminator Loss:

7.2 Generator Loss:

8. Applications of GANs
 1.​ Image Translation:
●​ Converting images from one domain to another (e.g., day-to-night,
sketch-to-photo).
2.​ Super-Resolution:
●​ Converting low-resolution images to high-resolution versions.
3.​ Text-to-Image Synthesis:
●​ Generating images from textual descriptions.
4.​ Data Augmentation:
●​ Generating synthetic samples to expand training datasets.

Summary Table of Key Formulas:

9. Visual Explanation of GAN Process (Illustration Reference)

You can create or include an illustration showcasing:

1.​ Generator takes noise as input and generates fake data.
2.​ Discriminator takes real and fake data as input and outputs probabilities.
3.​ Feedback loop between generator and discriminator.