Deep Learning Concepts Summary

1. Encoder-Decoder Models
The encoder-decoder architecture is foundational in deep learning, especially for
sequence-to-sequence tasks like machine translation, text summarization, speech recognition, and
image captioning.

Encoder: Takes an input (sequence or image) and converts it into a fixed-size latent vector called
the context vector or embedding. For example, in language translation, the encoder converts an
English sentence into a compressed vector representation.

Decoder: Takes this context vector and generates the target sequence, one step at a time. For
example, in translation, the decoder generates the French sentence word by word.

Example:
For English -> French translation:

Encoder: "I am happy" -> [compressed vector]

Decoder: [compressed vector] -> "Je suis heureux"

Limitation:
For long sequences, compressing all information into a single vector is inefficient - important details
may be lost, and performance drops.

2. Attention Mechanism
The attention mechanism solves the bottleneck problem of encoder-decoder models.

Instead of squeezing all input into one fixed-size vector, the decoder can dynamically look back at
all encoder outputs and decide which parts are most relevant at each decoding step. It does this by
computing attention weights, which indicate how important each input element is for generating the
current output.
Example:
While translating the English sentence "The cat sat on the mat":

When generating the French word "le," attention focuses on "the."

When generating "chat," it focuses on "cat."

Types of attention:
- Soft attention: Learns a weighted average over all inputs (fully differentiable).
- Hard attention: Selects one input location at a time (non-differentiable, often trained with
reinforcement learning).

This mechanism greatly improves performance in translation, summarization, speech synthesis, and
more.

3. Attention Over Images

Attention over images brings the attention mechanism into computer vision.

In tasks like image captioning or visual question answering (VQA):

Instead of processing the entire image as a whole, the model focuses on specific image regions
relevant to the current output.

Example:
While describing an image with a dog and a ball, the model:
- Focuses on the dog region when generating the word "dog."
- Focuses on the ball region when generating the word "ball."

By combining CNN feature maps (which retain spatial information) with attention mechanisms, the
model becomes more interpretable and more accurate.

4. Variational Autoencoders (VAEs)

VAEs are a type of generative model that learn to represent data as distributions rather than points
in latent space.
Encoder: Maps input x (e.g., an image) to a probabilistic latent representation z, typically a Gaussian
distribution.
Decoder: Samples from this latent distribution and tries to reconstruct the original input.

Key innovations:
- Instead of learning a fixed code (like in regular autoencoders), VAEs learn a distribution over latent
codes.
- They balance:
* Reconstruction loss: How well the output matches the input.
* KL divergence loss: How close the latent distribution is to a prior (often a standard Gaussian).

Benefits:
- Smooth and continuous latent space -> allows easy interpolation and sampling.

Useful for:
- Image generation
- Denoising
- Semi-supervised learning
- Anomaly detection

5. Generative Adversarial Networks (GANs)

GANs are another class of generative models with two competing networks:

Generator G: Produces synthetic data (e.g., fake images) from random noise.
Discriminator D: Tries to distinguish real data from fake.

Training process:
- G improves to fool D.
- D improves to catch G.

This adversarial game continues until G produces samples indistinguishable from real data.
Challenges:
- Training instability
- Mode collapse (generator producing limited diversity)
- Difficult hyperparameter tuning

Applications:
- Image synthesis (e.g., faces, artwork, realistic objects)
- Style transfer
- Data augmentation
- Super-resolution

Recent GAN variants like StyleGAN and BigGAN have revolutionized fields like media,
entertainment, and design.

6. Multi-task Deep Learning

Multi-task learning (MTL) involves training a single model to perform multiple related tasks
simultaneously.

Example in vision:
One model takes an image and:
- Classifies the object (e.g., "dog"),
- Predicts its location (bounding box),
- Segments its outline (segmentation mask).

Advantages:
- Shared representations: Learning common features improves generalization.
- Data efficiency: Less data per task required.
- Reduced overfitting: Regularization effect by solving related tasks.
- Efficiency: One model instead of many.

Architecture:
- Shared backbone (e.g., CNN, transformer).
- Task-specific heads (e.g., classifier, detector, segmenter).
MTL has been successful in areas like vision, natural language processing, healthcare, and
autonomous driving.

7. Multi-view Deep Learning

Multi-view learning integrates data from multiple sources (views) to improve performance.

Examples of views:
- Image + text (e.g., image captioning)
- Audio + video (e.g., emotion recognition)
- Different camera angles (e.g., 3D pose estimation)

Goal:
Learn a joint representation that combines complementary information from each view.

Example:
In video sentiment analysis:
- Visual view -> facial expressions,
- Audio view -> speech tone,
- Text view -> subtitles or transcripts.

Approaches:
- Feature concatenation: Combine features after extracting them.
- Cross-view attention: Learn interactions between views.
- Co-training: Train separate models on each view, then align them.

Multi-view learning improves robustness, especially when some modalities are missing or noisy.

8. Applications: Vision, NLP, Speech

-> Computer Vision:
- Object detection: Identifies and localizes objects (YOLO, Faster R-CNN).
- Image segmentation: Assigns a label to each pixel (U-Net, DeepLab).
- Image generation: Creates new images (StyleGAN, VAE, GANs).
- Visual question answering: Answers questions about an image using attention + CNNs.

-> Natural Language Processing (NLP):

- Machine translation: Translates between languages (Transformer, BERT, GPT).
- Summarization: Generates summaries of texts.
- Question answering: Finds answers from documents (e.g., BERT, RoBERTa).
- Sentiment analysis: Detects emotion or opinion in text.

-> Speech:
- Speech recognition: Converts speech to text (wav2vec, DeepSpeech).
- Speech synthesis: Generates realistic speech (Tacotron, WaveNet).
- Speaker identification: Recognizes who is speaking.
- Emotion detection: Determines the speaker's emotional state.

Key: Summary
Concept Purpose Example Applications
Encoder-Decoder Map variable-length input to output Machine translation, summarization
Attention Mechanism Focus on relevant input parts Translation, image captioning, speech
synthesis
Attention over Images Focus on image regions dynamically Image captioning, VQA
VAE Learn generative latent distributions Image generation, anomaly detection
GAN Generate realistic synthetic data Face synthesis, style transfer
Multi-task Learning Train on multiple related tasks Vision multitasking, NLP multitask models
Multi-view Learning Integrate multiple data sources Multimodal sentiment analysis, 3D vision
Applications Apply to real-world domains Vision, NLP, Speech