Design and Analysis of

Algorithms
A Beginner's Guide

By Manus AI

May 2025
 Table of Contents
Chapter 1: Introduction to Algorithms

Chapter 2: Fundamental Algorithmic Strategies

Chapter 3: Graph and Tree Algorithms

Chapter 4: Tractable and Intractable Problems

Chapter 5: Advanced Topics

Chapter 1: Introduction to Algorithms
Welcome to the fascinating world of algorithms! If you've ever wondered how
computers solve problems, from sorting a list of names to finding the best route on a
map, you're in the right place. This book is designed for absolute beginners, so we'll
start from the very basics and build up your understanding step by step.

What is an Algorithm?

Imagine you want to bake a cake. You need a recipe, right? The recipe lists the
ingredients and provides a sequence of steps: mix the flour and sugar, add eggs, bake
for 30 minutes, etc. If you follow these steps correctly, you end up with a cake.

In simple terms, an algorithm is just like a recipe, but for solving a computational
problem. It's a well-defined, step-by-step procedure or set of rules designed to
perform a specific task or solve a particular problem. An algorithm takes some input,
follows a sequence of instructions, and produces an output.

For example, if the problem is to find the largest number in a list of numbers (the
input), an algorithm might specify the following steps:
1. Assume the first number is the largest one you've seen so far.
2. Look at the next number in the list.
3. If this number is larger than the largest one seen so far, update your record of the
largest number.
4. Repeat steps 2 and 3 for all remaining numbers in the list.
5. The number recorded at the end is the largest number in the list (the output).

This sequence of steps is precise, unambiguous, and guaranteed to find the largest
number for any given list.
Characteristics of an Algorithm

Not every set of instructions qualifies as an algorithm. To be considered a valid

algorithm, a procedure must possess the following characteristics:

1. Input: An algorithm must have zero or more well-defined inputs. These

are the values or data on which the algorithm operates. In our cake recipe,
the inputs are the ingredients (flour, sugar, eggs). In the largest number
example, the input is the list of numbers.

2. Output: An algorithm must produce at least one well-defined output.

This is the result of the computation, the solution to the problem. For the
cake recipe, the output is the baked cake. For the largest number problem,
the output is the single largest number.

3. Finiteness: An algorithm must terminate after a finite number of steps for

all possible inputs. It shouldn't go on forever. The cake recipe has a final
step, and the largest number algorithm stops once it has checked all
numbers in the list.

4. Definiteness: Each step of an algorithm must be clear, precise, and

unambiguous. There should be no uncertainty about what to do next.
Instructions like "add a pinch of salt" might be okay in cooking, but in
algorithms, every instruction must be exact (e.g., "add 5 grams of salt").

5. Effectiveness: Every instruction must be basic enough that it can, in

principle, be carried out by a person using only pencil and paper. It means
the steps should be feasible and executable. Operations should be basic
and achievable.

Understanding these characteristics helps us distinguish a true algorithm from a

vague set of instructions and is crucial for designing correct and reliable solutions to
computational problems.
Why Analyze Algorithms?

Imagine you have two different recipes for baking a cake. One takes 30 minutes, and
the other takes 3 hours. Both produce a delicious cake, but which recipe would you
prefer if you're short on time? Probably the faster one.

Similarly, for a given computational problem, there might be multiple algorithms that
can solve it. For instance, there are many ways to sort a list of numbers. Why should
we care which algorithm we use if they all produce the correct sorted list?

The answer lies in efficiency. Different algorithms can consume different amounts of
resources. The primary resources we care about are:

1. Time: How long does the algorithm take to complete? We usually want
algorithms that run faster, especially when dealing with large amounts of
data.

2. Space: How much memory (computer memory) does the algorithm

require while running? We often prefer algorithms that use less memory.

Analyzing algorithms helps us compare different approaches and choose the one that
is most efficient for our needs in terms of time and space. It allows us to predict how
an algorithm will perform, especially as the size of the input grows. For example, an
algorithm might be fast for sorting 10 numbers but incredibly slow for sorting 1
million numbers. Analysis helps us understand this scaling behavior.

Furthermore, analyzing algorithms helps us understand the core difficulty of a

problem and provides insights into how to design better, more efficient solutions. It's
a fundamental skill for anyone involved in computer science and software
development.

Analysis of Algorithms

Now that we know what algorithms are and why we need them, let's dive into how
we analyze their efficiency. As mentioned, we primarily focus on time and space
resources. Measuring the exact time an algorithm takes can be tricky because it
depends on factors like the specific computer, the programming language used, and
even other programs running simultaneously. Similarly, exact memory usage can
vary.

To overcome this, we use a more abstract and theoretical approach called

Asymptotic Analysis. Instead of measuring exact time or space, we focus on how
the resource requirements (time or space) grow as the size of the input increases. The
input size (often denoted by 'n') could be the number of items in a list to be sorted,
the number of nodes in a graph, etc.

Asymptotic analysis allows us to classify algorithms based on their growth rate,

providing a high-level comparison that is independent of specific hardware or
software implementations.

Asymptotic Notations (Complexity Bounds)

We use special notations to describe the growth rate of an algorithm's resource usage.
These notations define bounds on the function representing the algorithm's
complexity.

1. Big-O Notation (O): Upper Bound

◦ Concept: Big-O notation gives the worst-case scenario or an

upper bound on the growth rate. If an algorithm has a time
complexity of O(n²), it means that its execution time grows no
faster than a quadratic function of the input size 'n', for large
enough 'n'. It might perform better, but it won't perform worse
than this bound asymptotically.

◦ Example: Imagine searching for an item in an unsorted list of

'n' items. In the worst case, you might have to check every
single item (if the item is the last one or not present at all). The
number of checks is proportional to 'n'. So, the time
complexity is O(n), meaning the time grows linearly with the
input size.

2. Omega Notation (Ω): Lower Bound

◦ Concept: Omega notation provides a lower bound on the

growth rate, representing the best-case scenario. If an
algorithm is Ω(n), it means its execution time grows at least as
 fast as a linear function of 'n' for large 'n'. It won't perform
better than this bound asymptotically.

◦ Example: For the same search in an unsorted list, even in the

best case, you have to look at least at the first item. If the
problem requires processing every item (like finding the sum
of all elements), the best case might still involve looking at all
'n' items. If an algorithm must examine every input element, its
time complexity is at least Ω(n).

3. Theta Notation (Θ): Tight Bound

◦ Concept: Theta notation provides a tight bound, meaning the

growth rate is bounded both from above and below by the
same function. If an algorithm is Θ(n), its execution time
grows linearly with 'n' in both the best and worst cases (within
constant factors). This gives the most precise description of
the algorithm's asymptotic behavior.

◦ Example: If an algorithm always requires exactly 'n' steps to

process an input of size 'n' (like printing all elements in a list),
its complexity is Θ(n).

Common Growth Rates (from fastest to slowest growth):

* O(1): Constant time (independent of input size)
* O(log n): Logarithmic time
* O(n): Linear time
* O(n log n): Linearithmic time
* O(n²): Quadratic time
* O(n³): Cubic time
* O(2n): Exponential time
* O(n!): Factorial time

Algorithms with polynomial time complexity (like O(n), O(n²), O(n³)) are generally
considered efficient, while those with exponential or factorial complexity become
impractical very quickly as 'n' increases.
Best, Average, and Worst-Case Behavior
When analyzing an algorithm, we often consider three scenarios:

• Best Case: The input configuration for which the algorithm runs fastest.
(Described by Ω notation)

• Worst Case: The input configuration for which the algorithm runs
slowest. This is often the most important analysis because it gives a
guarantee on the maximum time required. (Described by O notation)

• Average Case: The expected performance averaged over all possible

inputs. This can be complex to calculate as it requires assumptions about
the distribution of inputs. (Often described using O or Θ notation, but the
analysis is different)

For example, consider inserting an element into a sorted array. The best case is
inserting at the end (O(1) if space is available), the worst case is inserting at the
beginning (requiring shifting all elements, O(n)), and the average case involves
shifting about half the elements (also O(n)).

Performance Measurements
While asymptotic analysis gives theoretical bounds, sometimes we need practical
performance measurements:

• Benchmarking: Running the algorithm on specific hardware with

representative inputs and measuring the actual execution time and
memory usage. This gives concrete performance data but is specific to the
test environment.

• Profiling: Using special tools (profilers) to analyze which parts of the

algorithm consume the most time or memory. This helps identify
bottlenecks for optimization.

Theoretical analysis and practical measurement complement each other.

Time and Space Trade-offs
Often, you can design an algorithm to be faster by using more memory, or make it
use less memory at the cost of taking more time. This is known as the time-space
trade-off.

• Example: Suppose you want to count the frequency of each word in a

large text file.
◦ Time-Optimized: You could use a hash map (a data structure)
to store each unique word encountered and its count. Looking
up or updating the count for a word is very fast (close to O(1)
on average). However, this requires memory proportional to
the number of unique words.

◦ Space-Optimized: You could process the text multiple times.

For each unique word, scan the entire text again to count its
occurrences. This uses very little extra memory but takes
much longer (potentially O(m*n), where 'm' is unique words
and 'n' is total words).

Choosing the right balance depends on the specific constraints of the problem, such
as available memory and required response time.

Analysis of Recursive Algorithms

Many algorithms, especially those using strategies like divide-and-conquer, are

naturally expressed recursively. A recursive algorithm is one that calls itself to solve
smaller instances of the same problem.

Analyzing the complexity of recursive algorithms involves recurrence relations. A

recurrence relation is an equation or inequality that describes a function in terms of
its value on smaller inputs.

For example, the time complexity T(n) of a recursive algorithm might be expressed
as:
T(n) = a * T(n/b) + f(n)
Where:
* T(n) is the time for input size 'n'.
* 'a' is the number of recursive calls made.
* T(n/b) is the time taken by each recursive call on a subproblem of size n/b.
* f(n) is the time taken by the non-recursive part (e.g., dividing the problem,
combining results).

We need methods to solve these recurrence relations to find the asymptotic

complexity (like O, Θ, Ω).

1. Substitution Method
• Concept: This is like mathematical induction. We guess a form for the
solution (e.g., T(n) = O(n log n)) and then use the recurrence relation and
mathematical induction to prove our guess is correct.

• Steps:
1. Guess the form of the solution.

2. Verify the guess using mathematical induction:

▪ Base Case: Show the guess holds for small values
of 'n'.

▪ Inductive Step: Assume the guess holds for all

values smaller than 'n' (specifically for n/b in our
example) and substitute this assumption back into
the recurrence relation. Show that this implies the
guess also holds for 'n'.

• Challenge: Making a good initial guess can be difficult. Experience and

intuition help.

2. Recursion Tree Method

• Concept: This method visualizes the recursion as a tree. Each node
represents the cost incurred at a particular level of recursion. We sum up
the costs at each level and then sum the costs across all levels to get the
total cost.
 • Steps:
1. Draw the recursion tree. The root represents the initial call
T(n) with cost f(n).

2. The children of the root represent the 'a' recursive calls T(n/b),
each with their own cost f(n/b).

3. Continue expanding the tree until the leaf nodes represent the
base cases of the recursion.

4. Calculate the cost at each level by summing the costs of all

nodes at that level.

5. Sum the costs of all levels to get the total cost T(n).

• Benefit: Helps visualize the work done and can guide the guess for the
substitution method.

• Example (Merge Sort): T(n) = 2T(n/2) + Θ(n)

◦ Level 0: Cost = cn

◦ Level 1: 2 nodes, each cost c(n/2). Total cost = 2 * c(n/2) = cn

◦ Level 2: 4 nodes, each cost c(n/4). Total cost = 4 * c(n/4) = cn

◦ ... Level log2n: n nodes, each cost c(1). Total cost = n * c(1) =
cn

◦ Total cost = Sum of costs at each level = cn * (number of

levels) = cn * (log2n + 1) = Θ(n log n).

3. Master's Theorem
• Concept: Provides a "cookbook" style solution for recurrence relations of
the form T(n) = a * T(n/b) + f(n), where a ≥ 1 and b > 1 are constants, and
f(n) is an asymptotically positive function.

• How it works: It compares the growth rate of the non-recursive part f(n)
with the function nlogb a. There are three cases:
◦ Case 1: If f(n) = O(nlogb a - ε) for some constant ε > 0 (i.e.,
f(n) grows polynomially slower than nlogb a), then T(n) =
Θ(nlogb a).
 ◦ Case 2: If f(n) = Θ(nlogb a), then T(n) = Θ(nlogb a * log n).

◦ Case 3: If f(n) = Ω(nlogb a + ε) for some constant ε > 0 (i.e.,

f(n) grows polynomially faster than nlogb a), AND if a * f(n/b)
≤ c * f(n) for some constant c < 1 and sufficiently large n (this
is called the regularity condition), then T(n) = Θ(f(n)).

• Benefit: Provides a quick way to solve many common recurrence

relations without resorting to substitution or recursion trees.

• Limitation: Doesn't cover all possible recurrence relations. There are

gaps between the cases (e.g., if f(n) grows slower, but not polynomially
slower).

Example (Using Master Theorem for Merge Sort):

Recurrence: T(n) = 2T(n/2) + Θ(n)
Here, a = 2, b = 2, f(n) = Θ(n).
Calculate nlogb a = nlog2 2 = n1 = n.
Compare f(n) = Θ(n) with nlogb a = n.
Since f(n) = Θ(nlogb a), we are in Case 2.
Therefore, T(n) = Θ(nlogb a * log n) = Θ(n log n).

Examples of Analysis

Let's solidify these concepts with a couple of simple examples.

Example 1: Linear Search

* Problem: Find if an element x exists in an unsorted array A of size n .
* Algorithm: Iterate through the array from the first element to the last, comparing
each element with x . If x is found, return true. If the end of the array is reached
without finding x , return false.
* Analysis:
* Input Size: n (number of elements in the array).
* Basic Operation: Comparison.
* Best Case: x is the first element. Only 1 comparison needed. Time complexity =
Ω(1).
* Worst Case: x is the last element, or x is not in the array. n comparisons
needed. Time complexity = O(n).
* Average Case: Assuming x is equally likely to be at any position, on average, we
expect to check n/2 elements. Time complexity = O(n).
* Overall (Worst-Case): The most common way to represent complexity is using
the worst-case upper bound, so we say Linear Search is O(n).
* Space Complexity: Requires a constant amount of extra space (for loop index,
maybe a flag). Space complexity = O(1).

Example 2: Constant Time Operation

* Problem: Access the element at index i in an array A .
* Algorithm: Directly access A[i] .
* Analysis:
* Input Size: Can be considered n (size of array), but the operation itself doesn't
depend on n .
* Basic Operation: Array indexing.
* Best, Worst, Average Case: Accessing an element by its index takes the same
amount of time regardless of the array size or the index value (assuming the index is
valid). Time complexity = Θ(1).
* Space Complexity: No extra space needed. Space complexity = O(1).

Practice Questions (University Exam

Style)

1. Question: Define algorithm and list its essential characteristics. Explain

why analyzing algorithm efficiency is important, discussing the concepts
of time and space complexity.
Answer: An algorithm is a well-defined, step-by-step procedure for
solving a specific computational problem. Its characteristics are: Input
(zero or more), Output (at least one), Finiteness (terminates), Definiteness
(unambiguous steps), and Effectiveness (feasible steps). Analyzing
efficiency is crucial for comparing different algorithms that solve the
same problem and choosing the one that performs best, especially for
large inputs. Time complexity measures how the execution time grows
with input size, while space complexity measures how memory usage
grows. Efficient algorithms save computational resources and provide
faster results.
 2. Question: Solve the following recurrence relation using the Master
Theorem: T(n) = 9T(n/3) + n². Clearly state which case of the Master
Theorem applies and why.
Answer: The recurrence is T(n) = 9T(n/3) + n². Here, a = 9, b = 3, and
f(n) = n². First, calculate nlogb a = nlog3 9 = nlog3 3² = n². Now, compare
f(n) = n² with nlogb a = n². Since f(n) = Θ(nlogb a), this falls under Case 2
of the Master Theorem. Therefore, the solution is T(n) = Θ(nlogb a * log
n) = Θ(n² log n).

Further Learning: Video Resources

For visual explanations of the concepts discussed in this chapter, check out these
helpful videos:

• Topic: Introduction to Asymptotic Notations (Big O, Omega, Theta)

• Suggested Video: https://www.youtube.com/watch?v=Vg5_g_cH31Y

(Knowledge Gate - Asymptotic Notations)

• Topic: Master Theorem for Solving Recurrence Relations

• Suggested Video: https://www.youtube.com/watch?v=OynWkEj0S-s

(Abdul Bari - Master Theorem)

(Note: These videos provide good overviews. You can explore other videos from
channels like Abdul Bari, Knowledge Gate, or Gate Smashers for more examples and
related topics like recursion tree and substitution methods.)
Chapter 2: Fundamental Algorithmic
Strategies
In the previous chapter, we learned what algorithms are and how to analyze their
efficiency. Now, let's explore some common blueprints or strategies used to design
algorithms. Think of these strategies as general approaches or techniques that can be
adapted to solve a wide variety of problems. Understanding these fundamental
strategies will equip you with a powerful toolkit for tackling new computational
challenges.

Introduction to Algorithmic Paradigms

An algorithmic paradigm or strategy is a high-level approach or method used to find

solutions to problems. It's a template that can guide the development of an algorithm
for a specific problem. While some problems might be solvable using multiple
strategies, often one strategy fits better or leads to a more efficient solution than
others.

In this chapter, we will delve into several key paradigms:

• Brute-Force: The most straightforward approach, trying all possibilities.

• Greedy: Making the locally optimal choice at each step hoping to find a
global optimum.

• Dynamic Programming: Breaking down a problem into overlapping

subproblems and solving each subproblem only once, storing the results.

• Branch-and-Bound: Systematically searching for a solution by

exploring branches of a search tree, pruning branches that cannot lead to
an optimal solution.

• Backtracking: Exploring potential solutions incrementally, abandoning a

path (backtracking) as soon as it's determined that it cannot lead to a valid
solution.
We will also touch upon Heuristics, which are practical approaches that aim for a
good-enough solution quickly, especially when finding the perfect optimal solution is
too slow or complex.

Let's start with the most basic strategy: Brute-Force.

Brute-Force Strategy

Concept:
The Brute-Force strategy is often the simplest and most direct way to solve a
problem. It works by systematically enumerating all possible candidates for the
solution and checking whether each candidate satisfies the problem's statement. If
there's a solution, the brute-force approach is generally guaranteed to find it,
provided you consider all possibilities correctly.

Think of it like trying every key on a keychain until you find the one that opens the
lock. It might not be the fastest way, but it's straightforward and guaranteed to work
if the right key is there.

When to Use:
* When the problem size is small, and the total number of candidate solutions is
manageable.
* As a starting point for developing more sophisticated algorithms.
* To solve simple problems or subproblems within a more complex algorithm.
* When simplicity of implementation is more important than speed.

Drawback:
The main disadvantage of brute-force is its potential inefficiency. For many
problems, the number of candidate solutions grows extremely rapidly with the input
size (e.g., exponentially or factorially), making the brute-force approach impractical
for large inputs.

Example: Finding the Maximum Element in a List

We already saw a simple brute-force approach in Chapter 1: finding the largest
number in a list. The algorithm iterates through all elements, comparing each to the
current maximum found so far. It checks every possibility (each element as the
potential maximum) to guarantee finding the true maximum.
Example: The Traveling Salesperson Problem (TSP) - Brute-Force Approach

• Problem Overview: Imagine a salesperson who needs to visit a set of

cities exactly once and return to the starting city. The goal is to find the
shortest possible route (tour) that covers all cities.

• Brute-Force Idea: The most direct way is to list all possible tours
(permutations of cities), calculate the total distance for each tour, and then
select the tour with the minimum distance.

• Theoretical Steps (Brute-Force TSP):

1. Start with a list of all cities to be visited (let's say n cities).

2. Choose a starting city (it doesn't matter which one, as it's a

cycle).

3. Generate every possible sequence (permutation) of the

remaining n-1 cities.

4. For each sequence, form a complete tour by adding the

starting city at the beginning and end of the sequence.

5. Calculate the total distance for the current tour by summing

the distances between consecutive cities in the tour, including
the distance from the last city back to the starting city.

6. Keep track of the shortest tour found so far and its total
distance.

7. Compare the distance of the current tour with the minimum

distance found so far. If the current tour is shorter, update the
minimum distance and store this tour as the best one found
yet.

8. Repeat steps 4-7 for all possible sequences generated in step 3.

9. After checking all sequences, the stored best tour is the

optimal solution.

• Pseudocode: Brute-Force TSP

function BruteForceTSP(cities):
n = number of cities
if n <= 1:
return 0 // No distance or trivial case

// Assume cities[0] is the starting city

other_cities = cities[1 to n-1]
min_distance = infinity
best_tour = null

// Generate all permutations of other_cities

permutations = generate_all_permutations(other_cities)

for each permutation p in permutations:

current_distance = 0
current_city = cities[0]

// Calculate distance for the tour: start -> p -> start

// Distance from start city to first city in permutation
current_distance += distance(current_city, p[0])

// Distance between cities in the permutation

for i from 0 to length(p) - 2:
current_distance += distance(p[i], p[i+1])

// Distance from last city in permutation back to start city

current_distance += distance(p[length(p)-1], cities[0])

// Check if this tour is the shortest found so far

if current_distance < min_distance:
min_distance = current_distance
// Construct the full tour for storage if needed
best_tour = [cities[0]] + p + [cities[0]]

return min_distance // or best_tour

// Helper function (conceptual)

function generate_all_permutations(items):
// Returns a list of all possible orderings of items
...

• Explanation of Variables:

◦ cities : List of cities to visit.

 ◦ n : Total number of cities.

◦ other_cities : List of cities excluding the starting city.

◦ min_distance : Stores the shortest tour distance found so

far.

◦ best_tour : Stores the sequence of cities in the shortest

tour found so far.

◦ permutations : List containing all possible orderings of

other_cities .

◦ p : A single permutation (ordering) of other_cities .

◦ current_distance : The total distance of the tour

corresponding to permutation p .

◦ current_city : Used to track the city from which

distance is calculated (starts with cities[0] here).

◦ distance(city1, city2) : A function assumed to

return the distance between two cities.

◦ generate_all_permutations(items) : A
conceptual helper function to generate permutations.

• Time Complexity Analysis (Brute-Force TSP):

◦ If there are n cities, we fix one as the start/end point.

◦ We need to generate all possible orderings (permutations) of

the remaining n-1 cities.

◦ The number of permutations of n-1 items is (n-1)!

(factorial).

◦ For each permutation, we calculate the tour length, which

involves summing n distances (takes O(n) time).

◦ Total time complexity = O((n-1)! * n) = O(n!).

◦ Factorial complexity grows incredibly fast! For just 10 cities,

9! = 362,880 permutations. For 20 cities, 19! is enormous.
This approach quickly becomes computationally infeasible for
even moderately sized problems.
 • Space Complexity: We need space to store the permutations and the best
tour found so far. Generating all permutations might require significant
space, potentially O(n * n!), although iterative generation can reduce this.
Storing the best tour takes O(n) space.

• University Exam Style Question:

◦ Question: Describe the brute-force approach to solving the

Traveling Salesperson Problem (TSP). Explain its time
complexity and why it is generally considered impractical for
large instances.

◦ Answer: The brute-force approach for TSP involves

generating all possible tours (permutations of visiting the
cities, starting and ending at the same city), calculating the
total distance for each tour, and selecting the tour with the
minimum distance. If there are 'n' cities, there are (n-1)!
possible permutations to check after fixing a starting city.
Calculating the distance for each tour takes O(n) time.
Therefore, the total time complexity is O(n * (n-1)!) or O(n!).
This factorial growth makes the algorithm extremely slow and
computationally infeasible for anything more than a small
number of cities (e.g., >15-20), rendering it impractical for
most real-world TSP instances.

While brute-force is simple to understand, its inefficiency, especially for problems

like TSP, motivates the need for more clever algorithmic strategies, which we will
explore next.

Greedy Strategy

Concept:
The Greedy strategy is another intuitive approach to problem-solving. As the name
suggests, it involves making the choice that seems best at the current moment,
without worrying about the future consequences of that choice. At each step, the
algorithm selects the option that offers the greatest immediate benefit or appears
most promising according to some predefined criterion (the "greedy choice"). The
hope is that by repeatedly making these locally optimal choices, we will end up with
a globally optimal solution for the entire problem.

Think of giving change using the fewest coins possible. You would greedily start by
giving the largest denomination coin (like a quarter) that is less than or equal to the
remaining amount, then repeat with the next largest, and so on. For standard currency
systems, this greedy approach works perfectly.

When to Use:
* Problems exhibiting the Greedy Choice Property: A globally optimal solution can
be arrived at by making a locally optimal choice. In other words, the choice made at
the current step doesn't prevent reaching the overall best solution.
* Problems exhibiting Optimal Substructure: An optimal solution to the problem
contains within it optimal solutions to subproblems (similar to dynamic
programming, but the greedy choice is simpler).
* When we need a relatively simple and often efficient algorithm, even if it doesn't
guarantee the absolute best solution for all problem types (in which case it might be
used as a heuristic).

Drawback:
The main challenge with the greedy strategy is that it doesn't always work! Making
the locally best choice can sometimes lead you down a path where you miss the true
global optimum. It's crucial to prove that the greedy choice property holds for the
specific problem you are trying to solve before relying on a greedy algorithm for
optimality.

Example: Fractional Knapsack Problem

• Problem Overview: You have a knapsack with a maximum weight

capacity (W) and a set of items, each with a specific weight (wi) and
value (vi). Unlike the 0/1 Knapsack problem (where you must take an
item whole or leave it), in the Fractional Knapsack problem, you can take
fractions of items. The goal is to maximize the total value of items (or
fractions of items) placed in the knapsack without exceeding the weight
capacity W.

• Greedy Idea: To maximize the total value, it makes sense to prioritize

items that give the most value per unit of weight. The greedy choice is to
repeatedly take as much as possible of the item with the highest value-to-
weight ratio remaining.
• Theoretical Steps (Greedy Fractional Knapsack):

1. Calculate the value-to-weight ratio (vi / wi) for each item i .

2. Sort the items in descending order based on their value-to-

weight ratios.

3. Initialize the total value in the knapsack to zero and the current
weight to zero.

4. Iterate through the sorted items, starting with the highest ratio.

5. For the current item, determine how much of it can be added

to the knapsack.

6. If the entire item fits (item's weight ≤ remaining capacity), add

the whole item to the knapsack. Update the total value by
adding the item's value and update the current weight by
adding the item's weight.

7. If only a fraction of the item fits, calculate the fraction that

fills the remaining capacity. Add this fraction of the item to the
knapsack. Update the total value by adding the fractional
value (fraction * item's value) and set the current weight to the
maximum capacity W.

8. Stop the process once the knapsack is full (current weight

equals W) or when all items have been considered.

9. The final total value represents the maximum value

achievable.

• Pseudocode: Greedy Fractional Knapsack

function GreedyFractionalKnapsack(items, W):
// Calculate value-to-weight ratios for each item
for each item in items:
item.ratio = item.value / item.weight

// Sort items by ratio in descending order

sort items based on item.ratio (descending)

total_value = 0
current_weight = 0

// Iterate through sorted items

for each item in sorted_items:
// Check if knapsack is already full
if current_weight == W:
break

// Determine weight to take from this item

amount_to_take = min(item.weight, W - current_weight)

// Add value of the taken amount

total_value += amount_to_take * item.ratio
current_weight += amount_to_take

// Return the maximum value

return total_value

• Explanation of Variables:

◦ items : List of item objects, each having value ,

weight , and calculated ratio .

◦ W : Maximum weight capacity of the knapsack.

◦ item.ratio : Value per unit of weight for an item.

◦ sorted_items : The list of items sorted by ratio

descendingly.

◦ total_value : The accumulated value in the knapsack.

◦ current_weight : The accumulated weight in the

knapsack.
 ◦ amount_to_take : The weight of the current item (or
fraction) being added.

• Time Complexity Analysis (Greedy Fractional Knapsack):

◦ Calculating ratios for n items takes O(n) time.

◦ Sorting the items based on ratios takes O(n log n) time (using
an efficient sorting algorithm like Merge Sort or Heap Sort).

◦ The loop iterates through the items at most once, taking O(n)
time.

◦ The dominant step is sorting.

◦ Total time complexity = O(n log n).

• Space Complexity: If sorting is done in-place, the extra space required is

minimal (O(1) beyond storing ratios if needed). If sorting requires extra
space, it might be O(n).

• Optimality: For the Fractional Knapsack problem, this greedy strategy is

proven to yield the optimal solution.

• University Exam Style Question:

◦ Question: Explain the greedy strategy for solving the

Fractional Knapsack problem. Provide the pseudocode for the
algorithm and analyze its time complexity. Does this greedy
approach guarantee an optimal solution for the 0/1 Knapsack
problem? Why or why not?

◦ Answer: The greedy strategy for Fractional Knapsack

calculates the value-to-weight ratio for each item and sorts
them in descending order based on this ratio. It then iterates
through the sorted items, adding as much as possible of the
item with the highest ratio to the knapsack until the capacity is
filled. If an item cannot be added entirely, a fraction of it is
added to fill the remaining capacity. (Include Pseudocode from
above). The time complexity is dominated by the sorting step,
making it O(n log n). This greedy approach does guarantee
optimality for the Fractional Knapsack problem because you
can always maximize value by prioritizing items with the
 highest value density. However, this same greedy strategy
does not guarantee an optimal solution for the 0/1 Knapsack
problem (where items must be taken whole). A
counterexample is a knapsack of capacity 50, with items: A
(value=60, weight=10, ratio=6), B (value=100, weight=20,
ratio=5), C (value=120, weight=30, ratio=4). The greedy
approach takes A and B (total value=160, weight=30). The
optimal solution is B and C (total value=220, weight=50).

Other Greedy Examples:

• Minimum Spanning Tree (MST): Problems like finding the MST in a

graph (a tree connecting all vertices with minimum total edge weight) can
be solved using greedy algorithms.
◦ Prim's Algorithm: Starts from a vertex and greedily grows
the tree by adding the cheapest edge connecting a vertex in the
tree to a vertex outside the tree.

◦ Kruskal's Algorithm: Greedily selects the cheapest available

edge from the entire graph, as long as adding the edge doesn't
form a cycle.
(We will cover these in detail in Chapter 3).

• Dijkstra's Algorithm: Finds the shortest path from a source vertex to all
other vertices in a graph with non-negative edge weights. It greedily
selects the unvisited vertex with the shortest known distance from the
source.

Greedy algorithms are powerful and efficient when applicable, but always remember
to verify if the greedy choice property holds for your specific problem to ensure
optimality.

Dynamic Programming (DP)

Concept:
Dynamic Programming is a powerful algorithmic technique used for solving
complex problems by breaking them down into simpler, overlapping subproblems. It
solves each subproblem only once and stores its result, typically in a table or array, to
avoid redundant computations. When the same subproblem is encountered again, the
stored result is simply retrieved instead of being re-calculated.

This strategy is effective for problems that exhibit two key properties:

1. Overlapping Subproblems: The problem can be broken down into

subproblems that are reused multiple times. A naive recursive approach
would solve these subproblems repeatedly, leading to inefficiency. DP
avoids this by computing each subproblem just once.

2. Optimal Substructure: An optimal solution to the overall problem can

be constructed from the optimal solutions of its subproblems. This allows
us to build up the solution from the bottom (smaller subproblems) to the
top (the original problem).

DP vs. Divide and Conquer:

Both DP and Divide and Conquer break problems into subproblems. However, in
Divide and Conquer (like Merge Sort or Quick Sort), the subproblems are typically
independent and don't overlap. In DP, the subproblems overlap, meaning the same
subproblem might be needed to solve several different larger problems. DP exploits
this overlap by storing results.

DP vs. Greedy:
A Greedy algorithm makes a locally optimal choice at each step without considering
future steps. DP, on the other hand, typically explores multiple options at each step
(by considering solutions to subproblems) and combines them to find the overall
optimal solution. DP is generally more powerful than Greedy but often less intuitive
and potentially more complex in terms of time or space.

General Approach (Bottom-Up):

1. Identify the subproblems.
2. Define a recurrence relation that expresses the solution to a larger problem in
terms of solutions to its smaller subproblems.
3. Determine the base cases for the smallest subproblems.
4. Create a table (e.g., an array or matrix) to store the results of the subproblems.
5. Solve the subproblems in a specific order (usually starting from the base cases and
working upwards) and fill the table.
6. The final entry in the table corresponding to the original problem will contain the
solution.
Example: Fibonacci Sequence
A classic example illustrating overlapping subproblems is calculating the nth
Fibonacci number (F(n) = F(n-1) + F(n-2), with F(0)=0, F(1)=1).
* Naive Recursion: fib(n) calls fib(n-1) and fib(n-2) . fib(n-1)
calls fib(n-2) and fib(n-3) , etc. Notice fib(n-2) is computed twice.
This redundancy grows exponentially.
* DP (Memoization - Top-Down): Use recursion but store the result of fib(k)
the first time it's computed. If called again, return the stored value.
* DP (Tabulation - Bottom-Up): Create an array dp of size n+1 . Set
dp[0]=0 , dp[1]=1 . Then iterate from i=2 to n , calculating dp[i] =
dp[i-1] + dp[i-2] . The answer is dp[n] . This avoids recursion and
computes each Fibonacci number exactly once. Time: O(n), Space: O(n) (can be
optimized to O(1) space).

Example: 0/1 Knapsack Problem

• Problem Overview: You have a knapsack with capacity W and n items,

each with weight wi and value vi. You must decide for each item whether
to include it in the knapsack or not (0/1 choice - no fractions allowed).
The goal is to maximize the total value of items in the knapsack without
exceeding the capacity W.

• Why Greedy Fails: As shown before, the greedy approach based on

value-to-weight ratio doesn't guarantee optimality for the 0/1 version.

• Dynamic Programming Idea: Let dp[i][j] represent the maximum

value that can be achieved using only the first i items (from item 1 to
item i ) with a maximum knapsack capacity of j .
Our goal is to find dp[n][W] .

• Recurrence Relation: Consider the i -th item (with value v[i] and
weight w[i] ). When deciding the value for dp[i][j] , we have two
choices for this i -th item:

1. Don't include item i : In this case, the maximum value is

the same as the maximum value achievable using the first
i-1 items with capacity j . So, the value is
dp[i-1][j] .
 2. Include item i : This is only possible if the item's weight
w[i] is less than or equal to the current capacity j ( w[i]
<= j ). If we include item i , its value v[i] is added. The
remaining capacity becomes j - w[i] , and we need to
find the maximum value we could achieve using the first
i-1 items with this remaining capacity. So, the value is
v[i] + dp[i-1][j - w[i]] .

We choose the option that gives the maximum value. Therefore, the
recurrence relation is:
dp[i][j] = dp[i-1][j] (if w[i] > j - item i cannot fit)
dp[i][j] = max(dp[i-1][j], v[i] + dp[i-1][j -
w[i]]) (if w[i] <= j )

• Base Cases:

◦ dp[0][j] = 0 for all j (no items, no value).

◦ dp[i][0] = 0 for all i (no capacity, no value).

• Theoretical Steps (DP 0/1 Knapsack - Bottom-Up):

1. Create a 2D table dp of size (n+1) x (W+1) , where n

is the number of items and W is the capacity.

2. Initialize the first row ( dp[0][j] ) and first column

( dp[i][0] ) to 0.

3. Iterate through the items from i = 1 to n .

4. For each item i , iterate through possible capacities j from

1 to W .

5. Inside the inner loop, apply the recurrence relation:

▪ Get the weight w[i] and value v[i] of the
current item i (adjusting for 0-based or 1-based
indexing as needed).

▪ If w[i] > j (item i is heavier than current

capacity j ):
Set dp[i][j] = dp[i-1][j] (cannot
include item i ).
 ▪ Else ( w[i] <= j ):
Calculate value without item i :
value_without = dp[i-1][j] .
Calculate value with item i : value_with =
v[i] + dp[i-1][j - w[i]] .
Set dp[i][j] = max(value_without,
value_with) .

6. After filling the entire table, the value dp[n][W] contains

the maximum possible value that can be carried in the
knapsack.

7. (Optional) To find the actual items included, you can

backtrack through the dp table starting from dp[n][W] .

• Pseudocode: Dynamic Programming 0/1 Knapsack

function DPKnapsack01(weights, values, n, W):
// Create DP table: dp[i][j] = max value using items 1..i with capac
dp = create_2d_array(n + 1, W + 1)

// Initialize base cases (row 0 and column 0)

for j from 0 to W:
dp[0][j] = 0
for i from 0 to n:
dp[i][0] = 0

// Fill the DP table

for i from 1 to n:
for j from 1 to W:
// Get weight and value of item i (adjust index if 0-based)
current_weight = weights[i]
current_value = values[i]

// Apply recurrence relation

if current_weight > j:
// Item i is too heavy
dp[i][j] = dp[i-1][j]
else:
// Option 1: Don't include item i
value_without_i = dp[i-1][j]
// Option 2: Include item i
value_with_i = current_value + dp[i-1][j - current_weigh
// Choose the maximum
dp[i][j] = max(value_without_i, value_with_i)

// Return the result stored in the bottom-right cell

return dp[n][W]

• Explanation of Variables:

◦ weights : Array/List of weights of the n items (e.g.,

weights[i] is weight of item i ).

◦ values : Array/List of values of the n items (e.g.,

values[i] is value of item i ).

◦ n : The total number of items considered.

◦ W : The maximum capacity of the knapsack.

 ◦ dp : A 2D array (table) where dp[i][j] stores the
maximum value achievable using items from 1 up to i with
a knapsack capacity of exactly j .

◦ i : Loop variable representing the current item being

considered (from 1 to n ).

◦ j : Loop variable representing the current knapsack capacity

being considered (from 1 to W ).

◦ current_weight : Weight of the i -th item.

◦ current_value : Value of the i -th item.

◦ value_without_i : Max value achievable using items 1

to i-1 with capacity j .

◦ value_with_i : Max value achievable if item i is

included (its value plus max value from items 1 to i-1 with
remaining capacity j - current_weight ).

• Time Complexity Analysis (DP 0/1 Knapsack):

◦ We fill a table of size (n+1) x (W+1) .

◦ Each cell dp[i][j] is computed in constant time O(1)

using the recurrence relation.

◦ Total time complexity = O(n * W).

◦ Note: This is called pseudo-polynomial time because it

depends on the numeric value of the capacity W , not just the
number of items n . If W is very large (e.g., exponential in
n ), this can be slow.

• Space Complexity: We need to store the dp table, which requires O(n *

W) space. (This can be optimized to O(W) space if we only need the final
value and not the items themselves, by noticing that dp[i][j] only
depends on the previous row dp[i-1] ).

• University Exam Style Question:

◦ Question: Explain the principle of Dynamic Programming

using the 0/1 Knapsack problem. Provide the recurrence
relation, explain the meaning of the DP state dp[i][j] ,
 and analyze the time and space complexity of the bottom-up
tabular approach.

◦ Answer: Dynamic Programming solves problems by breaking

them into overlapping subproblems and storing their solutions
to avoid recomputation. For the 0/1 Knapsack problem,
dp[i][j] represents the maximum value achievable using
the first i items with a knapsack capacity of j . The
recurrence relation is: dp[i][j] = dp[i-1][j] if
w[i] > j , and dp[i][j] = max(dp[i-1][j],
v[i] + dp[i-1][j - w[i]]) if w[i] <= j . This
considers either not taking item i or taking item i if
possible. The bottom-up approach fills an
(n+1) x (W+1) table. Since each cell takes O(1) to
compute, the total time complexity is O(nW). The space
complexity is also O(nW) for the table (optimizable to O(W)).

Dynamic Programming for TSP:

While the brute-force O(n!) approach for TSP is infeasible, Dynamic Programming
offers a significantly better (though still exponential) solution, known as the Held-
Karp algorithm. It uses a state like dp[S][i] , representing the shortest path
starting at the origin, visiting all cities in the set S , and ending at city i . The
complexity is roughly O(n² * 2n), which is much better than O(n!) and feasible for
n up to around 20-25.

Dynamic Programming is a cornerstone of algorithm design, applicable to a vast

range of optimization problems, from sequence alignment in bioinformatics to
shortest path calculations and resource allocation problems.

Branch-and-Bound (B&B)

Concept:
Branch-and-Bound is an algorithm design paradigm primarily used for solving
optimization problems (finding the minimum or maximum value). It systematically
searches the entire space of possible solutions, but it does so more intelligently than
brute-force by pruning away large parts of the search space that cannot possibly
contain the optimal solution.
Imagine searching for the cheapest flight. You might explore different routes
(branches), but if a partial route already costs more than the cheapest complete flight
found so far, you abandon exploring that route further (bounding).

B&B explores the solution space by building a state space tree. The root represents
the initial problem, and nodes represent partial solutions. The "branching" part
involves dividing a problem into smaller subproblems (creating child nodes). The
"bounding" part involves calculating a bound (an estimate) for the best possible
solution achievable from a given node. If this bound indicates that the subproblem
rooted at this node cannot lead to a better solution than the best one already found,
the node (and the entire subtree below it) is pruned, meaning it's not explored further.

Key Components:
1. Branching: A rule to divide a problem/node into smaller subproblems/child
nodes.
2. Bounding Function: A function that calculates a lower bound (for minimization
problems) or an upper bound (for maximization problems) on the optimal solution
value for any solution derived from a given node.
3. Search Strategy: How to explore the state space tree (e.g., Best-First Search,
Depth-First Search, Breadth-First Search). Best-First Search, where the most
promising node (based on the bound) is explored next, is common.
4. Pruning Rule: If the bound for a node is worse than the value of the best
complete solution found so far (called the incumbent), prune the node.

When to Use:
* Optimization problems (minimization or maximization).
* Problems where a good bounding function can be calculated efficiently.
* Often applied to NP-hard problems (like TSP, Knapsack, Integer Programming)
where finding the exact solution is hard, but B&B can often find the optimal solution
much faster than brute-force, although worst-case performance can still be
exponential.

Example: Traveling Salesperson Problem (TSP) - Branch-and-Bound Approach

• Problem: Find the shortest tour visiting n cities exactly once and
returning to the start.

• Branch-and-Bound Idea: We build a state space tree where nodes

represent partial tours. We need a way to calculate a lower bound on the
cost of any complete tour that can be built starting from a partial tour.
• Lower Bound Calculation (Example): A simple lower bound for a node
representing a partial path ending at city u can be calculated as: (Cost of
the partial path so far) + (Cost of the cheapest edge leaving city u to an
unvisited city) + (Sum of the cheapest edges leaving each remaining
unvisited city to some other unvisited city or the start city) + (Cost of the
cheapest edge from an unvisited city back to the start city). A more
common and often tighter bound involves minimum spanning trees or
assignment problem relaxations, but the core idea is to estimate the
minimum possible additional cost needed to complete the tour.

• Theoretical Steps (Branch-and-Bound TSP - Conceptual):

1. Initialize the minimum tour cost found so far ( min_cost )

to infinity.

2. Create the root node of the state space tree, representing the
starting city with a partial path containing only the start city.
Calculate its lower bound (e.g., sum of minimum outgoing
edges from each city).

3. Maintain a list of active (live) nodes, initially containing only

the root node. Often, a priority queue is used, ordered by the
lower bound (Best-First Search).

4. While the list of active nodes is not empty:

5. Select the most promising node (e.g., the one with the smallest
lower bound) from the list. Let this be node P representing a
partial tour ending at city u .

6. If the lower bound of node P is greater than or equal to

min_cost , prune this node (discard it) and go back to step
4 (no better solution can be found from here).

7. Branching: For each unvisited city v adjacent to u :

a. Create a new child node C representing the partial tour
extended by adding city v .
b. Calculate the cost of this new partial tour.
c. If this new partial tour includes all cities:
i. Add the cost of returning to the starting city to complete the
tour.
ii. If this complete tour cost is less than min_cost , update
 min_cost to this new cost and store this tour as the best
one found so far.
iii. Prune this path (no need to branch further).
d. Else (the tour is still partial):
i. Calculate the lower bound for the child node C (estimating
the minimum cost to complete the tour from city v ).
ii. If the lower bound for C is less than min_cost , add
node C to the list of active nodes.

8. Remove node P from the active list (it has been processed).

9. After the active node list is empty, the stored min_cost

and its corresponding tour represent the optimal solution.

• Pseudocode: Branch-and-Bound TSP (Conceptual Best-First)

function BranchAndBoundTSP(cities):
n = number of cities
min_cost = infinity
best_tour = null

// Priority Queue stores nodes: (lower_bound, current_cost, partial_

// Ordered by lower_bound (ascending for minimization)
pq = new PriorityQueue()

// Create root node (start at city 0, cost 0)

initial_tour = [cities[0]]
initial_cost = 0
initial_bound = calculate_lower_bound(initial_tour, cities)
pq.add((initial_bound, initial_cost, initial_tour))

while pq is not empty:

(current_bound, current_cost, current_tour) = pq.remove_min()

// Pruning based on bound vs best solution found

if current_bound >= min_cost:
continue // Prune this branch

last_city = current_tour.last()

// Check if tour is complete

if length(current_tour) == n:
final_cost = current_cost + distance(last_city, cities[0])
if final_cost < min_cost:
min_cost = final_cost
best_tour = current_tour + [cities[0]]
continue // Found a complete tour, prune

// Branching: Explore neighbors

for each city v not in current_tour:
new_tour = current_tour + [v]
new_cost = current_cost + distance(last_city, v)

// Calculate lower bound for the new partial tour

new_bound = calculate_lower_bound(new_tour, cities)

// Pruning based on bound before adding to queue

if new_bound < min_cost:
pq.add((new_bound, new_cost, new_tour))
 return min_cost, best_tour

// Helper function (needs specific implementation)

function calculate_lower_bound(partial_tour, all_cities):
// Calculates a lower bound on the cost of completing the tour
// ... (e.g., using MST on remaining nodes or minimum outgoing edges
...

• Explanation of Variables:

◦ cities : List of cities.

◦ n : Number of cities.

◦ min_cost : Cost of the best complete tour found so far

(incumbent).

◦ best_tour : Sequence of cities in the best tour found so

far.

◦ pq : Priority queue storing active nodes (partial solutions)

prioritized by their lower bound.

◦ initial_tour , initial_cost ,
initial_bound : State of the root node.

◦ current_bound , current_cost ,
current_tour : State of the node being processed.

◦ last_city : The last city added to the current_tour .

◦ final_cost : Cost of a completed tour.

◦ v : A potential next city to visit.

◦ new_tour , new_cost , new_bound : State of a child

node created by adding city v .

◦ calculate_lower_bound : Function to estimate the

minimum cost to complete a tour from a partial one.

◦ distance(city1, city2) : Function returning

distance between two cities.
• Time Complexity Analysis (Branch-and-Bound TSP):

◦ Worst Case: In the worst case, if the bounds are not effective,
B&B might explore the entire state space tree, similar to brute-
force, leading to exponential complexity (e.g., O(n!) or O(n² *
2n) depending on the bounding and branching).

◦ Average/Practical Case: The effectiveness heavily depends

on the quality of the lower bound function and the order of
exploration. Good bounds can prune significant portions of the
tree, making B&B much faster than brute-force in practice for
many instances, often finding optimal solutions for moderately
sized problems (e.g., n=40-60) that are intractable for DP or
brute-force.

◦ Deriving a tight theoretical bound is difficult and problem-

instance dependent.

• Space Complexity: Depends on the number of active nodes stored in the

priority queue. In the worst case, this can also be exponential.

• Example: Bin Packing (Branch-and-Bound)

◦ Problem: Given n items of different sizes and an unlimited

supply of bins of capacity C , pack the items into the
minimum number of bins.

◦ B&B Idea: Nodes in the tree represent partial assignments of

items to bins. Branching could involve trying to place the next
item into each existing non-full bin or starting a new bin for it.
A lower bound can be calculated (e.g., the sum of all item
sizes divided by the bin capacity C , rounded up). Pruning
occurs if the number of bins used in a partial solution plus a
lower bound on the bins needed for remaining items exceeds
the best complete solution found so far.

• University Exam Style Question:

◦ Question: Explain the core principles of the Branch-and-

Bound (B&B) strategy. What are the essential components of a
B&B algorithm? How does B&B attempt to improve upon
 brute-force search, particularly in the context of optimization
problems like TSP or Bin Packing?

◦ Answer: Branch-and-Bound is a search strategy for

optimization problems. It explores a state space tree of
possible solutions (branching) but uses a bounding function to
estimate the best possible outcome achievable from a node
(partial solution). If this bound is worse than the best complete
solution found so far (incumbent), the entire subtree rooted at
that node is pruned (bounding), avoiding exhaustive
exploration. Essential components are: a branching rule, a
bounding function, a search strategy (like Best-First), and a
pruning rule. B&B improves on brute-force by intelligently
discarding parts of the search space that cannot contain the
optimal solution, potentially leading to significant
performance gains, although the worst-case complexity can
still be exponential. It relies on finding good, efficiently
computable bounds.

Backtracking

Concept:
Backtracking is another algorithmic technique for solving problems, particularly
constraint satisfaction problems (CSPs), by incrementally building candidates for the
solutions and abandoning a candidate ("backtracking") as soon as it determines that
the candidate cannot possibly be completed to a valid solution.

It explores the set of all possible solutions by trying to extend a partially built
solution, one step at a time. If at any step the partial solution can be extended, the
algorithm proceeds. If the partial solution cannot be extended to a complete valid
solution, or if it violates some problem constraint, the algorithm backtracks: it
undoes the last step and tries an alternative choice at that step. If all alternatives at
the current step have been tried and failed, it backtracks further up to the previous
step.
Think of navigating a maze. You follow a path until you hit a dead end. When you hit
a dead end, you backtrack to the last junction where you had a choice of paths and
try a different path.

Key Idea: Explore choices systematically. If a choice leads to a dead end (violates
constraints or doesn't lead to a solution), undo the choice and try the next available
option.

State Space Tree: Like Branch-and-Bound, Backtracking implicitly builds a state

space tree. Nodes represent partial solutions. The algorithm performs a depth-first
search (DFS) on this tree. When it hits a node from which no valid solution can be
completed, it backtracks to the parent node.

Backtracking vs. Brute-Force: Brute-force generates all possible candidates and

then checks if they are valid solutions. Backtracking is smarter because it builds
candidates incrementally and prunes branches of the search tree as soon as it
determines that a partial candidate cannot lead to a valid solution, thus avoiding the
generation of many invalid candidates.

When to Use:
* Problems where solutions are built step-by-step and constraints can be checked at
each step.
* Constraint Satisfaction Problems (e.g., N-Queens, Sudoku, Map Coloring).
* Finding all possible solutions or a single solution that satisfies given criteria.
* Pathfinding and combinatorial search problems.

Example: N-Queens Problem

• Problem Overview: Place N chess queens on an N x N chessboard

such that no two queens threaten each other. This means no two queens
can be in the same row, same column, or same diagonal.

• Backtracking Idea: Try placing queens one column at a time, starting

from the leftmost column. For each column, try placing a queen in each
row. Before placing a queen in a specific square (row, col), check if it's
safe (i.e., not attacked by any previously placed queens in columns 0 to
col-1). If it's safe, place the queen and recursively try to place a queen in
the next column (col+1). If the recursive call successfully places all
remaining queens, we've found a solution. If the recursive call fails
(returns false), or if no row in the current column is safe, backtrack:
 remove the queen placed in the current column and try the next row in the
same column. If all rows in the current column have been tried
unsuccessfully, return false to the previous column's call.

• Theoretical Steps (Backtracking N-Queens - for column col ):

1. Base Case: If col is greater than or equal to N (all

columns filled), a solution has been found. Return true.

2. Iterate through each row i from 0 to N-1 for the current

column col .

3. Check if placing a queen at position ( i , col ) is safe. This

involves checking the current row i , the upper-left diagonal,
and the lower-left diagonal for any previously placed queens
(in columns 0 to col-1 ).

4. If the position ( i , col ) is safe:

a. Place a queen at ( i , col ) (e.g., mark the board).
b. Recursively call the function to place queens in the next
column ( col + 1 ).
c. If the recursive call returns true (meaning it found a solution
for subsequent columns), then return true (propagate success
upwards).
d. If the recursive call returns false (meaning placing the
queen at ( i , col ) didn't lead to a solution), backtrack:
remove the queen from ( i , col ) (e.g., unmark the board).

5. If the loop finishes without finding a safe row in the current

column col that leads to a solution, return false (no solution
possible from the previous state).

• Pseudocode: Backtracking N-Queens

// Main function to initiate the process
function solveNQueens(N):
board = create_NxN_board_initialized_to_empty()
if solveNQueensUtil(board, 0, N) == false:
print("Solution does not exist")
return false
// If a solution exists, it's stored in 'board'
print_solution(board, N) // Optional: display the board
return true

// Recursive helper function to place queens

function solveNQueensUtil(board, col, N):
// Base Case: If all columns are filled, we found a solution
if col >= N:
return true

// Try placing a queen in each row of the current column 'col'

for i from 0 to N-1:
// Check if placing queen at board[i][col] is safe
if isSafe(board, i, col, N):
// Place the queen
board[i][col] = 1 // Mark position with a queen

// Recur to place queens in the next column

if solveNQueensUtil(board, col + 1, N) == true:
return true // Solution found down this path

// Backtrack: If placing queen in board[i][col] doesn't lead

// to a solution, remove the queen and try next row
board[i][col] = 0 // Unmark the position

// If no row in this column 'col' leads to a solution

return false

// Helper function to check if board[row][col] is safe

function isSafe(board, row, col, N):
// Check this row on the left side (columns 0 to col-1)
for k from 0 to col-1:
if board[row][k] == 1:
return false

// Check upper diagonal on left side

r, c = row, col
while r >= 0 and c >= 0:
 if board[r][c] == 1:
return false
r = r - 1
c = c - 1

// Check lower diagonal on left side

r, c = row, col
while r < N and c >= 0:
if board[r][c] == 1:
return false
r = r + 1
c = c - 1

// If no conflicts found
return true

// Helper function to print the board (optional)

function print_solution(board, N):
...

• Explanation of Variables:

◦ N : The size of the chessboard (N x N) and the number of

queens.

◦ board : A 2D array representing the chessboard.

board[i][j] = 1 if a queen is at row i , col j , else 0.

◦ col : The current column index being considered for placing

a queen.

◦ i : The current row index being checked within column

col .

◦ solveNQueensUtil : The recursive function performing

the backtracking.

◦ isSafe : Helper function checking if placing a queen at

(row, col) is valid given previously placed queens.

◦ k , r , c : Loop/index variables used within isSafe .

• Time Complexity Analysis (Backtracking N-Queens):

◦ In the worst case, the algorithm might explore a significant

portion of the possible placements.

◦ The state space tree has a branching factor of up to N at each

level (trying each row) and a depth of N (number of columns).

◦ A loose upper bound is O(NN), as for each of the N columns,

we might try N rows.

◦ However, the isSafe check prunes many branches early. A

tighter (but still exponential) upper bound is closer to O(N!),
because once a queen is placed in a row, that row is excluded
for subsequent columns.

◦ The exact complexity is difficult to pin down but is

significantly better than O(NN) due to pruning, yet still
exponential.

• Space Complexity: Primarily determined by the recursion depth, which

is O(N). We also need O(N²) space to store the board.

• Example: Backtracking for TSP

◦ Similar to Branch-and-Bound, but typically without the strong

bounding function. It explores paths (partial tours) using DFS.

◦ Start at city 0. Recursively visit an unvisited neighbor city.

◦ Keep track of the current path cost.

◦ If a path is extended such that its cost already exceeds the best
tour found so far, backtrack (pruning). This simple pruning
makes it slightly better than pure brute-force.

◦ If all cities are visited, calculate the cost back to the start city
and update the minimum cost found if the current tour is
better.

◦ If all neighbors from a city have been explored without

leading to a better complete tour, backtrack.

◦ Complexity remains roughly O(N!) in the worst case without

strong bounds.
 • University Exam Style Question:

◦ Question: Describe the Backtracking algorithmic paradigm.

How does it systematically explore the solution space?
Explain how backtracking is used to solve the N-Queens
problem, including the role of the isSafe check and the
backtracking step.

◦ Answer: Backtracking is a technique that incrementally builds

solution candidates. It uses a depth-first search approach on
the state space tree. At each step, it tries to extend a partial
solution. If an extension violates constraints or cannot lead to
a full solution, it undoes the last step (backtracks) and tries an
alternative. For the N-Queens problem, it tries placing queens
column by column. In column col , it iterates through rows
i . The isSafe function checks if placing a queen at ( i ,
col ) conflicts with queens in previous columns (same row
or diagonals). If safe, it places the queen and recursively calls
for column col+1 . If the recursive call fails, it backtracks
by removing the queen from ( i , col ) and trying the next
row. This pruning based on the isSafe check avoids
exploring many invalid configurations.

Heuristics

Concept:
So far, we've looked at strategies like Brute-Force, DP, B&B, and Backtracking that
aim to find the exact, optimal solution to a problem. However, for many complex
problems (especially NP-hard ones like TSP), finding the guaranteed optimal
solution can take an impractically long time, even with clever algorithms like DP or
B&B, as the input size grows.

This is where heuristics come in. A heuristic is a technique designed to solve a

problem faster when classic methods are too slow, or to find an approximate solution
when classic methods fail to find any exact solution. It's essentially a practical
approach, a rule of thumb, or an educated guess that often leads to a good-enough
solution, but without guaranteeing optimality or even feasibility.
Think of finding your way in a new city without a map. A heuristic might be "always
walk towards the tallest building" if you're trying to reach the city center. This might
work well often, but it doesn't guarantee the shortest path or even that you'll reach
the center (maybe the tallest building is elsewhere).

Characteristics of Heuristics:
1. Approximation: They aim for a good solution, not necessarily the best one.
2. Speed: They are typically much faster than exact algorithms, often running in
polynomial time.
3. No Optimality Guarantee: The solution found might be suboptimal or, in some
cases, not even a valid solution (though usually they aim for validity).
4. Problem-Specific: Heuristics are often tailored to the specific structure or
characteristics of the problem being solved.
5. Trade-off: They trade optimality, completeness, accuracy, or precision for speed.

Why Use Heuristics?

* When exact algorithms are too slow for the required input size.
* When the problem complexity makes finding an exact solution impossible within
reasonable time/resource limits.
* When an approximate or "good enough" solution is acceptable.
* As part of larger algorithms (e.g., providing a good starting point or bound for
Branch-and-Bound).

Application Domains:
Heuristics are widely used in various fields:
* Artificial Intelligence: Search algorithms (like A search uses a heuristic function),
game playing (chess engines).
* Operations Research: Vehicle routing, scheduling, facility location.
* Computer Networking: Routing protocols.
* Software Engineering: Virus scanning (heuristic detection).
* Optimization:* Solving large-scale TSP, Bin Packing, etc.
Example: Heuristics for the Traveling Salesperson Problem (TSP)
Since finding the optimal TSP tour is NP-hard, heuristics are commonly used for
practical instances.

1. Nearest Neighbor Heuristic:

◦ Idea: A simple greedy-like heuristic. Start at an arbitrary city.

Repeatedly visit the nearest unvisited city until all cities have
been visited. Finally, return to the starting city.

◦ Steps:
1. Select an arbitrary starting city.

2. Find the unvisited city closest to the current city.

3. Move to this closest city and mark it as visited.

4. Repeat steps 2-3 until all cities are visited.

5. Return to the starting city from the last visited city.

◦ Analysis: Very fast (O(n²)). Easy to implement. However, it

often produces tours that are significantly longer than the
optimal one (no guarantee on solution quality). The choice of
starting city can also affect the result.

2. Insertion Heuristics (e.g., Nearest Insertion, Farthest Insertion):

◦ Idea: Start with a small sub-tour (e.g., two cities). Repeatedly

select an unvisited city and insert it into the position in the
current sub-tour that causes the minimum increase in the total
tour length.

◦ Steps (Nearest Insertion):

1. Start with a sub-tour containing an arbitrary city.

2. Find the unvisited city closest to any city already in

the sub-tour.

3. Find the edge (pair of adjacent cities) in the sub-

tour such that inserting the selected city between
them minimizes the insertion cost (cost(i, k) +
cost(k, j) - cost(i, j), where k is the city to insert
between i and j).

4. Insert the city into that position.

 5. Repeat steps 2-4 until all cities are included in the
tour.

◦ Analysis: Generally performs better than Nearest Neighbor.

Complexity is typically O(n²). Still no optimality guarantee.

3. Christofides Algorithm (More Advanced Heuristic/Approximation

Algorithm):

◦ Idea: A more sophisticated heuristic that does provide a

guarantee on solution quality (it's technically an
approximation algorithm, which we'll discuss later). It
guarantees that the tour found is no more than 1.5 times the
length of the optimal tour (for Euclidean TSP).

◦ Steps (High Level): Involves finding a Minimum Spanning

Tree (MST) of the cities, finding minimum-weight perfect
matching on the odd-degree vertices of the MST, combining
them to form an Eulerian circuit, and then converting this
circuit into a Hamiltonian cycle (TSP tour) using shortcuts.

◦ Analysis: Polynomial time complexity (dominated by MST

and matching). Provides a good quality guarantee.

Heuristics vs. Approximation Algorithms:

While related, there's a distinction often made: An approximation algorithm is a
heuristic that comes with a provable guarantee on how far the found solution can be
from the true optimum (e.g., the 1.5-guarantee of Christofides). A general heuristic
might work well in practice but lacks such a formal guarantee.

• University Exam Style Question:

◦ Question: What is a heuristic algorithm? Explain its main
characteristics and why heuristics are often used for
computationally hard problems like the Traveling Salesperson
Problem (TSP). Describe the Nearest Neighbor heuristic for
TSP.

◦ Answer: A heuristic is a technique designed to find a good-

enough solution to a problem quickly, especially when exact
methods are too slow. It trades optimality or completeness for
speed. Characteristics include: aiming for approximation,
 being fast (often polynomial time), lacking optimality
guarantees, and often being problem-specific. Heuristics are
used for hard problems like TSP because finding the exact
optimal solution takes exponential time, which is infeasible for
large inputs. A heuristic provides a practical way to get a
reasonable solution quickly. The Nearest Neighbor heuristic
for TSP starts at an arbitrary city and repeatedly moves to the
closest unvisited city until all cities are visited, finally
returning to the start. While simple and fast (O(n²)), it doesn't
guarantee an optimal tour.

This concludes our overview of the fundamental algorithmic strategies. Each strategy
offers a different way to think about and approach problem-solving. Often, complex
problems might require combining ideas from multiple strategies.

Practice Questions (Chapter 2)

1. Question: Compare and contrast the Greedy and Dynamic Programming

strategies. Give an example problem where Greedy works optimally and
one where it fails but DP succeeds.
Answer: Both Greedy and DP solve problems by making choices based
on subproblems. Greedy makes the locally optimal choice at each step,
hoping it leads to a global optimum, without revisiting choices. DP
typically explores multiple options by solving overlapping subproblems
once and storing results, ensuring global optimality if the problem has
optimal substructure and overlapping subproblems. Greedy is often
simpler and faster but doesn't always work. DP is generally more
powerful but can be more complex.

◦ Greedy works optimally for Fractional Knapsack (prioritize

value/weight ratio).

◦ Greedy fails for 0/1 Knapsack, but DP works optimally by

building a table dp[i][j] representing the max value
using first i items with capacity j .
 2. Question: Explain the difference between Backtracking and Branch-and-
Bound. Which strategy is typically more suited for optimization
problems, and why?
Answer: Both Backtracking and Branch-and-Bound explore a state space
tree. Backtracking uses a DFS approach, abandoning a path
(backtracking) as soon as a constraint is violated or it's clear a valid
solution cannot be formed down that path. It's often used for constraint
satisfaction or finding all solutions. Branch-and-Bound is specifically
designed for optimization problems. It also explores the tree but uses a
bounding function to estimate the best possible outcome achievable from
a node. If this bound is worse than the best solution found so far
(incumbent), it prunes the entire branch. B&B often uses Best-First
search guided by the bounds. Branch-and-Bound is more suited for
optimization because the bounding mechanism directly aims to find the
minimum/maximum value by pruning suboptimal branches, which is its
core purpose.

Further Learning: Video Resources

Here are some suggested videos related to the strategies discussed:

• Greedy Algorithms: https://www.youtube.com/watch?v=ARvQcqJ_-NY

(Abdul Bari - Introduction to Greedy)

• Dynamic Programming: https://www.youtube.com/watch?

v=nqowUJzG-
iM&list=PLU7rFVD05MosxuUf9LV4ML6uzsSRol8nE&index=2 (From
your playlist - DP Intro/Fibonacci)

• 0/1 Knapsack (DP): https://www.youtube.com/watch?

v=zRza99HPvkQ&list=PLU7rFVD05MosxuUf9LV4ML6uzsSRol8nE&index=7
(From your playlist - Knapsack)

• Backtracking (N-Queens): https://www.youtube.com/watch?

v=xFv_Hl4B83A (Abdul Bari - N-Queens)

• Branch and Bound: https://www.youtube.com/watch?v=1oXTy0CjE_8

(Knowledge Gate - Branch and Bound Intro)
(Note: Please verify these links. If the playlist links are incorrect or missing specific
topics, the alternative suggestions from Abdul Bari/Knowledge Gate provide good
explanations.)
Chapter 3: Graph and Tree Algorithms
Welcome to the world of graphs and trees! These structures are fundamental in
computer science, modeling relationships and networks in countless applications,
from social networks and road maps to computer networks and project dependencies.
Understanding how to represent and manipulate graphs and trees is crucial for
solving a wide range of problems efficiently.

In this chapter, we will explore essential algorithms for working with these
structures. We will start with basic concepts and representations, then dive into
methods for traversing graphs, finding shortest paths, identifying minimum spanning
trees, ordering tasks, and analyzing network flows.

Introduction to Graphs and Trees

What is a Graph?
Informally, a graph is a collection of points (called vertices or nodes) connected by
lines (called edges or arcs). Graphs are used to represent relationships between
objects.

• Vertices (V): Represent the objects or entities (e.g., cities, people, web
pages, tasks).

• Edges (E): Represent the connections or relationships between vertices

(e.g., roads between cities, friendships, hyperlinks, dependencies).

A graph G is formally defined as a pair G = (V, E), where V is the set of vertices and
E is the set of edges.

Types of Graphs:
* Undirected Graph: Edges have no direction. If there's an edge between vertex A
and vertex B, it represents a two-way relationship (e.g., a friendship on Facebook, a
road between two cities).
* Directed Graph (Digraph): Edges have a direction, indicated by arrows. An edge
from A to B represents a one-way relationship (e.g., following someone on Twitter, a
one-way street, a prerequisite task).
* Weighted Graph: Each edge has an associated numerical value called a weight or
cost (e.g., distance between cities, cost of a connection, time required for a task).
* Unweighted Graph: Edges do not have weights (or can be considered to have a
weight of 1).
* Cyclic Graph: Contains at least one path that starts and ends at the same vertex (a
cycle).
* Acyclic Graph: Contains no cycles. A Directed Acyclic Graph (DAG) is
particularly important for representing dependencies (like task prerequisites).

What is a Tree?
A tree is a special type of graph that is undirected, connected (there is a path
between any two vertices), and acyclic (contains no cycles). Trees have a
hierarchical structure.

• Root: In a rooted tree, one vertex is designated as the root.

• Parent/Child: In a rooted tree, for any edge connecting vertex u and

v , if u is closer to the root, u is the parent of v , and v is the child
of u .

• Leaves: Vertices with no children.

• Binary Tree: A rooted tree where each node has at most two children
(left and right).

Trees are used in data structures (like binary search trees, heaps), representing
hierarchies (file systems), and as components in graph algorithms (like spanning
trees).

Graph Representation

To work with graphs algorithmically, we need ways to represent them in a computer's

memory. The two most common methods are:

1. Adjacency Matrix:

◦ Representation: A 2D array (matrix) adj of size |V| x |V|,

where |V| is the number of vertices.
 ◦ adj[i][j] = 1 (or the weight w ) if there is an edge
from vertex i to vertex j .

◦ adj[i][j] = 0 (or infinity/special value) if there is no

edge from i to j .

◦ For undirected graphs, the matrix is symmetric ( adj[i][j]

= adj[j][i] ).

◦ Pros: Checking if an edge exists between two vertices i and

j is very fast (O(1) lookup).

◦ Cons: Requires O(|V|²) space, regardless of the number of

edges. This is inefficient for sparse graphs (graphs with
relatively few edges compared to the maximum possible |V|²).

2. Adjacency List:

◦ Representation: An array (or map) of |V| lists. The list

adj[i] contains all vertices j such that there is an edge
from vertex i to vertex j .

◦ For weighted graphs, the list can store pairs (j, weight) .

◦ Pros: Space-efficient for sparse graphs. Space complexity is

O(|V| + |E|), where |E| is the number of edges.

◦ Cons: Checking if an edge exists between i and j might

require searching through the list adj[i] , which can take
O(degree(i)) time in the worst case (where degree(i) is the
number of neighbors of i ).

Choosing a Representation:
* Use an adjacency matrix if the graph is dense (many edges, |E| close to |V|²) or if
you need frequent O(1) checks for the existence of specific edges.
* Use an adjacency list if the graph is sparse (few edges, |E| much smaller than |V|
²), which is common in many real-world applications. Most algorithms discussed
here perform well with adjacency lists.
Traversal Algorithms

Graph traversal means visiting all the vertices and edges of a graph in a systematic
way. Traversal is a fundamental building block for many other graph algorithms. The
two main traversal strategies are Depth First Search (DFS) and Breadth First Search
(BFS).

Depth First Search (DFS)

Concept:
DFS explores the graph by going as deep as possible along each branch before
backtracking. It starts at a selected vertex (the source) and explores along a path until
it reaches a dead end (a vertex with no unvisited neighbors) or a previously visited
vertex. It then backtracks to the most recent vertex from which it can explore an
alternative unvisited path.

Think of it like exploring a maze by always taking the first available path you see
and only turning back when you hit a dead end, then trying the next available path
from the last junction.

DFS typically uses a stack (either explicitly or implicitly via recursion) to keep track
of the vertices to visit.

Theoretical Steps (Recursive DFS):

1. Mark the starting vertex s as visited.
2. Process the starting vertex s (e.g., print it, add it to a list).
3. For each neighbor v of the current vertex s :
4. Check if neighbor v has been visited.
5. If v has not been visited:
6. Recursively call DFS starting from vertex v .
7. (After exploring all neighbors of s , the recursion naturally backtracks).
8. To ensure all vertices in potentially disconnected graphs are visited, iterate through
all vertices in the graph. If a vertex hasn't been visited yet, start a new DFS from that
vertex.

Pseudocode: Recursive DFS

 // Main DFS function
function DFS(Graph):
visited = create_set_or_boolean_array(size = |V|, initial_value = fa
for each vertex u in Graph.vertices:
if visited[u] == false:
DFS_Visit(Graph, u, visited)

// Recursive helper function

function DFS_Visit(Graph, u, visited):
// Mark vertex u as visited
visited[u] = true

// Process vertex u (e.g., print, add to list)

process(u)

// Explore neighbors
for each neighbor v of u in Graph.adjacency_list[u]:
if visited[v] == false:
DFS_Visit(Graph, v, visited)

• Explanation of Variables:
◦ Graph : The input graph (represented typically by adjacency
lists).

◦ visited : A set or boolean array to keep track of visited

vertices.

◦ u : The current vertex being visited in DFS_Visit .

◦ v : A neighbor of vertex u .

◦ process(u) : Placeholder for any operation to be

performed on vertex u upon visiting it.

Time Complexity Analysis (DFS):

* Each vertex is visited exactly once (due to the visited check).
* When visiting a vertex u , we examine all its outgoing edges (neighbors in the
adjacency list).
* The total number of edge examinations across all vertices is proportional to the
sum of degrees of all vertices.
* For an adjacency list representation:
* Sum of degrees is 2|E| for undirected graphs.
* Sum of out-degrees is |E| for directed graphs.
* Initialization of visited takes O(|V|) time.
* Total time complexity = O(|V| + |E|).
* If using an adjacency matrix, checking neighbors takes O(|V|) time for each vertex,
leading to O(|V|²) complexity.

Space Complexity (DFS):

* O(|V|) for storing the visited set/array.
* O(|V|) for the recursion call stack in the worst case (for a path graph).
* Total space complexity = O(|V|).

Applications of DFS:
* Finding connected components in an undirected graph.
* Finding strongly connected components in a directed graph.
* Topological sorting of a DAG.
* Detecting cycles in a graph.
* Pathfinding.

University Exam Style Question:

* Question: Describe the Depth First Search (DFS) algorithm for graph traversal.
Explain its time and space complexity when using an adjacency list representation.
Give one application of DFS.
* Answer: DFS explores a graph by going as deep as possible along a path before
backtracking. It starts at a source vertex, marks it visited, processes it, and then
recursively visits its first unvisited neighbor. It continues down this path until a dead
end, then backtracks to explore other unvisited neighbors. A visited array
prevents cycles and redundant work. Using an adjacency list, each vertex is visited
once (O(|V|)) and each edge is examined once (or twice in undirected graphs, O(|E|)),
giving a time complexity of O(|V| + |E|). The space complexity is O(|V|) for the
visited array and the recursion stack. An application of DFS is detecting cycles
in a graph.

Breadth First Search (BFS)

Concept:
BFS explores the graph layer by layer. It starts at a selected vertex (the source) and
visits all its immediate neighbors first. Then, it visits all the neighbors of those
neighbors that haven't been visited yet, and so on. It explores the graph uniformly
outwards from the source.
Think of ripples spreading out from a stone dropped in water. BFS visits vertices in
increasing order of their distance (number of edges) from the source vertex.

BFS typically uses a queue to keep track of the vertices to visit.

Theoretical Steps (BFS):

1. Create a queue and enqueue the starting vertex s .
2. Mark the starting vertex s as visited.
3. While the queue is not empty:
4. Dequeue a vertex u from the front of the queue.
5. Process vertex u (e.g., print it, check if it's the target).
6. For each neighbor v of the dequeued vertex u :
7. Check if neighbor v has been visited.
8. If v has not been visited:
a. Mark v as visited.
b. Enqueue v .
9. To ensure all vertices in potentially disconnected graphs are visited, iterate through
all vertices. If a vertex hasn't been visited, start a new BFS from that vertex.

Pseudocode: BFS
function BFS(Graph, start_node):
visited = create_set_or_boolean_array(size = |V|, initial_value = fa
queue = new Queue()

// Start BFS from start_node

visited[start_node] = true
queue.enqueue(start_node)

while queue is not empty:

u = queue.dequeue()

// Process vertex u
process(u)

// Explore neighbors
for each neighbor v of u in Graph.adjacency_list[u]:
if visited[v] == false:
visited[v] = true
queue.enqueue(v)

// To handle disconnected graphs:

function BFS_Full(Graph):
visited = create_set_or_boolean_array(size = |V|, initial_value = fa
for each vertex s in Graph.vertices:
if visited[s] == false:
// Run BFS starting from s
queue = new Queue()
visited[s] = true
queue.enqueue(s)
while queue is not empty:
u = queue.dequeue()
process(u)
for each neighbor v of u in Graph.adjacency_list[u]:
if visited[v] == false:
visited[v] = true
queue.enqueue(v)

• Explanation of Variables:
◦ Graph : The input graph (adjacency lists).

◦ start_node : The vertex from which to start the traversal

(for a single connected component).

◦ visited : Set or boolean array to track visited vertices.

 ◦ queue : A queue data structure (FIFO).

◦ s : Starting node for a BFS run (used in BFS_Full ).

◦ u : Vertex dequeued from the queue.

◦ v : A neighbor of vertex u .

◦ process(u) : Placeholder for processing the visited vertex.

Time Complexity Analysis (BFS):

* Each vertex is enqueued and dequeued exactly once (O(|V|)).
* Each edge is examined exactly once (or twice for undirected graphs) when
exploring neighbors (O(|E|)).
* Initialization takes O(|V|).
* Total time complexity = O(|V| + |E|) (using adjacency lists).
* Using an adjacency matrix leads to O(|V|²) complexity.

Space Complexity (BFS):

* O(|V|) for storing the visited set/array.
* O(|V|) for the queue in the worst case (e.g., a star graph where the center node
connects to all others).
* Total space complexity = O(|V|).

Applications of BFS:
* Finding the shortest path between two nodes in an unweighted graph (in terms of
number of edges).
* Finding connected components.
* Testing if a graph is bipartite.
* Used as a subroutine in other algorithms (like Ford-Fulkerson, Prim's MST with
adjacency matrix).

University Exam Style Question:

* Question: Describe the Breadth First Search (BFS) algorithm. Explain how it
differs from DFS in terms of exploration order and the data structure used. What is a
key application where BFS is preferred over DFS?
* Answer: BFS explores a graph layer by layer, starting from a source node. It visits
all immediate neighbors, then their unvisited neighbors, and so on, exploring
outwards. It uses a queue (FIFO) to manage the order of vertices to visit. This
contrasts with DFS, which explores as deeply as possible along one path using a
stack (LIFO or recursion) before backtracking. BFS is preferred for finding the
shortest path in terms of the number of edges in an unweighted graph because it
explores vertices in increasing order of distance from the source.

Shortest Path Algorithms

Finding the shortest path between vertices in a weighted graph is a classic and
essential problem with numerous applications (e.g., route planning, network routing,
analyzing dependencies).

Types of Shortest Path Problems:

* Single-Source Shortest Path (SSSP): Find the shortest paths from a single
starting vertex s to all other vertices in the graph.
* All-Pairs Shortest Path (APSP): Find the shortest paths between every pair of
vertices in the graph.

The choice of algorithm depends on the problem type and graph properties
(especially the presence of negative edge weights).

Dijkstra's Algorithm (SSSP)

Concept:
Dijkstra's algorithm finds the shortest paths from a single source vertex s to all
other vertices in a weighted graph with non-negative edge weights. It works by
maintaining a set of vertices for which the shortest path from the source is already
known. It iteratively selects the vertex u not yet in this set that has the smallest
tentative distance from s , adds u to the set, and updates the distances of u 's
neighbors.

It's a greedy algorithm because at each step, it greedily selects the vertex with the
currently known shortest distance.

Theoretical Steps (Dijkstra's Algorithm):

1. Initialize distances: Set the distance to the source vertex s as 0
( dist[s] = 0 ) and the distance to all other vertices as infinity ( dist[v] =
∞ ).
2. Create a set S of visited vertices, initially empty. (Alternatively, use a
visited array).
3. Create a priority queue Q containing all vertices, prioritized by their current
dist values (minimum distance first).
4. While the priority queue Q is not empty:
5. Extract the vertex u from Q with the smallest dist value. This dist[u] is
now the final shortest distance from s to u .
6. Add u to the visited set S (or mark u as visited).
7. For each neighbor v of vertex u :
8. Calculate the alternative distance to v through u : alt = dist[u] +
weight(u, v) .
9. If alt < dist[v] (a shorter path to v has been found):
a. Update dist[v] = alt .
b. Update the priority of v in the priority queue Q with the new distance alt .
(This step is crucial for efficiency).
10. Once Q is empty, the dist array contains the shortest path distances from s
to all reachable vertices.

Pseudocode: Dijkstra's Algorithm (using Priority Queue)

function Dijkstra(Graph, source):
// Initialize distances and predecessors
dist = create_map_or_array(size=|V|, initial_value=infinity)
prev = create_map_or_array(size=|V|, initial_value=null)
dist[source] = 0

// Priority Queue stores (distance, vertex)

// Min-heap based on distance
pq = new PriorityQueue()
for each vertex v in Graph.vertices:
pq.add((dist[v], v))

while pq is not empty:

// Extract vertex u with the smallest distance
(d, u) = pq.extract_min()

// Optimization: If we extracted a vertex with outdated distance

if d > dist[u]:
continue

// Explore neighbors of u
for each neighbor v of u with edge weight w in Graph.adjacency_l
// Calculate distance through u
alt = dist[u] + w

// Relaxation step
if alt < dist[v]:
dist[v] = alt
prev[v] = u
// Update priority in PQ (decrease-key operation)
pq.decrease_key(v, alt) // Or add (alt, v) if PQ allows

// Return distances and predecessors (for path reconstruction)

return dist, prev

• Explanation of Variables:
◦ Graph : Input graph with non-negative weights (adjacency
lists often store pairs (neighbor, weight) ).

◦ source : The starting vertex.

◦ dist : Array/Map storing the current shortest distance found

from source to each vertex.
 ◦ prev : Array/Map storing the predecessor of each vertex in
the shortest path from source (used to reconstruct paths).

◦ pq : Priority queue (min-heap) storing vertices prioritized by

their dist values.

◦ u : Vertex extracted from pq with the smallest known

distance.

◦ v : A neighbor of u .

◦ w : Weight of the edge (u, v) .

◦ alt : The calculated distance from source to v passing

through u .

Time Complexity Analysis (Dijkstra's):

* The complexity depends heavily on the priority queue implementation.
* Initialization: O(|V|).
* Building initial priority queue: O(|V|) or O(|V| log |V|) depending on
implementation.
* Main loop runs |V| times (extracting each vertex once).
* Inside the loop:
* extract_min : O(log |V|) for binary heap, O(1) amortized for Fibonacci heap.
* Neighbor loop: Each edge (u, v) is processed once across the entire algorithm.
* decrease_key : O(log |V|) for binary heap, O(1) amortized for Fibonacci heap.
* Total operations: |V| extract_min operations and |E| decrease_key
operations.
* Using Binary Heap: O(|V| log |V| + |E| log |V|) = O(|E| log |V|) (since |E| is often
>= |V|-1 in connected graphs).
* Using Fibonacci Heap: O(|E| + |V| log |V|) (Theoretically better for dense graphs,
but often slower in practice due to higher constant factors).
* If using a simple array for the priority queue (finding min takes O(|V|)): O(|V|² + |
E|) = O(|V|²).

Space Complexity: O(|V|) for dist , prev , and the priority queue.

Limitation: Dijkstra's algorithm does not work correctly if the graph contains
negative edge weights. The greedy choice of selecting the vertex with the smallest
current distance might be wrong if a later path involving a negative edge could lead
to an even shorter overall path.
University Exam Style Question:
* Question: Explain Dijkstra's algorithm for finding single-source shortest paths.
What is the core idea (greedy choice)? What is its time complexity using a binary
heap priority queue? What is a major limitation of Dijkstra's algorithm?
* Answer: Dijkstra's algorithm finds shortest paths from a source s in a weighted
graph with non-negative edges. It maintains a set of visited nodes with known
shortest paths and uses a priority queue to store unvisited nodes prioritized by their
tentative shortest distance from s . The greedy choice is to repeatedly extract the
unvisited node u with the minimum tentative distance, finalize its distance, and
update (relax) the distances of its neighbors v if a shorter path via u is found
( dist[v] = min(dist[v], dist[u] + weight(u,v)) ). Using a
binary heap, the time complexity is O(|E| log |V|). A major limitation is that it fails if
the graph contains negative edge weights.

(Chapter 3 continues with Bellman-Ford, Floyd-Warshall, Transitive Closure, MST,

Topological Sort, Network Flow...)

Bellman-Ford Algorithm (SSSP)

Concept:
The Bellman-Ford algorithm also solves the Single-Source Shortest Path problem,
finding the shortest paths from a single source vertex s to all other vertices. Unlike
Dijkstra's, Bellman-Ford can handle graphs with negative edge weights. It can also
detect negative cycles reachable from the source (a cycle whose edges sum to a
negative value), which would imply that shortest paths are undefined (can be made
arbitrarily small by traversing the cycle).

The algorithm works by iteratively relaxing edges. It performs |V|-1 passes

(iterations) over all edges in the graph. In each pass, it updates the distance to
vertices if a shorter path is found by considering one more edge. After |V|-1 passes, if
the graph has no negative cycles reachable from the source, the distances converge to
the shortest path values. A final, |V|-th pass is used to detect negative cycles.

Theoretical Steps (Bellman-Ford Algorithm):

1. Initialize distances: Set dist[s] = 0 and dist[v] = ∞ for all other
vertices v .
2. Initialize predecessors: Set prev[v] = null for all vertices v .
3. Repeat |V|-1 times:
4. For each edge (u, v) with weight w in the graph's edge list E:
5. Relaxation: If dist[u] + w < dist[v] :
a. Update dist[v] = dist[u] + w .
b. Update prev[v] = u .
6. After |V|-1 iterations, check for negative cycles:
7. For each edge (u, v) with weight w in E:
8. If dist[u] + w < dist[v] :
a. A negative cycle reachable from s exists. Report the error (shortest paths are ill-
defined or indicate the cycle).
9. If no such edge is found in step 8, the dist array contains the correct shortest
path distances.

Pseudocode: Bellman-Ford Algorithm

function BellmanFord(Graph, source):
// Initialize distances and predecessors
dist = create_map_or_array(size=|V|, initial_value=infinity)
prev = create_map_or_array(size=|V|, initial_value=null)
dist[source] = 0

// Get list of all edges: edges = [(u1, v1, w1), (u2, v2, w2), ...]
edges = Graph.get_all_edges()
num_vertices = Graph.number_of_vertices()

// Relax edges |V|-1 times

for i from 1 to num_vertices - 1:
for each edge (u, v, w) in edges:
if dist[u] != infinity and dist[u] + w < dist[v]:
dist[v] = dist[u] + w
prev[v] = u

// Check for negative cycles

for each edge (u, v, w) in edges:
if dist[u] != infinity and dist[u] + w < dist[v]:
// Negative cycle detected
return "Error: Negative cycle detected", null

// Return distances and predecessors

return dist, prev

• Explanation of Variables:
◦ Graph : Input graph (can have negative weights).

◦ source : The starting vertex.

◦ dist : Array/Map storing the current shortest distance

estimate from source .

◦ prev : Array/Map storing predecessors for path

reconstruction.

◦ edges : A list containing all edges of the graph, typically as

tuples (u, v, w) where u is the source, v the
destination, and w the weight.

◦ num_vertices : The total number of vertices, |V|.

◦ i : Loop counter for the main relaxation phase.

 ◦ (u, v, w) : Represents an edge being processed.

Time Complexity Analysis (Bellman-Ford):

* Initialization: O(|V|).
* Main loop runs |V|-1 times.
* Inside the main loop, we iterate through all |E| edges.
* Relaxation step takes O(1) time.
* Total time for main loop = O(|V| * |E|).
* Negative cycle check loop iterates through all |E| edges once = O(|E|).
* Total time complexity = O(|V| * |E|).

Space Complexity: O(|V|) for dist and prev arrays.

Comparison with Dijkstra:

* Bellman-Ford is slower than Dijkstra (O(|V||E|) vs O(|E| log |V|)).
* Bellman-Ford works with negative edge weights, while Dijkstra does not.
* Bellman-Ford can detect negative cycles reachable from the source.

University Exam Style Question:

* Question: When is the Bellman-Ford algorithm preferred over Dijkstra's algorithm
for the single-source shortest path problem? Explain the main steps of Bellman-Ford
and its time complexity. How does it detect negative cycles?
* Answer: Bellman-Ford is preferred over Dijkstra's when the graph may contain
negative edge weights. Dijkstra's greedy approach fails with negative edges.
Bellman-Ford works by initializing distances (source=0, others=infinity) and then
iteratively relaxing all |E| edges for |V|-1 passes. Relaxation updates dist[v] if a
shorter path via edge (u, v) is found ( dist[v] = min(dist[v],
dist[u] + weight(u,v)) ). After |V|-1 passes, it performs one final pass over
all edges. If any distance can still be improved during this final pass, it indicates a
negative cycle reachable from the source. The time complexity is O(|V| * |E|).

Floyd-Warshall Algorithm (APSP)

Concept:
The Floyd-Warshall algorithm solves the All-Pairs Shortest Path problem. It finds
the shortest paths between every pair of vertices (i, j) in a weighted graph. Like
Bellman-Ford, it can handle negative edge weights, but it assumes there are no
negative cycles. (If negative cycles exist, some shortest paths might be undefined,
often represented as -infinity).

The algorithm uses Dynamic Programming. It considers intermediate vertices that

can be used in a path from i to j . It iteratively allows more vertices to be used as
intermediates. Let dist[i][j][k] be the shortest path from i to j using
only intermediate vertices from the set {1, 2, ..., k} . The core idea is that
the shortest path from i to j using intermediates up to k either doesn't use k as
an intermediate (in which case the path is the same as using intermediates up to
k-1 ), or it does use k (in which case the path goes from i to k using
intermediates up to k-1 , and then from k to j using intermediates up to k-1 ).

Theoretical Steps (Floyd-Warshall Algorithm):

1. Initialize a distance matrix dist[|V|][|V|] . For each pair (i, j) :
* dist[i][i] = 0 .
* dist[i][j] = weight(i, j) if there is a direct edge from i to j .
* dist[i][j] = ∞ if there is no direct edge from i to j .
2. (Optional) Initialize a predecessor matrix next[|V|][|V|] to reconstruct
paths. next[i][j] = j if there's an edge (i, j) or i=j , else null .
3. Iterate through all possible intermediate vertices k from 0 to |V|-1 (or 1 to |V|
depending on indexing).
4. Inside the k loop, iterate through all possible source vertices i from 0 to |V|-1.
5. Inside the i loop, iterate through all possible destination vertices j from 0 to |
V|-1.
6. Check if the path from i to j via intermediate k is shorter than the current
known shortest path from i to j :
* If dist[i][k] + dist[k][j] < dist[i][j] :
a. Update dist[i][j] = dist[i][k] + dist[k][j] .
b. (Optional) Update next[i][j] = next[i][k] .
7. After the loops complete, the dist[i][j] matrix contains the shortest path
distances between all pairs (i, j) .
8. (Optional) Check for negative cycles: If any dist[i][i] becomes negative
during the process, the graph contains a negative cycle.

Pseudocode: Floyd-Warshall Algorithm

function FloydWarshall(Graph):
num_vertices = Graph.number_of_vertices()

// Initialize distance matrix

dist = create_2d_array(num_vertices, num_vertices, initial_value=inf
// Optional: Initialize predecessor matrix for path reconstruction
// next_hop = create_2d_array(num_vertices, num_vertices, initial_va

// Initialize distances based on direct edges

for i from 0 to num_vertices - 1:
dist[i][i] = 0
// next_hop[i][i] = i
for each neighbor j of i with weight w in Graph.adjacency_list[i
dist[i][j] = w
// next_hop[i][j] = j

// Main loops: iterate through intermediate vertices k

for k from 0 to num_vertices - 1:
// Iterate through source vertices i
for i from 0 to num_vertices - 1:
// Iterate through destination vertices j
for j from 0 to num_vertices - 1:
// Check if path through k is shorter
if dist[i][k] != infinity and dist[k][j] != infinity and
dist[i][j] = dist[i][k] + dist[k][j]
// Optional: Update predecessor for path reconstruct
// next_hop[i][j] = next_hop[i][k]

// Optional: Check for negative cycles

for i from 0 to num_vertices - 1:
if dist[i][i] < 0:
return "Error: Negative cycle detected", null

// Return the matrix of shortest path distances

return dist //, next_hop

• Explanation of Variables:
◦ Graph : Input graph (can have negative weights, no negative
cycles assumed).

◦ num_vertices : |V|.

◦ dist : The |V|x|V| matrix storing shortest path distances.

dist[i][j] is the shortest distance from i to j .
 ◦ next_hop (Optional): |V|x|V| matrix storing the next vertex
in the shortest path from i to j . Used for path
reconstruction.

◦ k : Index of the intermediate vertex being considered.

◦ i : Index of the source vertex.

◦ j : Index of the destination vertex.

◦ w : Weight of a direct edge.

Time Complexity Analysis (Floyd-Warshall):

* Three nested loops, each running |V| times.
* The inner operation (comparison and update) takes O(1) time.
* Initialization takes O(|V|²) time.
* Total time complexity = O(|V|³).

Space Complexity: O(|V|²) for storing the distance matrix (and optionally the
predecessor matrix).

When to Use:
* For solving the All-Pairs Shortest Path problem.
* Suitable for dense graphs where |E| is close to |V|², as its O(|V|³) complexity is
independent of |E|.
* When the graph might contain negative edge weights (but no negative cycles).
* Simpler to implement than running Dijkstra or Bellman-Ford |V| times.

University Exam Style Question:

* Question: Explain the Floyd-Warshall algorithm for the All-Pairs Shortest Path
problem. What is the underlying principle (dynamic programming recurrence)? State
its time and space complexity.
* Answer: Floyd-Warshall finds shortest paths between all pairs of vertices (i,
j) in a weighted graph, handling negative edges (but not negative cycles). It uses
dynamic programming. The core idea is to iteratively consider each vertex k as a
potential intermediate vertex. For every pair (i, j) , it checks if the path from i
to k and then from k to j is shorter than the currently known shortest path from
i to j . The recurrence is effectively dist[i][j] = min(dist[i][j],
dist[i][k] + dist[k][j]) , where the update happens within the loop for
intermediate vertex k . It uses three nested loops, resulting in a time complexity of
O(|V|³). The space complexity is O(|V|²) to store the distance matrix.
Transitive Closure
Concept:
The transitive closure of a directed graph G = (V, E) is another graph G = (V, E),
where E* contains an edge (i, j) if and only if there is a path (of length one or
more) from vertex i to vertex j in the original graph G. Essentially, it tells us
about reachability: can we get from i to j ?

Applications:
* Analyzing dependencies (e.g., if task A depends on B, and B depends on C, then A
transitively depends on C).
* Reachability analysis in networks.

Methods:
1. Repeated DFS/BFS: Run DFS or BFS starting from each vertex v in the graph.
For each run starting at v , all vertices reached constitute the vertices j such that
there is an edge (v, j) in the transitive closure G. Time complexity: |V| * O(|V| +
|E|) = O(|V|² + |V||E|).
2. Floyd-Warshall Adaptation (Warshall's Algorithm):* We can adapt the Floyd-
Warshall algorithm to compute transitive closure efficiently. Instead of calculating
shortest path distances, we calculate boolean reachability.

Warshall's Algorithm (based on Floyd-Warshall):

Theoretical Steps:
1. Initialize a boolean matrix T[|V|][|V|] . T[i][j] = true if i == j
or if there is a direct edge (i, j) in G. Otherwise, T[i][j] = false .
2. Iterate through all possible intermediate vertices k from 0 to |V|-1.
3. Inside the k loop, iterate through all possible source vertices i from 0 to |V|-1.
4. Inside the i loop, iterate through all possible destination vertices j from 0 to |
V|-1.
5. Check if there is a path from i to j using k as an intermediate: If there is a
path from i to k ( T[i][k] is true) AND there is a path from k to j ( T[k]
[j] is true), then there must be a path from i to j .
6. Update T[i][j] = T[i][j] OR (T[i][k] AND T[k][j]) .
7. After the loops complete, T[i][j] is true if and only if there is a path from i
to j in the original graph G.
Pseudocode: Warshall's Algorithm

function WarshallTransitiveClosure(Graph):
num_vertices = Graph.number_of_vertices()

// Initialize boolean reachability matrix T

T = create_2d_array(num_vertices, num_vertices, initial_value=false)

// Initialize based on self-loops and direct edges

for i from 0 to num_vertices - 1:
T[i][i] = true
for each neighbor j of i in Graph.adjacency_list[i]:
T[i][j] = true

// Main loops: iterate through intermediate vertices k

for k from 0 to num_vertices - 1:
// Iterate through source vertices i
for i from 0 to num_vertices - 1:
// Iterate through destination vertices j
for j from 0 to num_vertices - 1:
// If path exists from i to k AND k to j, then path exis
T[i][j] = T[i][j] or (T[i][k] and T[k][j])

// Return the transitive closure matrix

return T

• Explanation of Variables:
◦ Graph : Input directed graph.

◦ num_vertices : |V|.

◦ T : The |V|x|V| boolean matrix representing transitive closure.

T[i][j] is true if j is reachable from i .

◦ k , i , j : Loop indices for intermediate, source, and

destination vertices.

Time Complexity Analysis (Warshall's): O(|V|³) - same structure as Floyd-

Warshall.
Space Complexity: O(|V|²) for the boolean matrix T .

University Exam Style Question:

* Question: What is the transitive closure of a directed graph? Describe Warshall's
algorithm for computing it and state its time complexity.
* Answer: The transitive closure G = (V, E) of a directed graph G = (V, E) contains
an edge (i, j) in E* if and only if there is a path from i to j in G. It
represents all reachability relationships. Warshall's algorithm computes this using a
dynamic programming approach similar to Floyd-Warshall. It initializes a boolean
matrix T based on direct edges ( T[i][j] = true if edge (i, j) exists or
i=j ). Then, for each intermediate vertex k , it updates T[i][j] to true if
T[i][j] was already true OR if both T[i][k] and T[k][j] are true. This
process iterates through all k , i , and j . The time complexity is O(|V|³).

Minimum Spanning Tree (MST)

Concept:
Given a connected, undirected, weighted graph, a Minimum Spanning Tree (MST)
is a subgraph that is a tree (contains no cycles), spans all the vertices (includes all
vertices from the original graph), and whose sum of edge weights is the minimum
possible among all such spanning trees.

Applications:
* Network design (e.g., connecting computers, houses, or cities with the minimum
amount of cable or road).
* Cluster analysis.
* Approximation algorithms for other problems (like TSP).

Two main greedy algorithms are used to find MSTs: Prim's and Kruskal's.

Prim's Algorithm
Concept:
Prim's algorithm builds the MST incrementally, starting from an arbitrary vertex. It
maintains a set of vertices already included in the MST. At each step, it finds the
minimum-weight edge that connects a vertex inside the MST set to a vertex outside
the MST set, and adds this edge and the corresponding outside vertex to the MST.

It's similar in structure to Dijkstra's algorithm, using a priority queue to efficiently

find the minimum-weight edge connecting to a vertex outside the current tree.
Theoretical Steps (Prim's Algorithm):
1. Initialize: Choose an arbitrary starting vertex s . Create a priority queue Q to
store vertices not yet in the MST, prioritized by the minimum weight of an edge
connecting them to a vertex already in the MST. Set the priority (key) of s to 0 and
all others to infinity. Maintain an array parent[v] to store the edge
(parent[v], v) that connects v to the MST.
2. While the priority queue Q is not empty:
3. Extract the vertex u from Q with the minimum key (smallest connection
weight). This vertex u and the edge (parent[u], u) (if u is not the start
node) are now part of the MST.
4. Mark u as visited or removed from consideration for Q .
5. For each neighbor v of u :
6. Check if v is still in the priority queue Q (i.e., not yet in the MST).
7. Check if the weight of the edge (u, v) is less than the current key (minimum
connection weight) associated with v in Q .
8. If both conditions are true:
a. Update parent[v] = u (record that (u, v) is now the best edge
connecting v to the MST).
b. Decrease the key of v in the priority queue Q to weight(u, v) .
9. Once Q is empty, the set of edges (parent[v], v) for all v (except the
start node) forms the MST.

Pseudocode: Prim's Algorithm (using Priority Queue)

function PrimsMST(Graph, start_node):
num_vertices = Graph.number_of_vertices()

// key[v] stores min weight edge connecting v to the MST vertices

key = create_map_or_array(size=num_vertices, initial_value=infinity)
// parent[v] stores the vertex in MST that connects to v via the min
parent = create_map_or_array(size=num_vertices, initial_value=null)
// mstSet[v] is true if v is included in MST
mstSet = create_boolean_array(size=num_vertices, initial_value=false

// Initialize start node

key[start_node] = 0
parent[start_node] = null // Start node is the root

// Priority Queue stores (key_value, vertex)

pq = new PriorityQueue()
for v from 0 to num_vertices - 1:
pq.add((key[v], v))

while pq is not empty:

// Extract vertex u not yet in MST with the smallest key
(k, u) = pq.extract_min()

// Add u to MST set

mstSet[u] = true

// Update keys of adjacent vertices v

for each neighbor v of u with edge weight w in Graph.adjacency_l
// If v is not in MST and edge (u,v) weight is smaller than
if mstSet[v] == false and w < key[v]:
key[v] = w
parent[v] = u
// Update priority in PQ
pq.decrease_key(v, w) // Or add (w, v) if PQ allows dupl

// Return the parent array representing the MST edges

// (Edges are (parent[v], v) for v != start_node)
return parent

• Explanation of Variables:
◦ Graph : Input connected, undirected, weighted graph.

◦ start_node : Arbitrary vertex to start building the MST

from.
 ◦ num_vertices : |V|.

◦ key : Array/Map storing the minimum weight of an edge

connecting vertex v to the currently formed tree.

◦ parent : Array/Map storing the parent of v in the MST

(defines the MST edges).

◦ mstSet : Boolean array indicating if a vertex is already

included in the MST.

◦ pq : Priority queue storing vertices not yet in the MST,

prioritized by their key value.

◦ u : Vertex extracted from pq (added to MST).

◦ v : Neighbor of u .

◦ w : Weight of edge (u, v) .

Time Complexity Analysis (Prim's):

* Identical structure to Dijkstra's algorithm.
* Depends on the priority queue implementation.
* Using Binary Heap: O(|E| log |V|).
* Using Fibonacci Heap: O(|E| + |V| log |V|).
* If using an adjacency matrix and searching for min edge linearly: O(|V|²).

Space Complexity: O(|V|) for key , parent , mstSet , and the priority queue.

Kruskal's Algorithm
Concept:
Kruskal's algorithm also finds an MST but uses a different greedy approach. It
considers all edges in the graph in increasing order of their weights. For each edge, it
checks if adding that edge to the currently formed forest (a collection of trees,
initially just individual vertices) would create a cycle. If it does not create a cycle,
the edge is added to the forest. This process continues until |V|-1 edges have been
added, at which point the forest becomes a single MST.

To efficiently check for cycles, Kruskal's algorithm typically uses a Disjoint Set
Union (DSU) data structure (also known as Union-Find).
Theoretical Steps (Kruskal's Algorithm):
1. Create a forest F where each vertex in the graph is initially a separate tree (set).
2. Create a list E_sorted containing all edges (u, v) from the graph, sorted
by weight w in non-decreasing order.
3. Initialize an empty set MST_edges to store the edges of the final MST.
4. Initialize edge count num_edges = 0 .
5. For each edge (u, v) with weight w in E_sorted (from smallest weight
to largest):
6. Check if vertices u and v belong to different trees (sets) in the forest F using
the DSU find operation.
7. If find(u) is not equal to find(v) (they are in different sets, adding the
edge won't create a cycle):
a. Add the edge (u, v) to MST_edges .
b. Increment num_edges .
c. Merge the trees (sets) containing u and v using the DSU union operation.
8. Stop when num_edges equals |V|-1 (or when all edges have been processed).
9. The set MST_edges now contains the edges of the MST.

Pseudocode: Kruskal's Algorithm (using DSU)

// Assumes existence of a Disjoint Set Union (DSU) data structure with:
// DSU.make_set(v): Creates a new set containing only v.
// DSU.find(v): Returns the representative (root) of the set containing
// DSU.union(u, v): Merges the sets containing u and v.

function KruskalsMST(Graph):
num_vertices = Graph.number_of_vertices()
MST_edges = new empty_list()

// Initialize DSU structure

dsu = new DSU()
for each vertex v in Graph.vertices:
dsu.make_set(v)

// Get all edges and sort them by weight

edges = Graph.get_all_edges() // List of (u, v, w)
sort edges by weight w (non-decreasing)

num_edges_added = 0

// Iterate through sorted edges

for each edge (u, v, w) in sorted_edges:
// Check if adding the edge creates a cycle using DSU
if dsu.find(u) != dsu.find(v):
// No cycle: Add edge to MST and merge sets
MST_edges.add((u, v, w))
dsu.union(u, v)
num_edges_added += 1

// Optimization: Stop if we have |V|-1 edges

if num_edges_added == num_vertices - 1:
break

// Check if a spanning tree was formed

if num_edges_added != num_vertices - 1:
return "Error: Graph is not connected", null // Or handle as nee
else:
return MST_edges

• Explanation of Variables:
◦ Graph : Input connected, undirected, weighted graph.

◦ num_vertices : |V|.
 ◦ MST_edges : List to store the edges selected for the MST.

◦ dsu : Disjoint Set Union data structure instance.

◦ edges : List of all edges (u, v, w) .

◦ sorted_edges : The list of edges sorted by weight w .

◦ num_edges_added : Counter for the number of edges

added to the MST.

◦ (u, v, w) : Represents an edge being considered.

Time Complexity Analysis (Kruskal's):

* Sorting the edges: O(|E| log |E|).
* Initializing DSU: O(|V|).
* Iterating through |E| edges:
* Each find operation: Nearly constant time on average O(α(|V|)) with path
compression and union by rank/size (α is the inverse Ackermann function, which
grows extremely slowly).
* Each union operation: Also O(α(|V|)).
* Total time for loop = O(|E| * α(|V|)).
* Overall complexity is dominated by the sorting step.
* Total time complexity = O(|E| log |E|). (Note: Since |E| can be up to O(|V|²), log |E|
is often O(log |V|), making it comparable to Prim's O(|E| log |V|)).

Space Complexity: O(|V| + |E|) to store edges, the DSU structure, and the resulting
MST.

Prim's vs. Kruskal's:

* Both are greedy and find the optimal MST.
* Prim's grows one tree from a starting node. Kruskal's grows a forest of trees that
eventually merge.
* Prim's is generally faster for dense graphs (using Fibonacci heap: O(|E| + |V| log |
V|)).
* Kruskal's is often easier to implement and can be faster for sparse graphs if edge
sorting is efficient (O(|E| log |E|) or O(|E| log |V|)).

University Exam Style Question:

* Question: Describe Kruskal's algorithm for finding a Minimum Spanning Tree
(MST). What data structure is crucial for its efficient implementation, and why?
State its time complexity.
* Answer: Kruskal's algorithm finds an MST by iteratively adding the lowest-weight
edge that does not form a cycle with already selected edges. It starts by sorting all
graph edges by weight. It then processes edges in increasing weight order. For each
edge (u, v) , it checks if u and v are already connected in the currently
formed forest. If not, the edge is added to the MST, and u and v 's components are
merged. This cycle check and merging is efficiently done using a Disjoint Set Union
(DSU) data structure. The find operation checks if u and v are in the same set
(connected), and the union operation merges their sets if the edge is added. The
time complexity is dominated by sorting the edges, resulting in O(|E| log |E|).

Topological Sorting

Concept:
A topological sort or topological ordering of a Directed Acyclic Graph (DAG) G =
(V, E) is a linear ordering of all its vertices such that for every directed edge (u,
v) from vertex u to vertex v , vertex u comes before vertex v in the ordering.

If the graph contains a cycle, a topological sort is not possible.

Applications:
* Task scheduling (e.g., course prerequisites, project task dependencies, compilation
dependencies). If task A must be done before task B, draw an edge A -> B. A
topological sort gives a valid sequence for executing the tasks.
* Detecting cycles in directed graphs.

Algorithms:
There are two common algorithms for topological sorting:

1. Kahn's Algorithm (Using In-degrees):

◦ Idea: Based on the fact that a DAG must have at least one
vertex with an in-degree (number of incoming edges) of 0.
The algorithm repeatedly finds a vertex with an in-degree of 0,
adds it to the sorted list, and removes it and its outgoing edges
from the graph. This removal might decrease the in-degree of
its neighbors, potentially creating new vertices with an in-
degree of 0.
 ◦ Theoretical Steps:
1. Compute the in-degree for every vertex in the
graph.

2. Initialize a queue Q and enqueue all vertices with

an initial in-degree of 0.

3. Initialize an empty list L to store the sorted

elements.

4. Initialize a counter count for visited vertices to

0.

5. While the queue Q is not empty:

6. Dequeue a vertex u from Q .

7. Add u to the end of the list L .

8. Increment count .

9. For each neighbor v of u :

a. Decrement the in-degree of v .
b. If the in-degree of v becomes 0, enqueue v
into Q .

10. After the loop, if count equals the total number

of vertices |V|, then the list L contains a valid
topological ordering.

11. If count is less than |V|, the graph contains a

cycle, and topological sorting is impossible.

2. DFS-Based Algorithm:

◦ Idea: Perform a Depth First Search (DFS) on the graph. The

key insight is that a vertex finishes (all its descendants are
visited, and the recursion returns) after all its descendants
have finished. Therefore, if we add a vertex to the front of a
list as it finishes, we get a reverse topological order. Reversing
this list gives a valid topological order.

◦ Theoretical Steps:
1. Initialize a list L to store the sorted vertices.
 2. Initialize a set visited to keep track of visited
vertices during the current DFS path (for cycle
detection) and a set visiting for nodes
currently in recursion stack.

3. Initialize a set finished to keep track of

vertices whose DFS exploration is fully complete.

4. For each vertex u in the graph:

5. If u has not been finished :

Call a recursive helper function DFS_Topo(u,
visited, visiting, finished, L) .

6. The DFS_Topo function:

a. Mark u as visited and add to
visiting set.
b. For each neighbor v of u :
i. If v is in visiting set -> Cycle detected,
return error.
ii. If v is not visited :
Recursively call DFS_Topo(v, ...) .
If recursive call returns error, propagate error.
c. Remove u from visiting set.
d. Mark u as finished .
e. Add u to the front of the list L .

7. If no cycle was detected, the final list L contains a

valid topological ordering.

Pseudocode: Kahn's Algorithm

function KahnsTopologicalSort(Graph):
num_vertices = Graph.number_of_vertices()
in_degree = create_map_or_array(size=num_vertices, initial_value=0)
adj = Graph.adjacency_list

// Compute in-degrees
for u in Graph.vertices:
for each neighbor v of u in adj[u]:
in_degree[v] += 1

// Initialize queue with vertices having in-degree 0

queue = new Queue()
for u in Graph.vertices:
if in_degree[u] == 0:
queue.enqueue(u)

sorted_list = new empty_list()

count = 0

// Process vertices from the queue

while queue is not empty:
u = queue.dequeue()
sorted_list.add(u)
count += 1

// Decrease in-degree of neighbors

for each neighbor v of u in adj[u]:
in_degree[v] -= 1
// If in-degree becomes 0, add to queue
if in_degree[v] == 0:
queue.enqueue(v)

// Check for cycles

if count != num_vertices:
return "Error: Graph contains a cycle", null
else:
return sorted_list

• Explanation of Variables (Kahn's):

◦ Graph : Input Directed Acyclic Graph (DAG).

◦ num_vertices : |V|.
 ◦ in_degree : Array/Map storing the in-degree of each
vertex.

◦ adj : Adjacency list representation of the graph.

◦ queue : Queue storing vertices with current in-degree 0.

◦ sorted_list : The list where the topologically sorted

vertices are collected.

◦ count : Counter for the number of vertices added to

sorted_list .

◦ u : Vertex dequeued from the queue.

◦ v : Neighbor of u .

Pseudocode: DFS-Based Algorithm

function DFSTopologicalSort(Graph):
num_vertices = Graph.number_of_vertices()
visited = create_set()
visiting = create_set() // For cycle detection
sorted_list = new empty_linked_list() // Efficient prepend
has_cycle = false

function DFS_Visit(u):
nonlocal has_cycle
if has_cycle: return

visited.add(u)
visiting.add(u)

for each neighbor v of u in Graph.adjacency_list[u]:

if v in visiting:
has_cycle = true
return
if v not in visited:
DFS_Visit(v)
if has_cycle: return

visiting.remove(u)
// Add to the front of the list *after* visiting all descendants
sorted_list.prepend(u)

// Call DFS for all unvisited vertices

for u in Graph.vertices:
if u not in visited:
DFS_Visit(u)
if has_cycle:
return "Error: Graph contains a cycle", null

return list(sorted_list) // Convert linked list to regular list

• Explanation of Variables (DFS-Based):

◦ Graph : Input directed graph.

◦ num_vertices : |V|.

◦ visited : Set of vertices for which DFS has been initiated

or completed.
 ◦ visiting : Set of vertices currently in the recursion stack
(on the current DFS path).

◦ sorted_list : Linked list storing the result (prepended).

◦ has_cycle : Boolean flag to indicate cycle detection.

◦ DFS_Visit : Recursive helper function.

◦ u : Current vertex being visited.

◦ v : Neighbor of u .

Time Complexity Analysis (Both Algorithms):

* Kahn's: Computing in-degrees takes O(|V| + |E|). Initializing queue takes O(|V|).
The while loop processes each vertex and edge once. Total time = O(|V| + |E|).
* DFS-Based: Standard DFS complexity. Total time = O(|V| + |E|).

Space Complexity (Both Algorithms):

* Kahn's: O(|V|) for in-degrees, queue, and result list.
* DFS-Based: O(|V|) for visited/visiting sets, recursion stack, and result list.
* Total space = O(|V|).

University Exam Style Question:

* Question: What is a topological sort of a Directed Acyclic Graph (DAG)?
Describe one algorithm (either Kahn's or DFS-based) to compute a topological sort
and state its time complexity.
* Answer: A topological sort of a DAG is a linear ordering of its vertices such that
for every directed edge (u, v) , vertex u appears before vertex v in the
ordering. It represents a valid sequence for tasks with dependencies. (Describe
either Kahn's or DFS-based algorithm here, including pseudocode steps). For
example, Kahn's algorithm computes in-degrees, adds nodes with in-degree 0 to a
queue, and then repeatedly dequeues a node u , adds it to the sorted list, and
decrements the in-degree of its neighbors v , adding v to the queue if its in-degree
becomes 0. Both Kahn's and the DFS-based algorithm have a time complexity of O(|
V| + |E|).
Network Flow Algorithm

Concept:
Network flow problems deal with graphs where edges have capacities representing
the maximum rate at which something (like fluid, data, goods, etc.) can flow through
them. The goal is often to find the maximum possible flow from a designated source
vertex s to a designated sink (or target) vertex t .

Formal Definitions:
* Flow Network: A directed graph G = (V, E) where each edge (u, v) has a non-
negative capacity c(u, v) . It includes a source s and a sink t .
* Flow: A function f(u, v) assigning a flow value to each edge, satisfying:
* Capacity Constraint: 0 ≤ f(u, v) ≤ c(u, v) for all edges (u, v) .
* Skew Symmetry: f(u, v) = -f(v, u) (flow from u to v is negative flow
from v to u).
* Flow Conservation: For every vertex u except s and t , the total flow
entering u equals the total flow leaving u (∑v f(v, u) = 0).
* Value of Flow: The total net flow leaving the source s (which equals the total net
flow entering the sink t ). Value(f) = ∑v f(s, v).
* Maximum Flow Problem: Find a flow f such that Value(f) is maximized.

Max-Flow Min-Cut Theorem:

A fundamental theorem stating that the maximum flow value from s to t in a
network is equal to the minimum capacity of an s-t cut. An s-t cut is a partition of the
vertices V into two sets, S and T, such that s is in S and t is in T. The capacity of
the cut is the sum of capacities of all edges going from a vertex in S to a vertex in T.

Ford-Fulkerson Method:

Concept:
Ford-Fulkerson is a general method or framework for solving the maximum flow
problem. It's an iterative approach. It starts with zero flow and repeatedly finds an
augmenting path from s to t in the residual graph. An augmenting path is a
path along which we can push more flow. The flow is increased along this path, and
the residual graph is updated. This continues until no more augmenting paths can be
found.
Residual Graph (Gf):
Given a flow f , the residual graph Gf represents the remaining capacity for pushing
more flow. For each edge (u, v) in the original graph G:
* If f(u, v) < c(u, v) , Gf includes a forward edge (u, v) with residual
capacity c_f(u, v) = c(u, v) - f(u, v) .
* If f(u, v) > 0 , Gf includes a backward edge (v, u) with residual
capacity c_f(v, u) = f(u, v) (representing the possibility of

pushing flow back").

Theoretical Steps (Ford-Fulkerson Method):

1. Initialize flow f(u, v) = 0 for all edges (u, v) in G.
2. While there exists an augmenting path p from s to t in the residual graph Gf:
3. Calculate the residual capacity (bottleneck capacity) of the path p:
c_f(p) = min{c_f(u, v) | (u, v) is an
edge in p} .
4. Increase the flow along path p by c_f(p) :
* For each forward edge (u, v) in p , increase f(u, v) by c<sub>f</
sub>(p) .
* For each backward edge (v, u) in p (corresponding to original edge (u,
v) ), decrease f(u, v) by c_f(p) .
5. Update the residual graph Gf based on the new flow f .
6. When no more augmenting paths can be found from s to t in Gf, the current
flow f is the maximum flow.

Finding Augmenting Paths:

The Ford-Fulkerson method itself doesn't specify how to find the augmenting path in
step 2. Different choices lead to different algorithms with varying complexities:
* Original Ford-Fulkerson: Could use DFS or any path-finding method. If
capacities are integers, it terminates, but complexity can be high if paths are chosen
poorly (potentially dependent on the maximum flow value).
* Edmonds-Karp Algorithm: A specific implementation of Ford-Fulkerson that
uses Breadth First Search (BFS) to find the augmenting path in the residual graph.
BFS guarantees finding the shortest augmenting path in terms of the number of
edges.

Edmonds-Karp Algorithm (Specific Implementation of Ford-Fulkerson):

Pseudocode: Edmonds-Karp
function EdmondsKarp(Graph, s, t):
num_vertices = Graph.number_of_vertices()
flow = create_2d_array(num_vertices, num_vertices, initial_value=0)
max_flow = 0

while true:
// Find augmenting path using BFS on residual graph G_f
// parent[v] stores the predecessor of v in the BFS path from s
parent = create_map_or_array(size=num_vertices, initial_value=nu
queue = new Queue()

queue.enqueue(s)
parent[s] = s // Mark source as visited

path_found = false
while queue is not empty:
u = queue.dequeue()
if u == t:
path_found = true
break // Path found

// Explore neighbors in residual graph

// Consider forward edges (original edge u->v with remaining
for each neighbor v of u:
residual_capacity = Graph.capacity(u, v) - flow[u][v]
if residual_capacity > 0 and parent[v] == null:
parent[v] = u
queue.enqueue(v)

// Consider backward edges (original edge v->u with existing

for each vertex v that has edge to u:
residual_capacity = flow[v][u] // Capacity of backward
if residual_capacity > 0 and parent[v] == null:
parent[v] = u // Store predecessor for path reconst
queue.enqueue(v)

// If no augmenting path found, break the main loop

if not path_found:
break

// Calculate bottleneck capacity of the found path

path_flow = infinity
curr = t
while curr != s:
 prev = parent[curr]
# Determine if (prev, curr) is forward or backward edge in r
if Graph.has_edge(prev, curr): # Forward edge used
residual_cap = Graph.capacity(prev, curr) - flow[prev][c
else: # Backward edge used (original edge curr->prev)
residual_cap = flow[curr][prev]

path_flow = min(path_flow, residual_cap)

curr = prev

// Augment flow along the path

max_flow += path_flow
v = t
while v != s:
u = parent[v]
if Graph.has_edge(u, v): # Forward edge used
flow[u][v] += path_flow
else: # Backward edge used (original edge v->u)
flow[v][u] -= path_flow
v = u

return max_flow, flow

• Explanation of Variables:
◦ Graph : Input flow network with capacities
Graph.capacity(u, v) .

◦ s , t : Source and sink vertices.

◦ flow : 2D array storing the current flow f(u, v) on each

edge.

◦ max_flow : The total maximum flow value found.

◦ parent : Array/Map used by BFS to store the augmenting

path found in the residual graph.

◦ queue : Queue used by BFS.

◦ u , v : Vertices being processed by BFS.

◦ residual_capacity : Capacity of an edge in the current

residual graph.

◦ path_flow : The bottleneck capacity of the augmenting

path found.
 ◦ curr , prev : Variables used for traversing the found path
backwards.

Time Complexity Analysis (Edmonds-Karp):

* Finding an augmenting path using BFS takes O(|E|) time (since the residual graph
has at most 2|E| edges).
* The number of augmentations required is bounded by O(|V| * |E|).
* Total time complexity = O(|V| * |E|²).
* (More advanced algorithms like Dinic's algorithm achieve better complexities, e.g.,
O(|V|²|E|) or even faster for specific graph types).

Space Complexity: O(|V|²) for storing flow, O(|V|) for BFS auxiliary structures
(parent, queue).

University Exam Style Question:

* Question: Explain the concept of the Max-Flow Min-Cut theorem. Describe the
general Ford-Fulkerson method for finding maximum flow, including the role of the
residual graph and augmenting paths. What specific algorithm uses BFS to find
augmenting paths within this method, and what is its time complexity?
* Answer: The Max-Flow Min-Cut theorem states that the maximum flow value
from a source s to a sink t in a network equals the minimum capacity of an s-t
cut (a partition of vertices separating s from t ). The Ford-Fulkerson method finds
max flow iteratively. It starts with zero flow and repeatedly finds an augmenting path
(a path with available capacity) from s to t in the residual graph. The residual
graph represents remaining capacities on forward edges and flow that can be pushed
back on backward edges. The flow is increased along the augmenting path by its
bottleneck capacity, and the residual graph is updated. This repeats until no more
augmenting paths exist. The Edmonds-Karp algorithm is an implementation of Ford-
Fulkerson that uses BFS to find the augmenting path, guaranteeing finding the
shortest path in terms of edges. Its time complexity is O(|V| * |E|²).

Practice Questions (Chapter 3)

1. Question: Given an unweighted graph, which traversal algorithm (DFS

or BFS) is suitable for finding the shortest path (in terms of number of
edges) between two nodes? Explain why.
Answer: Breadth First Search (BFS) is suitable for finding the shortest
 path in an unweighted graph. BFS explores the graph layer by layer,
meaning it finds all nodes at distance 1 from the source, then all nodes at
distance 2, and so on. Therefore, the first time it reaches the target node, it
is guaranteed to have found a path with the minimum number of edges.

2. Question: You need to find the cheapest way to connect all computers in
an office using network cables. The cost of laying cable between any two
computer locations is known. Which type of graph algorithm should you
use (e.g., Shortest Path, MST, Network Flow, Topological Sort)? Justify
your choice.
Answer: You should use a Minimum Spanning Tree (MST) algorithm
(like Prim's or Kruskal's). The problem asks for connecting all computers
(spanning all vertices) with the minimum total cost (minimum sum of
edge weights) without creating redundant connections (forming a tree, no
cycles). This is precisely the definition of the MST problem.

3. Question: Can Dijkstra's algorithm be used to find the shortest path in a

graph with negative edge weights? If not, which algorithm should be used
for the single-source shortest path problem in such cases, and what is its
main drawback compared to Dijkstra's?
Answer: No, Dijkstra's algorithm cannot reliably find the shortest path in
a graph with negative edge weights because its greedy approach might
commit to a path that seems shortest locally but is suboptimal globally
due to a negative edge encountered later. For the single-source shortest
path problem with potentially negative edges, the Bellman-Ford
algorithm should be used. Its main drawback compared to Dijkstra's is its
higher time complexity (O(|V||E|) vs O(|E| log |V|)). Bellman-Ford can
also detect negative cycles.

Further Learning: Video Resources

• DFS: https://www.youtube.com/watch?
v=PMMc4VsIacU&list=PLU7rFVD05MosxuUf9LV4ML6uzsSRol8nE&index=10
(From your playlist)
 • BFS: https://www.youtube.com/watch?
v=oDqjPvD54Ss&list=PLU7rFVD05MosxuUf9LV4ML6uzsSRol8nE&index=11
(From your playlist)

• Dijkstra's Algorithm: https://www.youtube.com/watch?

v=XB4MIexjvY0&list=PLU7rFVD05MosxuUf9LV4ML6uzsSRol8nE&index=16
(From your playlist)

• Bellman-Ford Algorithm: https://www.youtube.com/watch?

v=hxMWjiVw71s (Knowledge Gate)

• Floyd-Warshall Algorithm: https://www.youtube.com/watch?

v=oNI0rf2P9gE (Abdul Bari)

• Prim's MST Algorithm: https://www.youtube.com/watch?

v=HnD67eHGgjQ&list=PLU7rFVD05MosxuUf9LV4ML6uzsSRol8nE&index=14
(From your playlist)

• Kruskal's MST Algorithm: https://www.youtube.com/watch?

v=3rrNHYS9OcU&list=PLU7rFVD05MosxuUf9LV4ML6uzsSRol8nE&index=15
(From your playlist)

• Topological Sort (Kahn's): https://www.youtube.com/watch?

v=cIBFEhDCSL8&list=PLU7rFVD05MosxuUf9LV4ML6uzsSRol8nE&index=12
(From your playlist)

• Max Flow (Ford-Fulkerson/Edmonds-Karp): https://

www.youtube.com/watch?v=GiN3jRdgxU4 (Abdul Bari)

(Note: Please verify these links. If the playlist links are incorrect or missing specific
topics, the alternative suggestions provide good explanations.)
Chapter 4: Tractable and Intractable
Problems
In the previous chapters, we explored various algorithms and strategies for solving
problems like sorting, searching, finding shortest paths, and building minimum
spanning trees. We also analyzed their efficiency, often finding algorithms that run in
polynomial time (like O(n), O(n log n), O(n²), O(n³)), which we generally consider
efficient or "tractable" for practical purposes.

However, not all computational problems are created equal. Some problems seem
inherently harder than others. For instance, we saw that the brute-force solution for
the Traveling Salesperson Problem (TSP) takes factorial time (O(n!)), and even the
dynamic programming approach takes exponential time (O(n² * 2n)). These
algorithms become impractical very quickly as the number of cities grows. Are there
faster, polynomial-time algorithms for problems like TSP, or are they fundamentally
"harder"?

This chapter delves into the fascinating theory of computational complexity, which
classifies problems based on the resources (primarily time) required to solve them.
We will explore the concepts of tractability and intractability, introduce fundamental
complexity classes like P and NP, and discuss the significance of NP-complete
problems – the problems widely believed to be intractable.

Computability and Efficiency

Before discussing efficiency, it's worth briefly mentioning computability. Are all
problems solvable by any algorithm, regardless of how long it takes? The theory of
computation, pioneered by Alan Turing, showed that the answer is no. There are
problems, like the famous Halting Problem (determining whether an arbitrary
program will eventually stop or run forever), that are undecidable – no algorithm
can exist that solves them correctly for all possible inputs.
However, for most problems encountered in practice, we know algorithms exist. The
critical question then becomes: do efficient algorithms exist? This is where
complexity theory comes in, focusing on classifying decidable problems based on
their resource requirements.

Complexity Classes: P vs NP

Complexity theory categorizes problems into classes based on the computational

resources needed to solve them. The most fundamental distinction for practical
purposes is between problems solvable in polynomial time and those that are not (or
at least, not known to be).

Class P (Polynomial Time)

Definition:
The complexity class P consists of all decision problems that can be solved by a
deterministic algorithm in polynomial time. A decision problem is one with a yes/no
answer (e.g., "Is this list sorted?", "Does a path exist from A to B with cost less than
K?").

Polynomial time means the algorithm's running time is bounded by O(nk) for some
constant k , where n is the size of the input.

Examples of Problems in P:
* Searching an element in a list (O(n) or O(log n) if sorted).
* Sorting a list (e.g., Merge Sort O(n log n), Bubble Sort O(n²)).
* Checking if a graph is connected (O(|V| + |E|) using BFS/DFS).
* Finding the Minimum Spanning Tree (O(|E| log |V|) using Prim's/Kruskal's).
* Single-Source Shortest Path with non-negative weights (O(|E| log |V|) using
Dijkstra's).
* Matrix multiplication (O(n³), or faster with advanced algorithms).

Significance:
Problems in P are generally considered tractable or efficiently solvable. While an
O(n100) algorithm isn't practical, most polynomial-time algorithms encountered have
low exponents (like 1, 2, or 3). The key idea is that their running time doesn't
explode exponentially as the input size grows.

Class NP (Non-deterministic Polynomial Time)

Definition:
The complexity class NP consists of all decision problems for which a proposed
solution (often called a certificate or witness) can be verified by a deterministic
algorithm in polynomial time.

Explanation:
Imagine someone gives you a potential solution to a decision problem. If you can
check whether that solution is correct in polynomial time, the problem is in NP.

The name "Non-deterministic Polynomial Time" comes from an alternative

definition involving a hypothetical non-deterministic Turing machine (a machine that
can explore multiple computation paths simultaneously or "guess" the right path). A
problem is in NP if such a machine could solve it in polynomial time.

For beginners, the verification definition is more intuitive:

* "Is there a path in this graph from s to t with total weight less than 100?" If
someone gives you a path, you can easily sum its weights and check if it's less than
100 in polynomial time. So, this problem is in NP.
* "Does this graph have a Hamiltonian cycle (a tour visiting each city exactly
once)?" If someone gives you a sequence of vertices, you can check in polynomial
time if it forms a valid Hamiltonian cycle. So, TSP (decision version) is in NP.
* "Is this Boolean formula satisfiable?" If someone gives you a truth assignment
(true/false values for variables), you can plug them into the formula and evaluate it in
polynomial time to see if it results in true. So, SAT is in NP.

Important Relationship: P ⊆ NP
Every problem in P is also in NP. Why? If a problem can be solved in polynomial
time, then a proposed solution can certainly be verified in polynomial time. To verify,
you can simply ignore the proposed solution and solve the problem yourself in
polynomial time! If your solution matches the required outcome (e.g., the answer is
"yes"), then the verification succeeds.
The P vs NP Question:
One of the biggest unsolved questions in computer science and mathematics is
whether P = NP.
* If P = NP, it means that every problem for which a solution can be quickly verified
can also be quickly solved. This would have profound implications, potentially
leading to efficient solutions for many currently hard optimization problems.
* If P ≠ NP (which is widely believed), it means there are problems in NP that cannot
be solved in polynomial time. These problems are inherently harder than those in P.

The Clay Mathematics Institute has offered a US$1 million prize for a correct proof
of either P = NP or P ≠ NP.

Class NP-complete (NPC)

Definition:
A decision problem C is NP-complete if it satisfies two conditions:
1. C is in NP: Proposed solutions to C can be verified in polynomial time.
2. C is NP-hard: Every other problem in NP can be reduced to C in polynomial time
(explained below).

Significance:
NP-complete problems are, in a sense, the hardest problems in NP. They are the
problems that capture the essence of the difficulty of the entire class NP.

• If any single NP-complete problem could be solved by a polynomial-time

algorithm, then every problem in NP could be solved in polynomial time
(meaning P = NP).

• Conversely, if P ≠ NP, then no NP-complete problem can be solved in

polynomial time.

Discovering that a problem is NP-complete is strong evidence that searching for an

efficient, exact polynomial-time algorithm for it is likely futile. Efforts should
instead focus on other approaches like approximation algorithms or heuristics.
Class NP-hard
Definition:
A problem H is NP-hard if every problem in NP can be reduced to H in polynomial
time. In other words, H is at least as hard as any problem in NP.

NP-hard vs NP-complete:
* NP-complete problems must be both NP-hard and belong to the class NP (meaning
they are decision problems verifiable in polynomial time).
* NP-hard problems only need to satisfy the "at least as hard as any problem in NP"
condition. They don't necessarily have to be in NP themselves.
* Example: The optimization version of TSP ("Find the shortest Hamiltonian cycle")
is NP-hard. It's not technically in NP because NP deals with decision problems (yes/
no answers). However, it's clearly related to the decision version ("Is there a cycle
with length < K?"), which is NP-complete.
* Example: The Halting Problem is undecidable, but it is also NP-hard (though
proving this is complex).

If a polynomial-time algorithm exists for any NP-hard problem, then P = NP.

Visualizing the Classes (Assuming P ≠ NP):

Imagine a hierarchy:
* P: Problems solvable efficiently.
* NP: Problems verifiable efficiently. Contains P.
* NP-complete: The hardest problems within NP. Reside at the boundary of NP (if
P≠NP).
* NP-hard: Problems at least as hard as NP-complete problems. Includes NPC
problems and potentially harder problems outside NP.

Cook's Theorem (Cook-Levin

Theorem)

How do we know NP-complete problems even exist? The foundation was laid by
Stephen Cook (and independently by Leonid Levin) in 1971.
Theorem Statement (Simplified):
The Boolean Satisfiability Problem (SAT) is NP-complete.

What is SAT?
Given a Boolean formula involving variables (which can be TRUE or FALSE),
logical ANDs (∧), logical ORs (∨), and logical NOTs (¬), the SAT problem asks: Is
there an assignment of TRUE/FALSE values to the variables that makes the entire
formula evaluate to TRUE?

• Example: Formula: (x ∨ ¬y) ∧ (¬x ∨ y)

◦ Assignment x=TRUE, y=TRUE :
(T ∨ F) ∧ (F ∨ T) = T ∧ T = TRUE . Yes, this
formula is satisfiable.

• Example: Formula: (x) ∧ (¬x)

◦ No assignment can make this true. It's unsatisfiable.

Significance of Cook's Theorem:

* It was the first problem proven to be NP-complete.
* The proof (which is complex and involves simulating non-deterministic Turing
machines with Boolean formulas) showed that any problem in NP can be reduced to
SAT in polynomial time.
* This established the existence of NP-complete problems and provided a starting
point (SAT) from which the NP-completeness of many other problems could be
proven using reductions.

Reduction Techniques

Once we have one known NP-complete problem (like SAT), we don't need to repeat
Cook's complex proof for every new problem. Instead, we use polynomial-time
reductions.

Concept:
A problem A is polynomial-time reducible to problem B (written A ≤P B) if there
exists a polynomial-time algorithm (a reduction) that transforms any instance x of
problem A into an instance y of problem B, such that the answer to x (for
problem A) is "yes" if and only if the answer to y (for problem B) is "yes".
How Reductions Prove NP-completeness:
To prove that a problem X is NP-complete, we need to do two things:
1. Show X is in NP: Demonstrate that a proposed solution for X can be verified in
polynomial time.
2. Show X is NP-hard: Choose a known NP-complete problem Y (e.g., SAT, 3-SAT,
Vertex Cover) and show that Y ≤P X. This means designing a polynomial-time
algorithm that transforms any instance of Y into an instance of X while preserving
the yes/no answer.

If we can reduce a known hard problem Y to our new problem X efficiently, it

implies that X must be at least as hard as Y. Since Y is NP-complete (meaning all NP
problems reduce to it), X must also be NP-hard. Combined with step 1 (X is in NP),
this proves X is NP-complete.

Example Sketch: Reducing 3-SAT to Vertex Cover

* 3-SAT: A version of SAT where the formula is in Conjunctive Normal Form (CNF)
with exactly 3 literals per clause. (e.g., (x ∨ y ∨ ¬z) ∧ (¬x ∨ ¬y ∨ w)
∧ ... ). 3-SAT is known to be NP-complete.
* Vertex Cover: Given a graph G=(V, E) and an integer k, is there a subset of
vertices V' ⊆ V such that |V'| ≤ k and every edge in E has at least one endpoint in V'?
(Does a small set of vertices "touch" all edges?).
* Reduction Idea: Construct a specific graph G from a given 3-SAT formula. The
graph has components representing variables and clauses. Edges connect these
components in a way that a vertex cover of a certain size k in G exists if and only if
the original 3-SAT formula is satisfiable. The construction must be doable in
polynomial time.
* Conclusion: Since 3-SAT (known NPC) reduces to Vertex Cover in polynomial
time, and Vertex Cover is verifiable in polynomial time (given a subset V', check its
size and if it covers all edges), Vertex Cover is also NP-complete.

This reduction process has been used to build a vast web of known NP-complete
problems.

Standard NP-complete Problems

Knowing common NP-complete problems helps in recognizing potential

intractability when encountering new problems.
Here are a few famous examples (all are decision problems):

1. SAT (Boolean Satisfiability): Given a Boolean formula, is it satisfiable?

2. 3-SAT: Given a Boolean formula in 3-CNF (AND of clauses, each clause

is OR of 3 literals), is it satisfiable? (Often used as the starting point for
reductions).

3. Vertex Cover: Given a graph G and integer k, is there a vertex cover of

size at most k?

4. Clique: Given a graph G and integer k, does G contain a clique (a subset

of vertices where every pair is connected by an edge) of size at least k?

5. Independent Set: Given a graph G and integer k, does G contain an

independent set (a subset of vertices where no two vertices are connected
by an edge) of size at least k? (Closely related to Clique and Vertex
Cover).

6. Hamiltonian Cycle: Given a graph G, does it contain a cycle that visits

every vertex exactly once?

7. Traveling Salesperson Problem (TSP - Decision Version): Given a set

of cities, distances between them, and a budget k, is there a tour visiting
all cities exactly once with a total length of at most k?

8. Subset Sum: Given a set of integers S and a target integer k, is there a

subset of S whose elements sum up to exactly k?

9. 0/1 Knapsack (Decision Version): Given items with weights and values,
a capacity W, and a target value V, is it possible to choose items such that
their total weight is ≤ W and their total value is ≥ V?

10. Graph Coloring (Decision Version): Given a graph G and an integer k,

can the vertices of G be colored using at most k colors such that no two
adjacent vertices share the same color?

If you encounter a new problem and suspect it might be hard, try to see if it looks
similar to one of these known NP-complete problems. You might be able to prove its
NP-completeness by reducing a known NPC problem to it.
Dealing with NP-Hardness

If a problem you need to solve is proven or strongly suspected to be NP-hard (or NP-
complete), what can you do? Searching for an efficient, exact, polynomial-time
algorithm is likely doomed.

Common strategies include:

1. Approximation Algorithms: Find algorithms that run in polynomial time but
guarantee a solution that is within a certain factor of the optimal one (e.g., the
Christofides algorithm for TSP guarantees a solution within 1.5 times the optimal
length). (Covered in Chapter 5).
2. Randomized Algorithms: Use randomness to find a solution that is likely good or
correct, often faster than deterministic methods. (Covered in Chapter 5).
3. Heuristics: Use problem-specific rules of thumb or greedy approaches that work
well in practice but have no guarantee of optimality (e.g., Nearest Neighbor for
TSP). (Mentioned in Chapter 2).
4. Exact Exponential-Time Algorithms: If the input size n is small enough (e.g.,
n < 25 for TSP, n < 60 for some problems), an exponential-time algorithm (like
dynamic programming or branch-and-bound for TSP) might still be feasible.
5. Solving Special Cases: Identify restricted versions of the problem that might be
solvable in polynomial time (e.g., TSP on a line, graph coloring for bipartite graphs).

Understanding the boundary between P and NP-complete helps guide algorithm

design efforts towards the most appropriate techniques.

Practice Questions (Chapter 4)

1. Question: Explain the difference between the complexity classes P and

NP. Is P = NP? Why is this question important?
Answer: Class P contains decision problems solvable by a deterministic
algorithm in polynomial time (O(nk)). Class NP contains decision
problems where a proposed solution can be verified in polynomial time.
We know P is a subset of NP (P ⊆ NP). The question of whether P = NP
is a major unsolved problem. If P=NP, many problems currently
considered hard (like TSP, SAT) could be solved efficiently. If P≠NP
 (which is widely believed), then there are problems in NP inherently
harder than those in P, justifying the need for approximation algorithms,
heuristics, etc., for NP-complete problems.

2. Question: What does it mean for a problem to be NP-complete? Name

two standard NP-complete problems.
Answer: A problem is NP-complete if it is in NP (verifiable in
polynomial time) AND it is NP-hard (at least as hard as any problem in
NP; every problem in NP reduces to it in polynomial time). NP-complete
problems are the hardest problems in NP. If any one of them can be
solved in polynomial time, then P=NP. Two standard NP-complete
problems are SAT (Boolean Satisfiability) and Vertex Cover.

3. Question: What is a polynomial-time reduction? How is it used to prove

that a problem is NP-complete?
Answer: A polynomial-time reduction from problem A to problem B is a
polynomial-time algorithm that transforms any instance x of A into an
instance y of B such that x has a "yes" answer if and only if y has a
"yes" answer. To prove a problem X is NP-complete, we first show X is
in NP (verifiable). Then, we choose a known NP-complete problem Y and
design a polynomial-time reduction from Y to X (Y ≤P X). This shows X
is NP-hard. Since X is in NP and is NP-hard, it is NP-complete.

Further Learning: Video Resources

• P vs NP Introduction: https://www.youtube.com/watch?
v=YX40hbAHx3s (Abdul Bari - P vs NP)

• Complexity Classes (P, NP, NPC, NPH): https://www.youtube.com/

watch?v=gCpAE4K38-0 (Knowledge Gate - Complexity Classes)

• NP-Completeness and Reductions: https://www.youtube.com/watch?

v=e2cF8a5aAhE (Abdul Bari - NP Completeness)

• Cook-Levin Theorem: https://www.youtube.com/watch?

v=pQsdygaYcE4 (Simple explanation, less formal)

(Note: These concepts can be quite theoretical. Watching multiple explanations can
be helpful.)
Chapter 5: Advanced Topics
Having explored fundamental algorithms, design strategies, and the theoretical
landscape of problem complexity (P vs NP), we now venture into some advanced
techniques and concepts. When faced with NP-hard problems where exact, efficient
solutions are elusive, computer scientists employ sophisticated strategies like
approximation and randomization.

Furthermore, the hierarchy of complexity doesn't end with NP. There are problems
that might require even more resources, specifically memory (space), leading us to
classes like PSPACE.

This chapter provides an introduction to these advanced topics, equipping you with
an understanding of how to approach computationally hard problems and glimpse the
broader map of computational complexity.

Approximation Algorithms

Concept:
For many optimization problems (especially NP-hard ones like TSP, Vertex Cover,
Set Cover), finding the absolute optimal solution might take an infeasible amount of
time (e.g., exponential). Approximation algorithms offer a compromise: they run in
polynomial time but aim to find a solution that is provably close to the optimal one,
even if not perfectly optimal.

Instead of guaranteeing the best possible answer, they guarantee an answer that is
within a certain factor of the best possible answer.

Approximation Ratio:
The quality of an approximation algorithm is measured by its approximation ratio
(or factor). For a minimization problem, an algorithm has an approximation ratio of ρ
(rho) ≥ 1 if, for any input instance, the cost C of the solution produced by the
algorithm is at most ρ times the cost C of the optimal solution:
C≤ρ*C
For a maximization problem, the ratio ρ ≤ 1 is defined such that the value V of the
produced solution is at least ρ times the value V of the optimal solution:
V≥ρ*V
(Sometimes, for maximization, the ratio is defined as V*/V ≥ 1, similar to
minimization).

A ratio close to 1 indicates a better approximation.

When to Use:
* When solving NP-hard optimization problems where finding the exact optimum is
too slow.
* When a near-optimal solution is acceptable and significantly faster to obtain.
* When the problem structure allows for guarantees on solution quality.

Example 1: Vertex Cover Approximation

• Problem Recap: Given an undirected graph G=(V, E), find a minimum-

size subset of vertices V' ⊆ V such that every edge in E has at least one
endpoint in V'. (This is the optimization version, which is NP-hard).

• Approximation Algorithm Idea (Simple Greedy): Repeatedly select an

edge that hasn't been covered yet and add both its endpoints to the vertex
cover. This ensures the selected edge (and potentially others) is covered.

• Theoretical Steps:

1. Initialize the vertex cover set C to be empty.

2. Initialize a set E' containing all edges of the graph G.

3. While E' is not empty:

4. Select an arbitrary edge (u, v) from E' .

5. Add both vertex u and vertex v to the cover set C .

6. Remove edge (u, v) from E' .

7. Remove all other edges from E' that are incident to either
u or v (as they are now covered).

8. Return the cover set C .

• Pseudocode: Approximation Algorithm for Vertex Cover

function ApproxVertexCover(Graph):
C = new empty_set() // The vertex cover set
E_prime = Graph.get_all_edges() // Copy of edge set

while E_prime is not empty:

// Select an arbitrary edge (u, v) from E_prime
(u, v) = E_prime.pick_any_edge()

// Add both endpoints to the cover

C.add(u)
C.add(v)

// Remove edges incident to u or v from E_prime

edges_to_remove = new empty_list()
for each edge (x, y) in E_prime:
if x == u or x == v or y == u or y == v:
edges_to_remove.add((x, y))

for each edge edge_to_remove in edges_to_remove:

E_prime.remove(edge_to_remove)

return C

• Explanation of Variables:

◦ Graph : Input undirected graph.

◦ C : Set storing the vertices in the calculated cover.

◦ E_prime : Set of edges not yet covered.

◦ (u, v) : An arbitrary edge selected from E_prime .

◦ edges_to_remove : Temporary list to hold edges incident

to the selected u or v .

• Time Complexity Analysis:

◦ The while loop runs at most |E| times (in the worst case, we
pick one edge at a time).

◦ Inside the loop, finding incident edges and removing them can
be done efficiently, often related to the degrees of u and v .
With appropriate data structures (like adjacency lists and
 marking edges), the total time can be kept polynomial, often
around O(|V| + |E|) or O(|E|).

• Approximation Ratio Analysis (Sketch):

◦ Let A be the set of edges selected in step 4 across all

iterations.

◦ No two edges in A share an endpoint (because once an edge

(u, v) is picked, all edges incident to u or v are
removed).

◦ The size of the cover C produced is exactly 2 * |A| (we

add both endpoints for each edge in A ).

◦ Let C* be the optimal minimum vertex cover. To cover the

edges in A , C* must contain at least one endpoint for each
edge in A . Since edges in A don't share endpoints, C*
must contain at least |A| vertices.

◦ Therefore, |C*| ≥ |A| .

◦ We have |C| = 2 * |A| . Substituting |A| ≤ |C*| ,

we get |C| ≤ 2 * |C*| .

◦ This algorithm is a 2-approximation algorithm for Vertex

Cover. The size of the cover it finds is guaranteed to be no
more than twice the size of the optimal minimum cover.

• University Exam Style Question:

◦ Question: What is an approximation algorithm, and what is

meant by an approximation ratio? Describe a polynomial-time
2-approximation algorithm for the Minimum Vertex Cover
problem.

◦ Answer: An approximation algorithm is a polynomial-time

algorithm for an optimization problem (often NP-hard) that
finds a provably good, but not necessarily optimal, solution.
Its quality is measured by the approximation ratio ρ, which
bounds how far the found solution's cost/value can be from the
optimal one (e.g., C ≤ ρ * C* for minimization). A 2-
approximation algorithm for Vertex Cover works as follows:
 While there are uncovered edges, pick an arbitrary uncovered
edge (u, v), add both u and v to the cover set, and remove all
edges incident to u or v. This runs in polynomial time and
guarantees a cover size at most twice the optimal size.

Example 2: TSP Approximation (Metric TSP)

• Problem: Find a minimum cost tour visiting all cities exactly once, with
the assumption of Metric TSP: the distances satisfy the triangle
inequality ( dist(i, j) ≤ dist(i, k) + dist(k, j) for
any cities i, j, k). This is common for Euclidean distances.

• Approximation Algorithm Idea (MST Double-Edge): Leverage the

fact that MSTs can be found efficiently. Create a tour by traversing the
MST.

• Theoretical Steps:

1. Compute the Minimum Spanning Tree (MST) T of the graph

G representing the cities and distances. (Use Prim's or
Kruskal's - O(|E| log |V|)).

2. Create a multigraph T' by duplicating every edge in the MST

T. Now, every vertex in T' has an even degree.

3. Find an Eulerian circuit in T'. An Eulerian circuit visits every

edge exactly once (possible because all degrees are even). This
can be found in O(|E|) time (Hierholzer's algorithm).

4. Convert the Eulerian circuit into a Hamiltonian cycle (TSP

tour) by taking shortcuts. Traverse the Eulerian circuit.
Maintain a list of visited cities. When moving from city u to
city v in the circuit, if v has already been visited, skip it
and proceed to the next city in the circuit. If v has not been
visited, add it to the tour list and move directly from the
previous city in the tour list to v .

5. The final list of visited cities (in order) forms the approximate
TSP tour.

• Pseudocode: MST-Based TSP Approximation

function ApproxMetricTSP_MST(Graph):
num_vertices = Graph.number_of_vertices()

# 1. Compute MST
mst_edges = PrimsMST(Graph) // Or KruskalsMST

# 2. Create graph with doubled MST edges (implicitly or explicitly)

# Let adj_mst be the adjacency list for the MST
adj_mst = build_adjacency_list_from_edges(mst_edges)

# 3. Find Eulerian circuit (e.g., using Hierholzer's)

eulerian_circuit = find_eulerian_circuit(adj_mst)
# Example: [c1, c2, c3, c2, c4, c1]

# 4. Create tour with shortcuts

visited = create_set()
tsp_tour = new empty_list()

for each city c in eulerian_circuit:

if c not in visited:
tsp_tour.add(c)
visited.add(c)

# Add start city to end to complete the cycle

tsp_tour.add(tsp_tour[0])

return tsp_tour

# Helper functions (conceptual)

function build_adjacency_list_from_edges(edges):
...
function find_eulerian_circuit(adjacency_list):
...

• Time Complexity Analysis: Dominated by MST calculation, O(|E| log |

V|) or potentially O(|V|²) depending on graph density and MST algorithm
choice. Finding the Eulerian circuit and taking shortcuts are typically
faster.
 • Approximation Ratio: This algorithm guarantees a tour length C such
that C ≤ 2 * C* , where C* is the optimal TSP tour length, provided
the triangle inequality holds.

◦ Proof Sketch: Cost(MST) ≤ Cost(Optimal TSP Tour) because

removing one edge from the optimal tour leaves a spanning
tree, which must cost at least as much as the MST. The cost of
the doubled-edge graph T' is 2 * Cost(MST). The Eulerian
circuit cost is Cost(T'). Taking shortcuts only reduces the cost
due to the triangle inequality. So, Cost(Final Tour) ≤
Cost(Eulerian Circuit) = 2 * Cost(MST) ≤ 2 * Cost(Optimal
TSP Tour).

• Note: The Christofides algorithm improves this to a 1.5-approximation by

using minimum-weight perfect matching instead of simply doubling all
MST edges.

Randomized Algorithms

Concept:
Randomized algorithms incorporate randomness as part of their logic. They use
random numbers (generated by a pseudo-random number generator) to make
decisions at certain points during their execution. This randomness can lead to
algorithms that are simpler, faster (in terms of expected time), or sometimes the only
practical way to solve certain problems, especially in distributed systems or when
dealing with adversaries.

Why Use Randomness?

* Simplicity: Random choices can sometimes avoid complex deterministic logic
needed to handle worst-case scenarios.
* Efficiency: Can lead to better average-case or expected performance than
deterministic counterparts (e.g., QuickSort).
* Breaking Symmetry: Useful in distributed systems where deterministic
algorithms might deadlock.
* Dealing with Adversaries: Randomness can prevent an adversary from
consistently providing worst-case inputs.
Types of Randomized Algorithms:

1. Las Vegas Algorithms:

◦ Always produce the correct result.

◦ The running time varies depending on the random choices

made; we analyze the expected running time.

◦ Example: Randomized QuickSort.

2. Monte Carlo Algorithms:

◦ May produce an incorrect result with a certain (usually small

and bounded) probability.

◦ The running time is typically deterministic.

◦ The probability of error can often be reduced by running the

algorithm multiple times.

◦ Example: Miller-Rabin primality test.

Example 1: Randomized QuickSort (Las Vegas)

• Algorithm Idea: The standard QuickSort algorithm's worst-case O(n²)

time occurs when the chosen pivot consistently partitions the array
unevenly (e.g., always picking the smallest or largest element).
Randomized QuickSort mitigates this by choosing the pivot element
randomly from the subarray being sorted.

• Steps: Same as standard QuickSort, except the Partition step

selects a random element within the current subarray A[p..r] , swaps
it with A[r] , and then proceeds with the partitioning around A[r] as
usual.

• Expected Time Complexity: By choosing the pivot randomly, the

probability of consistently getting bad partitions becomes very low. On
average, the partitions are expected to be reasonably balanced. The
expected running time of Randomized QuickSort is O(n log n), matching
the best-case performance of deterministic QuickSort.

• Correctness: Always produces a correctly sorted array (Las Vegas).

Example 2: Miller-Rabin Primality Test (Monte Carlo)

• Problem: Determine if a given large number n is prime or composite.

• Challenge: Deterministic primality testing was historically difficult

(though a polynomial-time algorithm, AKS, exists, it's often slower in
practice). Factoring large numbers is believed to be hard.

• Miller-Rabin Idea: It's a probabilistic test based on properties related to

Fermat's Little Theorem and square roots of unity modulo n . It doesn't
definitively prove primality. Instead:
◦ If n is prime, the test always returns "prime".

◦ If n is composite, the test returns "composite" with a high

probability (e.g., > 3/4 for a single run). It might incorrectly
return "prime" (a false positive) with a small probability (e.g.,
< 1/4).

• Algorithm (High Level): Select a random number a (a potential

"witness"). Perform calculations involving a and n based on number
theory. If the calculations reveal n is definitely composite, return
"composite". If the calculations are consistent with n being prime,
return "prime" (might be wrong if n is composite but a wasn't a good
witness).

• Reducing Error: By repeating the test k times with independent

random choices of a , the probability of incorrectly identifying a
composite number as prime drops exponentially (e.g., < (1/4)k).

• Type: Monte Carlo - it can err (false positive for primality), but its
running time is deterministic (polynomial in log n).

• University Exam Style Question:

◦ Question: What are the two main types of randomized

algorithms? Explain the difference between them in terms of
correctness and running time guarantees. Give an example of
each type.

◦ Answer: The two main types are Las Vegas and Monte Carlo.
Las Vegas algorithms always produce the correct result, but
their running time is random (we analyze expected time).
 Example: Randomized QuickSort. Monte Carlo algorithms
have deterministic running times but may produce an incorrect
result with a bounded probability. Example: Miller-Rabin
primality test (can incorrectly identify a composite as prime
with small probability).

Class of Problems Beyond NP -

PSPACE

Concept:
While the P vs NP question focuses on time complexity (polynomial time solvability/
verifiability), complexity theory also considers other resources, notably space
(memory).

The complexity class PSPACE contains all decision problems that can be solved by
a deterministic algorithm using only a polynomial amount of space (memory), with
no limit on the amount of time used.

Relationship to P and NP:

* P ⊆ PSPACE: If a problem can be solved in polynomial time, it can certainly be
solved using polynomial space, because a polynomial-time algorithm can only access
a polynomial number of memory cells.
* NP ⊆ PSPACE: This is less obvious but true. A way to see this is that we can
verify a potential solution (certificate) for an NP problem by trying all possible
certificates. If the certificate has polynomial length (required for poly-time
verification), we can iterate through all possible certificates one by one, reusing the
same polynomial space for each check. This might take exponential time, but only
requires polynomial space.

So, we have the relationship: P ⊆ NP ⊆ PSPACE

Known Relationships and Open Questions:

* It is known that P ≠ EXPTIME (problems solvable in exponential time).
EXPTIME contains PSPACE.
* It is known that PSPACE = NPSPACE (problems solvable by a non-deterministic
algorithm in polynomial space) - Savitch's Theorem.
* The major open questions remain:
* Is P = NP?
* Is NP = PSPACE?
* Is P = PSPACE?

It is widely suspected that P ≠ NP and NP ≠ PSPACE, implying P ⊂ NP ⊂ PSPACE,

but proofs are lacking.

Intuition: PSPACE problems can take a very long time (potentially exponential or
worse) but must be solvable without using an excessive amount of memory.

PSPACE-complete Problems:
Similar to NP-completeness, PSPACE-complete problems are the hardest problems
in PSPACE. If any PSPACE-complete problem can be solved using only polynomial
time, then P = PSPACE. If any can be solved using polynomial time and polynomial
space, then P = PSPACE.

Examples:
1. Quantified Boolean Formulas (QBF): This is the canonical PSPACE-complete
problem. Given a Boolean formula where variables are quantified using both
universal (∀, "for all") and existential (∃, "there exists") quantifiers (e.g., ∀x ∃y (x ∨
y)), is the formula true?
* Checking satisfiability involves potentially exploring an exponentially large tree of
possibilities, but it can often be done by recursively evaluating sub-formulas, reusing
space.

1. Generalized Games: Many two-player games (like Chess, Checkers, Go)

played on an n x n board, when generalized, have decision problems (e.g.,
"Does the first player have a winning strategy from this position?") that
are PSPACE-complete or even harder (EXPTIME-complete).

2. Regular Expression Equivalence: Determining if two regular

expressions (potentially using extended features like intersection or
negation) define the same language can be PSPACE-complete.
Significance: PSPACE provides a theoretical home for problems that seem harder
than NP problems, often involving game-like scenarios or alternating quantifiers,
where space is the limiting factor rather than just time for verification.

• University Exam Style Question:

◦ Question: Define the complexity class PSPACE. What is the
known relationship between P, NP, and PSPACE? Give an
example of a PSPACE-complete problem.

◦ Answer: PSPACE is the class of decision problems solvable

by a deterministic algorithm using a polynomial amount of
memory (space), with no bound on time. The known
relationship is P ⊆ NP ⊆ PSPACE. Whether these inclusions
are strict (P ≠ NP? NP ≠ PSPACE?) are major open questions.
An example of a PSPACE-complete problem is QBF
(Quantified Boolean Formulas), which asks if a fully
quantified Boolean formula is true.

Chapter Summary

This chapter explored strategies and concepts beyond standard polynomial-time

algorithms and the NP complexity class.
* Approximation Algorithms provide a way to tackle NP-hard optimization
problems by finding polynomial-time solutions that are provably close to optimal,
measured by an approximation ratio.
* Randomized Algorithms leverage randomness for simplicity or efficiency,
categorized into Las Vegas (always correct, random time) and Monte Carlo (can err,
deterministic time) types.
* PSPACE represents problems solvable in polynomial space, potentially requiring
exponential time. It forms a class containing both P and NP, with PSPACE-complete
problems like QBF representing its hardest challenges.

Understanding these advanced topics is crucial for addressing the full spectrum of
computational problems encountered in research and practice.
Practice Questions (Chapter 5)

1. Question: You are given an NP-hard optimization problem. Under what

circumstances would you choose to develop an approximation algorithm
versus using a heuristic like Nearest Neighbor?
Answer: You would favor developing an approximation algorithm if a
provable guarantee on the solution quality (relative to the optimum) is
important and achievable in polynomial time. If such a guarantee is not
needed, not possible, or if extreme speed and simplicity are paramount
even at the cost of potentially very poor solution quality, a simpler
heuristic like Nearest Neighbor might be chosen, despite its lack of
guarantees.

2. Question: Explain the difference between the expected running time

analysis of a Las Vegas algorithm (like Randomized QuickSort) and the
probabilistic correctness analysis of a Monte Carlo algorithm (like Miller-
Rabin).
Answer: For a Las Vegas algorithm, the analysis focuses on the average
running time over all possible random choices, assuming the algorithm
always produces the correct output. We calculate the expected time
complexity (e.g., O(n log n) for Randomized QuickSort). For a Monte
Carlo algorithm, the running time is usually fixed (deterministic), but the
output might be incorrect. The analysis focuses on bounding the
probability of error (e.g., the probability Miller-Rabin incorrectly
identifies a composite as prime is < 1/4 per round).

3. Question: Why is it believed that PSPACE contains problems that are

harder than NP-complete problems (assuming NP ≠ PSPACE)?
Answer: NP problems require solutions (certificates) to be verifiable in
polynomial time. PSPACE problems only require polynomial space for
solving, allowing for potentially exponential time. Problems like QBF
involve alternations of quantifiers (∀, ∃) which seem to require exploring
game trees or nested possibilities that might take exponential time but can
be evaluated recursively using polynomial space. This suggests a
potentially higher level of complexity than just finding a verifiable
certificate (NP).
Further Learning: Video Resources

• Approximation Algorithms (Vertex Cover): https://www.youtube.com/

watch?v=0LjYeDElj3c (Simple explanation)

• Approximation Algorithms (TSP): https://www.youtube.com/watch?

v=ARvQcqJ_-NY&t=1530s (Section within Abdul Bari's Greedy video
touches on TSP heuristics)

• Randomized Algorithms (Intro): https://www.youtube.com/watch?

v=DUSD9FxNZEI (Knowledge Gate)

• Randomized QuickSort: https://www.youtube.com/watch?

v=SL4xXU3PgeA (Knowledge Gate)

• PSPACE: https://www.youtube.com/watch?v=IAqVlJ1t4hA (Brief

overview, part of P vs NP discussion)

• QBF (PSPACE-Complete): https://www.youtube.com/watch?

v=rV94AFKC5gg (More advanced theory video)

(Note: Finding beginner-friendly videos specifically on PSPACE can be challenging,

as it's a more advanced theoretical topic.)