CSC2211 ALGORITHMS

INTRODUCTION

Dr. Shohag Barman

Assistant Professor
, Department of Computer Science
American International University-Bangladesh (AIUB)
[email protected]
Room: DN0119

1
Course Objectives

• The objective of this course is to teach how we can compare

algorithms, the complexity analysis of time and space, effective
data structure for solving difficult problems and implementation
of different algorithms with the practical examples.

• The purpose of the course is

– a) to raise your level of sophistication in thinking about the design and
analysis of algorithms;
– b) learn some of the classic algorithms and recent improvements;
– c) exercise your creativity in designing algorithms.

2
AIUB::CSC2105::Algorithm
Course Prerequisites
• Programming
– Data types, operations
– Conditional statements
– Loops
– Procedures and functions
– C/ C++/ Java
• Discrete Mathematics (proof theorems)
• Data Structures (array, structure, pointer, file, etc...)
• Computer lab (edit, compile, execute, debug)
• If you lack of any of the above please refine yourselves.

3
AIUB::CSC2105::Algorithm
Importance of the course

• This course is a continuation of the courses Programming

Language 1 & 2, and Data Structure.
• Algorithm is required for all areas of computer science –
especially for developing problem solving ability.
• This course will give the basic for the understanding of the
courses –Theory of Computation, Artificial Intelligence, etc.
• This course will give the basic for the understanding of the
concepts – Design and Analysis of Algorithm.

4
AIUB::CSC2105::Algorithm
Course Contents

• RAM model, Basic notation

• Recurrences & Master Method
• Dynamic Programming
• Greedy strategy
• Graphs Algorithms
• Greedy Graph Algorithm
• Shortest Path Algorithms
• Basic idea of NP – Completeness
• Basic idea of Elementary Geometric Methods & Review

5
Resources & References

• Introduction to Algorithms, Third Edition, Thomas H.

Cormen, Charle E. Leiserson, Ronald L. Rivest, Clifford
Stein (CLRS).
• Fundamental of Computer Algorithms, Ellis Horowitz, Sartaj
Sahni, Sanguthevar Rajasekaran (HSR)
• Helpful link for Problem Solving :
http://acm.uva.es/problemset/
• Lectures and Laboratory works will be provided online at the
course website weekly.

6
7
Course Evaluation
Midterm Quiz (Best 1 out of 2) 20
Laboratory Performance 20
Assignment 10
Class Attendance 10
Midterm Written Exam 40
Midterm Total 100 40%
Final term Quiz (Best 1 out of 2) 20
Laboratory Performance 20
Assignment 10
Class Attendance 10
Final term Written Exam 40
Final Term Total 100 60%
Grand Total Final Grade of the Course 100

8
Course Evaluation ( Online Semester )
Marking system for Theory and Lab Marking system for Theory and Lab (Final
(Midterm) Term)
Quizzes 20% Quizzes 20%
Lab Performance 20% Lab Performance 20%
Attendances 10% Attendances 10%
Theory Quizzes (MCQ) 10% Theory Quizzes (MCQ) 10%
Lab Quizzes (MCQ) 10% Lab Quizzes (MCQ) 10%
Assignments 10% Assignments 10%

Viva Exams 20% Viva Exams 20%

Total 100% Total 100%

Grand Total - - - - - - - - 50% of Midterm + 50% of Final Term

9
Welcome to the
course
Algorithms

AIUB::CSC2105::Algorithm Lecture01-Introduction  10
??????

What will we do/learn in

this course?

11
The Goals of this Course

• To think algorithmically
• To understand and learn the idea behind algorithm
design techniques
• To get to know a toolbox of classical algorithms.
• To reason (in a precise and formal way) about the
efficiency and the correctness of algorithms.

12
AIUB::CSC2105::Algorithm
??????

What is
Algorithm?

13
AIUB::CSC2105::Algorithm
 Informally
produces
takes some some value
value or set Algorithm: any well- or set of
of values as defined values, as
input computational output
procedure

• An algorithm is thus a sequence of computational steps that transform the input into the
output.
• Solving a given problem:
– Data structure: Organization of data to solve the problem at hand.
– Algorithm: Outline, the essence of a computational procedure, step-by-step instructions.
– Program: Implementation of an algorithm in some programming language.

14
AIUB::CSC2105::Algorithm
Kinds of Problem to be solved

• Sorting and Searching are the basic and most common computational
problem.
• Clever algorithms are employed for the Internet
– to manage large volume of data transfer.
– Finding good routes on which the data will travel.
– Search engine to quickly find requested pages.
– Etc…

• Numerical algorithms and number theory are employed in electronic

commerce to keep and secure information such as credit card numbers,
passwords, and bank statements.

15
AIUB::CSC2105::Algorithm
How to Develop an Algorithm?

• Precisely define the problem.

– Precisely specify the input and output.
– Consider all cases.
• Come up with a simple plan to solve the problem at hand.
– The plan is language independent.
– The precise problem specification influences the plan.
• Turn the plan into an implementation
– The problem representation (data structure) influences the implementation.

16
AIUB::CSC2105::Algorithm
??????

Suppose computers were infinitely fast

and computer memory was free. Would
you have any reason to study
algorithms?

17
AIUB::CSC2105::Algorithm
Algorithm
Analysis

18
AIUB::CSC2105::Algorithm
Analysis of Algorithms

• Efficiency:
– Running time
– Space used
• Efficiency as a function of the input size:
– Number of data elements (numbers, points).
– The number of bits of an input number .

19
AIUB::CSC2105::Algorithm
The RAM Model

• RAM model represents a “generic” implementation of the

algorithm
• Each “simple” operation (+, -, =, if, call) takes exactly 1 step.
• Loops and subroutine calls are not simple operations, but
depend upon the size of the data and the contents of a
subroutine. We do not want “sort” to be a single step
operation.
• Each memory access takes exactly 1 step.

20
AIUB::CSC2105::Algorithm
The RAM model (cont.)

• It is important to choose the level of detail.

• The RAM model:
– Instructions (each taking constant time), we usually choose one type of
instruction as a characteristic operation that is counted:
• Arithmetic (add, subtract, multiply, etc.)
• Data movement (assign)
• Control flow (branch, subroutine call, return)
• Comparison (logical ops)
– Data types – integers, characters, and floats

21
AIUB::CSC2105::Algorithm
Example

Algorithm arrayMax(A, n)
#
operations
currentMax  A[0] 2
for (i =1; i<n; i++) 2n
(i=1 once, i<n n times, i++ (n-1) times)
if A[i]  currentMax then 2(n  1)
currentMax  A[i] 2(n  1)
return currentMax 1
Total 6n
1

22
AIUB::CSC2105::Algorithm
Insertion Sort

23
AIUB::CSC2105::Algorithm
 Loop invariants and the correctness of insertion sort
• Loop Invariant: is a property of a program loop that is true before (and after)
each iteration.
• Initialization: It is true prior to the first iteration of the loop.

• Maintenance: If it is true before an iteration of the loop, it remains true before

the next iteration.

• Termination: When the loop terminates, the invariant gives us a useful

property that helps show that the algorithm is correct.

24
AIUB::CSC2105::Algorithm
Analysis of Insertion Sort
• Time to compute the running time as a function of the input size (exact
analysis).

cost times
for j := 2 to n do c1 n
key := A[j] c2 n-1
// Insert A[j] into A[1..j-1] 0 n-1
i := j-1 c3 n-1

n
while i>0 and A[i]>key do j 2
tj
c4

n
A[i+1]:=A[i] j 2
(t j  1)
c5

n
i-- j 2
(t j  1)
A[i+1]:=key c6 n-1
c7

25
Analysis of Insertion Sort

• The running time of an algorithm is the sum of the

running times of each statement.
• A statement with cost c that is executed n times
contributes c*n to the running time.

n
 
n n
j 2
tj (t j  1) j 2
(t j  1)
j 2
• The total running time T(n) of insertion sort is
T(n)= c1*n + c2(n-1) + c3(n-1) + c4 + c5 +
c6

+ c7 (n-1)
26
…Analysis of Insertion Sort

• Often the performance depends on the details

of the input (not only the length n).
• This is modeled by tj.

• In the case of insertion sort the time tj

depends on the original sorting of the input
array.
27
AIUB::CSC2105::Algorithm
Performance Analysis

• Performance often draws the line between what is feasible and what is impossible.
• Often it is sufficient to count the number of iterations of the core (innermost) part.
– No distinction between comparisons, assignments, etc (that means roughly the same cost for all of them).
– Gives precise enough results.

• In some cases the cost of selected operations dominates all other costs.
– Disk I/O versus RAM operations.
– Database systems.

28
AIUB::CSC2105::Algorithm
 Best/ Worst/ Average Case
• Best case: works fast on some input.
• Worst case: (usually) maximum time of algorithm on any input of size.
• Average case: (sometimes) expected time of algorithm over all inputs of size. Need
assumption of statistical distribution of inputs.

• Analyzing insertion sort’s

– Best case: running time » (n-1), i.e., linear time.
– Worst case: running time » (n2-n)/2 , i.e., quadratic time.
– Average case: running time » (n2+n-2)/4 , i.e., quadratic time.

AIUB::CSC2105::Algorithm Introduction29
…Best/ Worst/ Average Case

– For inputs of all sizes:

worst-case
6n average-case
Running time

5n

4n best-case
3n

2n

1n

1 2 3 4 5 6 7 8 9 10 11 12 …..

Input instance size

30
AIUB::CSC2105::Algorithm
Best/ Worst/ Average Case

• Worst case is usually used:

– It is an upper-bound.
– In certain application domains (e.g., air traffic control, surgery) knowing the worst-
case time complexity is of crucial importance.
– For some algorithms worst case occurs fairly often

31
AIUB::CSC2105::Algorithm
Asymptotic Notation
• Asymptotic Notations are languages that allow us to analyze an algorithm's
running time by identifying its behavior as the input size for the algorithm
increases.
• This is also known as an algorithm's growth rate.(Wiki)

32
AIUB::CSC2105::Algorithm
Asymptotic Notation

• The “big-Oh” O-Notation

– asymptotic upper bound
– f(n) = O(g(n)), if there exists constants c>0 and n0>0, s.t. f(n) £ c g(n) for n ³ n0
– f(n) and g(n) are functions
over non- negative integers c g ( n )
f (n )

Running Time
• Used for worst-case analysis

n0 Input Size

We write f(n) = O(g(n)), If there are positive constants n0 and c such that, to the
right of n0 the f(n) always lies on or below c*g(n).

33
AIUB::CSC2105::Algorithm
• Example 1: f(n)=2n+3 , g(n)=n
• Now, we have to find Is f(n)=O(g(n))?
• To check f(n)=O(g(n)), it must satisfy the given
condition:
• f(n)<=c.g(n)
• 2n+3 <= c.n
• Let's assume c=5, n=1 then
• 2*1+3<=5*1
• 5<=5

34
Asymptotic Notation
• The “big-Omega” W-Notation
– asymptotic lower bound

– f(n) = W(g(n)) if there exists constants c>0 and n0>0, s.t. f(n) ≥ c g(n) for n ³ n0

• Used to describe best-case running times or lower bounds of algorithmic problems.

E.g., lower-bound of searching in an unsorted array is W(n).

Running Time
f (n )
c g ( n )

n0 Input Size

We write f(n) = Ω(g(n)), If there are positive constants n0 and c such that, to the right
of n0 the f(n) always lies on or above c*g(n).
Ω(g(n)) = { f(n) : There exist positive constant c and n0 such that 0 ≤ c g(n) ≤ f(n), for all
n ≥ n0}

35
...Asymptotic Notation
• The “big-Theta” Q-Notation
– asymptoticly tight bound

– f(n) = Q(g(n)) if there exists constants c1>0, c2>0, and n0>0, s.t. for n ³ n0 c1 g(n) £ f(n) £ c2 g(n)

• f(n) = Q(g(n)) if and only if f(n) = O(g(n)), f(n) = W(g(n))

c 2 g (n )
• O(f(n)) is often abused instead of Q(f(n))

Running Time
f (n )
c1 g (n )

n0 Input Size

We write f(n) = Θ(g(n)), If there are positive constants n0 and c1 and c2 such that, to
the right of n0 the f(n) always lies between c1*g(n) and c2*g(n) inclusive. Θ(g(n)) = {f(n) :
There exist positive constant c1, c2 and n0 such that 0 ≤ c1 g(n) ≤ f(n) ≤ c2 g(n), for all n
≥ n0}

36
AIUB::CSC2105::Algorithm
Asymptotic Notation

37
AIUB::CSC2105::Algorithm
Time complexity
• Int n0, n1, c;
n0 = 10;
n1 = 10;
C = n0+n1;
O(1)

Int i, n, r;
r=0;
for(i=1; i<=n;i++){
r=r+1;
}
1- 2 (+ and = )
2- 4
• O(2n)-> O(n)
38
Time complexity(Cont.)

int i, j, n;
c=0;
for(i=0;i<n;i++){
for(j=0; j<n;j++){
c =c+1;
}
}
1-1
2-4

O(n2)

39
Space Complexity

• Int n0, n1, c;

n0 = 10;
n1 = 10;
C = n0+n1;
O(1)

40
 Time complexity familiar tasks

Task Growth rate

Matrix/vector multiply O(N2)
Getting a specific element from a list O(1)
Dividing a list in half, dividing one halve in half, etc O(log2N)
Binary Search O(log2N)
Scanning (brute force search) a list
O(N)
Nested for loops (k levels)
O(Nk)
MergeSort
O(N log2N)
BubbleSort
O(N2)
Generate all subsets of a set of data
O(2N)
Generate all permutations of a set of data
O(N!)

41
AIUB::CSC2105::Algorithm
Correctness of Algorithms

• An algorithm is correct if for any legal input it

terminates and produces the desired output.
• Automatic proof of correctness is not possible (so
far).
• There are practical techniques and rigorous
formalisms that help to reason about the
correctness of (parts of) algorithms.

42
AIUB::CSC2105::Algorithm
Partial and Total Correctness

– Partial correctness
IF this point is reached, THEN this is the desired output

Any legal input Algorithm Output

 Total correctness
INDEED this point is reached, AND this is the desired output

Any legal input Algorithm Output

43
AIUB::CSC2105::Algorithm
Assertions

• To prove partial correctness we associate a number of assertions (statements about the

state of the execution) with specific checkpoints in the algorithm.

– E.g., A[1], …, A[ j ] form an increasing sequence

• Preconditions – assertions that must be valid before the execution of an algorithm or a

subroutine (INPUT).

• Postconditions – assertions that must be valid after the execution of an algorithm or a

subroutine (OUTPUT).

44
AIUB::CSC2105::Algorithm
Pre/post-conditions

• Example:
– Write a pseudocode algorithm to find the two smallest numbers in a sequence
of numbers (given as an array).

• INPUT: an array of integers A[1..n], n > 0

• OUTPUT: (m1, m2) such that
– m1<m2 and for each iÎ[1..n]: m1 £ A[i] and, if A[i] ¹ m1, then m2 £ A[i].

– m2 = m1 = A[1] if "j,iÎ[1..n]: A[i]=A[j]

45
AIUB::CSC2105::Algorithm