Introduction

Slide 1
“Everybody in the country
should learn how to program a
computer... Because it teaches
you how to think.”
-Steve Jobs

2
Why do we study Theory of Computation ?

Slide 3
What is Computation ?
•Sequence of mathematical operations ?
– What are, and are not, mathematical operations?

•Sequence of well-defined operations

– How many operations ?
• The fewer, the better.

– Which operations ?
• The simpler, the better.

8/5/2019 Chapter 1 Introduction Slide 4

Computation

• Computation is a general term for any type of

information processing that can be represented as an
algorithm precisely (mathematically).

Slide 5
Computation

• Computation is a general term for any type of

information processing that can be represented as an
algorithm precisely (mathematically).
Examples:
• Adding two numbers in our brains, on a piece of
paper or using a calculator.

Slide 6
Computation

• Computation is a general term for any type of

information processing that can be represented as an
algorithm precisely. (mathematically)
Examples:
• Adding two numbers in our brains, on a piece of
paper or using a calculator.
• Converting a decimal number to its binary
presentation or vise versa.

Slide 7
Computation

• Computation is a general term for any type of

information processing that can be represented as an
algorithm precisely (mathematically).
Examples:
• Adding two numbers in our brains, on a piece of
paper or using a calculator.
• Converting a decimal number to its binary
presentation or vise versa.
• Finding the greatest common divisors of two
numbers.
• …

Slide 8
Theory of Computation

• A very fundamental and traditional branch of Theory of

Computation seeks:

Slide 9
Theory of Computation

• A very fundamental and traditional branch of Theory of

Computation seeks:
1. A more tangible definition for the intuitive notion of algorithm
which results in a more concrete definition for computation.

Slide 10
Theory of Computation

• A very fundamental and traditional branch of Theory of

Computation seeks:
1. A more tangible definition for the intuitive notion of algorithm
which results in a more concrete definition for computation.
2. Finding the boundaries (limitations) of computation.

Slide 11
Algorithm
• A finite sequence of simple instructions that is guaranteed to
halt in a finite amount of time.

Slide 12
Algorithm
• A finite sequence of simple instructions that is guaranteed to
halt in a finite amount of time.
• This is a very abstract definition, since:
– We didn’t specify the nature of this simple instructions.
• For example an instruction can be “increment a number
by one” or “Calculate the triple integral”

Slide 13
Algorithm
• A finite sequence of simple instructions that is guaranteed to
halt in a finite amount of time.
• This is a very abstract definition, since:
– We didn’t specify the nature of this simple instructions.
• For example an instruction can be “increment a number
by one” or “Calculate the triple integral”
– We didn’t specify the entity which can execute these instructions.

Slide 14
Algorithm
• A finite sequence of simple instructions that is guaranteed to
halt in a finite amount of time.
• This is a very abstract definition, since:
– We didn’t specify the nature of this simple instructions.
• For example an instruction can be “increment a number
by one” or “Calculate the triple integral”
– We didn’t specify the entity which can execute these
instructions.
• For example is this entity a person, a computer, …
• If it is a computer what is the processor type? How much
memory does it have? …. ?

Slide 15
What do we study in Theory of Computation ?
• What is computable, and • What a computer can and
what is not ? cannot do
• Are you trying to write a
non-existing program?

• Basis of
– Algorithm analysis – Can you make your program
more efficient?
– Complexity theory

8/5/2019 Chapter 1 Introduction Slide 16

What do we study in Complexity Theory ?
• What is easy, and what is • What is easy, and what is
difficult, to compute ? hard for computers to do?
• Is your cryptographic
scheme safe?

8/5/2019 Chapter 1 Introduction Slide 17

Applications in Computer Science
•Analysis of algorithms •Compilers
•Complexity Theory •Circuit design
•Cryptography

8/5/2019 Chapter 1 Introduction Slide 18

History of Theory of Computation

• 1936 Alan Turing invented the Turing machine, and

proved that there exists an unsolvable problem.

• 1940’s Stored-program computers were built.

• 1943 McCulloch and Pitts invented finite automata.

• 1956 Kleene invented regular expressions and proved

the equivalence of regular expression and finite
automata.

8/5/2019 Chapter 1 Introduction Slide 19

History of Theory of Computation

• 1956 Chomsky defined Chomsky hierarchy, which

organized languages recognized by different
automata into hierarchical classes.

• 1959 Rabin and Scott introduced nondeterministic

finite automata and proved its equivalence to
(deterministic) finite automata.

• 1950’s-1960’s More works on languages, grammars,

and compilers

8/5/2019 Chapter 1 Introduction Slide 20

History of Theory of Computation

• 1965 Hartmantis and Stearns defined time

complexity, and Lewis, Hartmantis and Stearns
defined space complexity.

• 1971 Cook showed the first NP-complete problem,

the satisfiability problem.

• 1972 Karp Showed many other NP-complete

problems.

8/5/2019 Chapter 1 Introduction Slide 21

History of Theory of Computation

• 1976 Diffie and Helllman defined Modern

Cryptography based on NP-complete problems.

• 1978 Rivest, Shamir and Adelman proposed a public-

key encryption scheme, RSA.

8/5/2019 Chapter 1 Introduction Slide 22

Alphabet and Strings

• An alphabet is a finite, non-empty set of symbols.

– {0,1 } is a binary alphabet.
– { A, B, …, Z, a, b, …, z } is an English alphabet.

• A string over an alphabet  is a sequence of any

number of symbols from .
– 0, 1, 11, 00, and 01101 are strings over {0, 1 }.
– Cat, CAT, and compute are strings over the English
alphabet.

8/5/2019 Chapter 1 Introduction Slide 23

Empty String

• An empty string, denoted by , is a string

containing no symbol.
– is a string over any alphabet.

8/5/2019 Chapter 1 Introduction Slide 24

Length

• Definition − It is the number of symbols present in a string.

(Denoted by |S|).
• Examples −
• If S = ‘cabcad’, |S|= 6

• If |S|= 0, it is called an empty string (Denoted by λ or ε)

Slide 25
Length

• The length of a string x, denoted by length(x),

is the number of positions of symbols in the
string.
Let Σ = {a, b, …, z}
length(automata) = 8
length(computation) = 11
length(ε) = 0

• x(i), denotes the symbol in the ith position of a

string x, for 1 i  length(x).

8/5/2019 Chapter 1 Introduction Slide 26

String Operations

•Concatenation
•Substring
•Reversal

8/5/2019 Chapter 1 Introduction Slide 27

Concatenation

• The concatenation of strings x and y,

denoted by xy or x y, is a string z such that:
– z(i) = x(i) for 1  i  length(x)
– z(i) = y(i) for length(x)<ilength(x)+length(y)
• Example
– automatacomputation = automatacomputation

8/5/2019 Chapter 1 Introduction Slide 28

Concatenation

The concatenation of string x for n times,

where n0, is denoted by xn
– x0 = 
– x1 = x
– x2 = x x
– x3 = x x x
– …

8/5/2019 Chapter 1 Introduction Slide 29

Substring

Let x and y be strings over an alphabet Σ

The string x is a substring of y if there exist strings
w and z over Σ such that y = w x z.
– ε is a substring of every string.
– For every string x, x is a substring of x itself.
Example
– ε, comput and computation are substrings of
computation.

8/5/2019 Chapter 1 Introduction Slide 30

Reversal

Let x be a string over an alphabet Σ

The reversal of the string x, denoted by x r, is a
string such that
– if x is ε, then xr is ε.
– If a is in Σ, y is in Σ* and x = a y, then xr = yr a.

8/5/2019 Chapter 1 Introduction Slide 31

Example of Reversal
(automata)r
= (utomata)r a
= (tomata)r ua
= (omata)r tua
= (mata)r otua
= (ata)r motua
= (ta)r amotua
= (a)r tamotua
= ()r atamotua
= atamotua
8/5/2019 Chapter 1 Introduction Slide 32
Σ*

• The set of strings created from any number (0

or 1 or …) of symbols in an alphabet  is
denoted by *.
• That is, * = i=0 i

– Let = {0, 1}.

– * = {, 0, 1, 00, 01, 10, 11, 000, 001, 010, 011, … }.

8/5/2019 Chapter 1 Introduction Slide 33

Σ+

• The set of strings created from at least one symbol

(1 or 2 or …) in an alphabet  is denoted by +.
• That is, + = i=1 i
= i=0.. i - 0

=  i=0.. i - {}
• Let = {0, 1}. + = {0, 1, 00, 01, 10, 11, 000, 001, 010, 011,
… }.
* and + are infinite sets.
8/5/2019 Chapter 1 Introduction Slide 34
Languages

• A language over an alphabet Σ is a set of

strings over Σ.
– Let Σ = {0, 1} be the alphabet.
– Le = {Σ* | the number of 1’s in  is even}.
– , 0, 00, 11, 000, 110, 101, 011, 0000, 1100, 1010,
1001, 0110, 0101, 0011, … are in Le

8/5/2019 Chapter 1 Introduction Slide 35

Operations on Languages

• Complementation
• Union
• Intersection
• Concatenation
• Reversal
• Closure

8/5/2019 Chapter 1 Introduction Slide 36

Complementation

Let L be a language over an alphabet Σ.

The complementation of L, denoted byL, is Σ*–
L.
Example:
Let Σ = {0, 1} be the alphabet.
Le = {Σ* | the number of 1’s in  is even}.
Le= {Σ* | the number of 1’s in  is not even}.
Le= {Σ* | the number of 1’s in  is odd}.
8/5/2019 Chapter 1 Introduction Slide 37
Union

Let L1 and L2 be languages over an alphabet Σ.

The union of L1 and L2, denoted by L1L2, is
{x | x is in L1 or L2}.
Example:
{x{0,1}*|x begins with 1}  {x{0,1}*|x ends with 0}
1Σ∗0

8/5/2019 Chapter 1 Introduction Slide 38

Intersection

Let L1 and L2 be languages over an alphabet Σ.

The intersection of L1 and L2, denoted by

L1L2, is { x | x is in L1 and L2}.

Example:
{ x{0,1}*| x begins with 0}  { x{0,1}*| x ends with
0}
= { x{0,1}*| x begins and ends with 0}
8/5/2019 Chapter 1 Introduction Slide 39
Concatenation
Let L1 and L2 be languages over an alphabet Σ.
The concatenation of L1 and L2, denoted by L1L2, is
{w1w2| w1 is in L1 and w2 is in L2}.
Example

{ x  {0,1}*| x begins with 0}{x  {0,1}*| x ends with 0}

= { x  {0,1}*| x begins and ends with 0 and length(x)  2}

{ x  {0,1}*| x ends with 0}{x  {0,1}*| x begins with 0}

= { x  {0,1}*| x has 00 as a substring}

8/5/2019 Chapter 1 Introduction
Reversal

Let L be a language over an alphabet Σ.

The reversal of L, denoted by Lr, is {wr| w is in L}.
Example
{x  {0,1}*| x begins with 0} r
= {x  {0,1}*| x ends with 0}
{x  {0,1}*| x has 00 as a substring} r
= {x  {0,1}*| x has 00 as a substring}

8/5/2019 Chapter 1 Introduction Slide 41

Kleene’s closure/star

• The Kleene star, ∑*, is a unary operator on a set

of symbols or strings, ∑, that gives the infinite set
of all possible strings of all possible lengths
over ∑ including λ.
• ∑* = ∑0 ∪ ∑1 ∪ ∑2 ∪……. where ∑p is the set of all
possible strings of length p.
• If ∑ = {a, b}, ∑* = {λ, a, b, aa, ab, ba,
bb,………..}

8/5/2019 Chapter 1 Introduction Slide 42

Kleene Closure / Plus

• The set ∑+ is the infinite set of all possible strings

of all possible lengths over ∑ excluding λ.
• ∑+ = ∑1 ∪ ∑2 ∪ ∑3 ∪…….
• ∑+ = ∑* − { λ }
• If ∑ = { a, b } , ∑+ = { a, b, aa, ab, ba,
bb,………..}

Slide 43
 GRAMMARS

• According to chomsky hierarchy, grammars are divided of 4 types:

• Type 0 known as unrestricted grammar.
• Type 1 known as context sensitive grammar.
• Type 2 known as context free grammar.
• Type 3 Regular Grammar.

• T or ∑ = set of alphabet/Character/terminals = ( a, b,c,…., +,-,*, …. )

• V = Set of non-terminals or variables =( A,B,C, …., )
• ∑UV = combination of variables and terminals

Slide 44
Type 0 ( Unrestricted Grammar )

• Type-0 grammars include all formal grammars. Type 0 grammar language are
recognized by turing machine.
• These languages are also known as the recursively enumerable languages.
• Grammar Production in the form of

Slide 45
Type 0 ( Unrestricted Grammar )

• Grammar: Unrestricted
• Language: Recursively Enumerable, Recursive
• Automaton: Turing machine
• Definition:(∑UV)*→(∑UV)* With at least one var in LHS
• Example:
• 1. S → Aa
• 2. sBb → abc
• 3. a → Acb (not type-0 as there is no variable on LHS)
• 4. aB → BabcS
Slide 46
 Type 1 (Context Sensitive )

• Type-1 grammars generate the context-sensitive languages. The language

generated by the grammar are recognized by the Linear Bound Automata.

Slide 47
Type 1 (Context Sensitive )

• Grammar: context-sensitive
• Language: context-sensitive
• Automaton: linear-bounded automaton
• Definition: (∑UV)*→ (∑UV)* Size of LHS ≤ RHS
• i.e. if x→y then│x│≤ │y│ and αAβ → αBβ
• where-
• α = left context,
• β = right context,
• A = a Variable,
• B = anything (∑UV)* Slide 48
 Type 1 (Context Sensitive )
Example:
1. abAbcd → abABbcd
Here ab is the left context, bcd is the right
context, B = AB
2. AC → A
Here A is the left context, Λ is the right context.
B = Λ. This production simply erases when the left
context is A and the right context is Λ
3. C→ Λ
Here the left and right contexts are ᴧ. And B = Λ.
This production simply erases C in any context.

Slide 49
 Type 2 ( Context Free )

• Type-2 grammars generate the context-free languages. The language generated

by the grammar is recognized by a Non Deterministic Push down Automata. Type-
2 grammars generate the context-free languages.

• In Type 2,

Slide 50
 Type 2 ( Context Free )
Grammar: context-free
Language: context-free
Automaton: pushdown automaton
Definition: A → (∑UV)*
Example:
1. S → Aa
2. Sb → Aa (not type-2 as left hand side other than
a varible one terminal is also there)
3. B → bAa
 Type 3 (Regular Grammar)

• Type-3 grammars generate the regular languages.These languages are exactly

all languages that can be decided by a finite state automaton.
• Type 3 is most restricted form of grammar.
• Type 3 should be in the given form only :

Slide 52
 Type 3 (Regular Grammar)
Grammar: Regular
Language: Regular
Automaton: finite-state automaton
Definition:
A → a , A → aB --> Right linear Grammar
A → a , A → Ba --> Left linear grammar
Example:
1. A → aB/Ba/a
2. A → BAa (not type-3 as right hand side has more
than one i.e 2 variables which is not allowed in regular grammar )
3. Ab → aB (not type-3 as LHS has symbols other than a variable)

Slide 53
 Null Production

Null Production A → ε is not allowed in Type 1, Type 2 and

Type 3. But in a special case, if the null string is a part of the
language i.e., ε ϵ L, then we allow null production in form of S→ε,
but with the condition that it will not be further used for substitution
purpose.

Slide 54
Slide 55
Regular expression

•A regular expression is a formula for

representing a (complex) language in terms of
“elementary” languages combined using the
three operations
•Union
•concatenation and Kleene closure.

Slide 56
 Regular definition for recognizing identifiers in C

letter -> a|b|...z|A|B|...|Z|_

digit -> 0|1|...|9
identifier -> letter(letter|digit)*

This definition will generate identifiers of the form

identifier: [a-zA-Z][ _a-zA-Z0-9]*

Slide 57
Regular expression

• Decimal = (sign)?(digit)+
• Identifier = (letter)(letter | digit)*

Slide 58
Regular expressions generating the languages
{w| w contains at least three 1s}
Σ∗1Σ∗1Σ∗1Σ∗
{w| w contains the substring 0101, i.e., w = x0101y for some x and
y}
Σ∗0101Σ∗

 {w| w contains at least three 1s}

Σ ∗1Σ ∗1Σ ∗1Σ ∗

{w| w has length at least 3 and its third symbol is a 0}

Σ Σ 0Σ ∗ Slide 59
Regular expressions generating the languages

• Give regular expressions generating the following languages:

1.{w : w contains exactly two 0s}
1*01*01*
2. {w : w contains at least two 0s}
∑*0∑*0∑*, here note that ∑ =(0 ∪ 1)
3. {w | w begins with a 1 and ends with a 0}
1∑*0

Slide 60
Regular expressions generating the languages

• {w | the length of w is at most 5}

• (∑U↋)(∑U↋)(∑U↋)(∑U↋)(∑U↋)
• f) {w | w contains an even number of 0s, or contains exactly
two 1s}
• Even number of 0s: (1*01*01*)*
• Contains Exactly two 1s: 0*10*10*
• Even number of 0s, or contains exactly two 1s:
(1*01*01*)* U 0*10*10*

Slide 61
 Regular expressions generating the languages
• Write regular expressions for the following languages over the
alphabet Σ = {a, b}:
• All strings that do not end with aa.
ε + a + b + (a + b)* (ab + ba + bb)
• All strings that contain an even number of b’s.
a*(ba*ba*)*
• All strings which do not contain the substring ba.
a*b*

Slide 62
Languages and Problems

• Problem
– Example: What are prime numbers > 20?
• Decision problem
– Problem with a YES/NO answer
– Example: Given a positive integer n, is n a prime number >
20?
• Language
– Example: {n | n is a prime number > 20}
= {23, 29, 31, 37, …}
8/5/2019 Chapter 1 Introduction Slide 63
Language Recognition and Problem
• A problem is represented by a set of strings of the
input whose answer for the corresponding problem is
“YES”.
• a string is in a language = the answer of the
corresponding problem for the string is “YES”
– Let “Given a positive integer n, is n a prime number > 20?”
be the problem P.
– If a string represents an integer i in {m | m is a prime
number > 20} , then the answer for the problem P for n = i is
true.
8/5/2019 Chapter 1 Introduction Slide 64
Describing Problems
• Problems described by functions
 functions assign ouputs to inputs
 described by tables, formulae, circuits, math logic, algorithms
Example: TRAVELLING SALESPERSON Problem
Input: n(n-1)/2 distances between n cities
Output: order to visit cities that gives shortest tour (or length of tour)
• Problems described by languages
 ``yes/no’’ problems or decision problems
 a language represents a problem or question
 input strings in the language: answer is ``yes’’; else ``no”
Example: HALTING Problem
Input: a string w and a Turing Machine M
Output: ``yes” if w  L(M ) ;``no” otherwise
 equivalent to asking if string (e, w)  LH
where LH  {(e, w) | Turing Machine M e halts on input w}