‫ر‬

‫الجامعة السعودية االلكتونية‬

‫ر‬
‫االلكتونية‬ ‫الجامعة السعودية‬

‫‪26/12/2021‬‬
Theory of Computing

Headline separator Module 11

Intractable Problems
Contents
1. Post’s Correspondence Problem
2. Other Undecidable Problems
3. The Classes P and NP
4. An NP-Complete Problem
Weekly Learning Outcomes
1. Explain class P and NP.
2. Explain NP-Complete Problems.
Required Reading
1. Post’s Correspondence Problem
2. Other Undecidable Problems
3. The Classes P and NP
4. An NP-Complete Problem
(Introduction to Automata Theory, Languages, and Computation
(2013) Global Edition 3rd Edition)
Recommended Reading

https://www.divaportal.org/smash/get/diva2:1087096/FULLTEXT03.pdf
http://infolab.stanford.edu/~ullman/focs/ch03.pdf

This Presentation is mainly dependent on the textbook: Introduction to Automata Theory, Languages, and Computation: Global Edition, 3rd edition (2013) PHI
by John E. Hopcroft, Rajeev Motwani and Jeffrey D. Ullman
• Post’s Correspondence Problem
Post’s Correspondence Problem
• Post’s Correspondence Problem (PCP) is an example of a
problem that does not mention TM’s in its statement,
yet is undecidable.
• From PCP, we can prove many other non-TM problems
undecidable.
▪ PCP Instances
• An instance of PCP is a list of pairs of nonempty strings
over some alphabet Σ.
• Say (w1, x1), (w2, x2), …, (wn, xn).
• The answer to this instance of PCP is “yes” if and only if
there exists a nonempty sequence of indices i1,…,ik, such
that wi1…win = xi1…xin.

7
Example (1)
• Let the alphabet be {0, 1}.
• Let the PCP instance consist of the two pairs (0, 01) and
(100, 001).
• We claim there is no solution.
• You can’t start with (100, 001), because the first
characters don’t match.

0 100 100 But we can never make

01 001 001 the first string as long
as the second.

Must start Can add the As many

with first second pair times as
pair for a match we like

8
Example (2)
• Suppose we add a third pair, so the instance becomes: 1 = (0, 01);
2 = (100, 001); 3 = (110, 10).
• Now 1,3 is a solution; both strings are 0110.
• In fact, any sequence of indexes in 12*3 is a solution.

9
Proving PCP is Undecidable

• We’ll introduce the Modified PCP (MPCP) problem.

• Same as PCP, but the solution must start with the first pair in the list.
• We reduce Lu to MPCP.
• But first, we’ll reduce MPCP to PCP.

10
Example: MPCP
• The list of pairs (0, 01), (100, 001), (110, 10), as an instance of
MPCP, has a solution as we saw.
• However, if we reorder the pairs, say (110, 10), (0, 01), (100,
001) there is no solution.
• No string 110… can ever equal a string 10… .

11
Representing PCP or MPCP Instances
• Since the alphabet can be arbitrarily large, we need to code
symbols.
• Say the i-th symbol will be coded by “a” followed by i in binary.
• Commas and parentheses can represent themselves.
• Thus, we have a finite alphabet in which all instances of PCP or
MPCP can be represented.
• Let LPCP and LMPCP be the languages of coded instances of PCP or
MPCP, respectively, that have a solution.

12
Reducing LMPCP to LPCP (1)
• Take an instance of LMPCP and do the following,
using new symbols * and $.
1. For the first string of each pair, add * after every
character.
2. For the second string of each pair, add * before every
character.
3. Add pair ($, *$).
4. Make another copy of the first pair, with *’s and an extra
* prepended to the first string.

13
Example: LMPCP to LPCP

MPCP instance, Add

in order: PCP instance: *’s

(110, 10) (1*1*0*, *1*0)

(0, 01) (0*, *0*1)
(100, 001) (1*0*0*, *0*0*1)
($, *$) Ender

(*1*1*0*, *1*0)
Special pair version of
first MPCP choice – only
possible start for a PCP
solution.

14
Reducing LMPCP to LPCP (2)

• If the MPCP instance has a solution string w, then padding

with stars fore and aft, followed by a $ is a solution string
for the PCP instance.
• Use same sequence of indexes, but special pair to start.
• Add ender pair as the last index.
• Conversely, the indexes of a PCP solution give us a MPCP
solution.
1. First index must be special pair – replace by first pair.
2. Remove ender.

15
Reducing Lu to LMPCP (1)
• We use MPCP to simulate the sequence of ID’s that M executes
with input w.
• If q0w⊦I1⊦I2⊦ … is the sequence of ID’s of M with input w, then
any solution to the MPCP instance we can construct will begin
with this sequence of ID’s.
• # separates ID’s and also serves to represent blanks at the end of an
ID.
• But until M reaches an accepting state, the string formed by
concatenating the second components of the chosen pairs will
always be a full ID ahead of the string from the first pair.
• If M accepts, we can even out the difference and solve the
MPCP instance.

16
Reducing Lu to LMPCP (2)
• Key assumption: M has a semi-infinite tape; it never moves left
from its initial head position.
• Alphabet of MPCP instance: state and tape symbols of M
(assumed disjoint) plus special symbol # (assumed not a state or
tape symbol).
• First MPCP pair: (#, #q0w#).
• We start out with the second string having the initial ID and a full ID
ahead of the first.
• (#, #).
• We can add ID-enders to both strings.
• (X, X) for all tape symbols X of M.
• We can copy a tape symbol from one ID to the next.

17
Example: Copying Symbols (1)
• Suppose we have chosen MPCP pairs to simulate some
number of steps of M, and the partial strings from these
pairs look like:

. . . #AB
. . . #ABqCD#AB

18
Reducing Lu to LMPCP (3)

• For every state q of M and tape symbol X, there are pairs:

1. (qX, Yp) if δ(q, X) = (p, Y, R).
2. (ZqX, pZY) if δ(q, X) = (p, Y, L) [any Z].
• Also, if X is the blank, # can substitute.
1. (q#, Yp#) if δ(q, B) = (p, Y, R).
2. (Zq#, pZY#) if δ(q, X) = (p, Y, L) [any Z].

19
Example: Copying Symbols (2)

• Continuing the previous example, if δ(q, C) = (p, E, R), then:

. . . #ABqCD#
. . . #ABqCD#ABEpD#
• If M moves left, we should not have copied B if we wanted a
solution.

20
Reducing Lu to LMPCP (4)
• If M reaches an accepting state f, then f “eats” the
neighboring tape symbols, one or two at a time, to
enable M to reach an “ID” that is essentially empty.
• The MPCP instance has pairs (XfY, f), (fY, f), and (Xf, f) for
all tape symbols X and Y.
• To even up the strings and solve: (f##, #).

21
Example: Cleaning Up After Acceptance

… #ABfCDE#AfDE#fE#f##
… #ABfCDE#AfDE#fE#f##

22
• Other Undecidable Problems
Undecidability of “= Σ*”

• We have reduced PCP to the problem: is a given CFL equal to all

strings over its terminal alphabet?

24
Undecidability of “CFL is Regular”
• Also undecidable: is a CFL a regular language?
• Same reduction from PCP.
• Proof: One direction: If LAc  LBc = Σ*, then it surely is regular.
• Conversely, we can show that if L = LAc  LBc is not Σ*, then it
can’t be regular.
• Proof: Suppose wx is a solution to PCP, where x is the indices.
• Define homomorphism h(0) = w and h(1) = x.
• h(0n1n) is not in L, because the repetition of any solution is also a
solution.
• However, h(y) is in L for any other y in {0,1}*.
• If L were regular, so would be h–1(L), and so would be its
complement = {0n1n |n > 1}.

25
• The Classes P and NP
Time-Bounded TM’s

• A Turing machine that, given an input of length n, always

halts within T(n) moves is said to be T(n)-time bounded.
• The TM can be multitape.
• Sometimes, it can be nondeterministic.
• The deterministic, multitape case corresponds roughly to
“an O(T(n)) running-time algorithm.”

27
The class P

• If a DTM M is T(n)-time bounded for some polynomial T(n), then

we say M is polynomial-time (“polytime ”) bounded.
• And L(M) is said to be in the class P.
• Important point: when we talk of P, it doesn’t matter whether we
mean “by a computer” or “by a TM” (next slide).

28
Polynomial Equivalence of Computers and
TM’s

• A multitape TM can simulate a computer that runs for time

O(T(n)) in at most O(T2(n)) of its own steps.
• If T(n) is a polynomial, so is T2(n).

29
Examples of Problems in P

• Is w in L (G), for a given CFG G?

• Input = w.
• Use CYK algorithm, which is O(n3).
• Is there a path from node x to node y in graph G?
• Input = x, y, and G.
• Use Dijkstra’s algorithm, which is O(n log n) on a graph of n nodes
and arcs.

30
Running Times Between Polynomials

• You might worry that something like O(nlogn) is not a

polynomial.
• However, to be in P, a problem only needs an algorithm that
runs in time less than some polynomial.
• Surely O(n log n) is less than the polynomial O(n2).

31
Knapsack (1)
• The Knapsack Problem: Given positive integers i1, i2 ,…, in, can
we divide them into two sets with equal sums?
• We can solve this problem in polytime by a dynamic
programming algorithm:
• Maintain a table of all the differences we can achieve by partitioning
the first j integers.

32
Knapsack (2)

• Basis: j = 0. Initially, the table has “true” for 0 and “false” for all
other differences.
• Induction: To consider ij, start with a new table, initially all false.
• Then, set k to true if, in the old table, there is a value m that was
true, and k is either m+ij or m-ij.
• Suppose we measure running time in terms of the sum of the
integers, say m.
• Each table needs only space O(m) to represent all the positive and
negative differences we could achieve.
• Each table can be constructed in time O(n)
• Since n < m, we can build the final table in O(m2) time.
• From that table, we can see if 0 is achievable and solve the problem.

33
Measuring Input Size

▪ “Input size” has a specific meaning: the length of the

representation of the problem instance as it is input
to a TM.
▪ For the Knapsack Problem, you cannot always write
the input in a number of characters that is polynomial
in either the number-of or sum-of the integers.

34
Knapsack – Bad Case
▪ Suppose we have n integers, each of which is around 2n.
▪ We can write integers in binary, so the input takes O(n2) space to
write down.
▪ But the tables require space O(n2n).
▪ They therefore require at least that order of time to construct.
▪ Thus, the proposed “polynomial” algorithm actually takes time
O(n22n) on an input of length O(n2).
▪ Or, since we like to use n as the input size, it takes time O(n2sqrt(n)) on
an input of length n.
▪ In fact, it appears no algorithm solves Knapsack in polynomial time.

35
The Class NP
▪ The running time of a nondeterministic TM is the maximum
number of steps taken along any branch.
▪ If that time bound is polynomial, the NTM is said to be
polynomial-time bounded.
▪ And its language/problem is said to be in the class NP.

36
Example: NP
▪ The Knapsack Problem is definitely in NP, even using the
conventional binary representation of integers.
▪ Use nondeterminism to guess one of the subsets.
▪ Sum the two subsets and compare.

37
P Versus NP
▪ One of the most important open problems is the question:
▪ Is P = NP?
▪ There are thousands of problems that are in NP but appear
not to be in P.
▪ But no proof that they aren’t really in P.

38
Complete Problems
▪ One way to address the P = NP question is to identify
complete problems for NP.
▪ An NP-complete problem has the property that if it is in P,
then every problem in NP is also in P.
▪ Defined formally via “polytime reductions.”

39
Complete Problems – Intuition

▪ A complete problem for a class embodies every problem in the

class, even if it does not appear so.
▪ Compare: PCP embodies every TM computation, even though it
does not appear to do so.
▪ Knapsack embodies every polytime NTM computation.

40
NP-Complete Problems
• A problem/language M is said to be NP-complete if for every
language L in NP, there is a polytime reduction from L to M.
• Fundamental property: if M has a polytime algorithm, then L
also has a polytime algorithm.
• I.e., if M is in P, then every L in NP is also in P, or “P = NP.”

41
 The Plan
SAT polytime
All of NP polytime reduces to 3-SAT polytime reduces
reduces to SAT, which 3-SAT to many other problems;
is therefore NP-complete they’re all NP-complete

SAT

3-
SAT
NP

42
Polytime Reductions (1)
▪ Goal: find a way to show problem L to be NP-complete by
reducing every language/problem in NP to L in such a way that
if we had a deterministic polytime algorithm for L, then we
could construct a deterministic polytime algorithm for any
problem in NP.
▪ We need the notion of a polytime transducer – a TM that:
1. Takes an input of length n.
2. Operates deterministically for some polynomial time p(n).
3. Produces an output on a separate output tape.

▪ Note: output length is at most p(n).

43
Polytime Transducer

state

input n

scratch
tapes

output < p(n)

Remember: important requirement

is that time < p(n).

44
Polytime Reductions (2)

▪ Let L and M be languages.

▪ Say L is polytime reducible to M if there is a polytime
transducer T such that for every input w to T, the output
x = T(w) is in M if and only if w is in L.

45
Picture of Polytime Reduction

in M

in L
T
not
in L not in M

46
Proof That Polytime Reductions

▪ Suppose M has an algorithm of polynomial time q(n).

▪ Let L have a polytime transducer T to M, taking polynomial
time p(n).
▪ The output of T, given an input of length n, is at most of
length p(n).
▪ The algorithm for M on the output of T takes time at most
q(p(n)).
▪ We now have a polytime algorithm for L:
▪ Given w of length n, use T to produce x of length < p(n),
taking time < p(n).
▪ Use the algorithm for M to tell if x is in M in time < q(p(n)).
▪ Answer for w is whatever the answer for x is.
▪ Total time < p(n) + q(p(n)) = a polynomial.

47
• An NP-Complete Problem
The Satisfiability Problem (SAT )
▪ Satisfiability: Given a boolean formula, does there exist a
truth assignment to the variables to make the expression
true
• Study of boolean functions generally is concerned with the
set of truth assignments (assignments of 0 or 1 to each of
the variables) that make the function true.
• NP-completeness needs only a simpler question (SAT): does
there exist a truth assignment making the function true?

49
Example: SAT

• (x+y)(-x + -y) is satisfiable.

• There are, in fact, two satisfying truth assignments:
1. x=0; y=1.
2. x=1; y=0.
• x(-x) is not satisfiable.

50
SAT as a Language/Problem
• An instance of SAT is a boolean function.
• Must be coded in a finite alphabet.
• Use special symbols (, ), +, - as themselves.
• Represent the i-th variable by symbol x followed by integer i in
binary.

51
Example: Encoding for SAT

• (x+y)(-x + -y) would be encoded by the string (x1+x10)(-x1+-

x10)

52
SAT is in NP

• There is a multitape NTM that can decide if a Boolean formula of

length n is satisfiable.
• The NTM takes O(n2) time along any path.
• Use nondeterminism to guess a truth assignment on a second
tape.
• Replace all variables by guessed truth values.
• Evaluate the formula for this assignment.
• Accept if true.

53
Cook’s Theorem
• SAT is NP-complete.
• Really a stronger result: formulas may be in conjunctive normal form (CSAT)
– later.
• To prove, we must show how to construct a polytime reduction from
each language L in NP to SAT.
• Start by assuming the most restricted possible form of NTM for L
(next slide).

54
Assumptions About NTM for L

1. One tape only.

2. Head never moves left of the initial position.
3. States and tape symbols are disjoint.
• Key Points: States can be named arbitrarily, and the
constructions many-tapes-to-one and two-way-infinite-
tape-to-one at most square the time.

55
More About the NTM M for L

• Let p(n) be a polynomial time bound for M.

• Let w be an input of length n to M.
• If M accepts w, it does so through a sequence I0⊦I1⊦…⊦Ip(n) of p(n)+1
ID’s.
• Assume trivial move from a final state.
• Each ID is of length at most p(n)+1, counting the state.

56
From ID Sequences to Boolean Functions

• The Boolean function that the transducer for L will construct from w
will have (p(n)+1)2 “variables.”
• Let variable Xij represent the j-th position of the i-th ID in the
accepting sequence for w, if there is one.
• i and j each range from 0 to p(n).

57
Picture of Computation as an Array

Initial ID X00 X01 … X0p(n)

I1 X10 X11 … X1p(n)

. .
. .
. .

Ip(n) Xp(n)0 Xp(n)1 … Xp(n)p(n)

58
Intuition

• From M and w we construct a boolean formula that forces the X’s

to represent one of the possible ID sequences of NTM M with
input w, if it is to be satisfiable.
• It is satisfiable iff some sequence leads to acceptance.

59
From ID’s to Boolean Variables
• The Xij’s are not boolean variables; they are states and tape
symbols of M.
• However, we can represent the value of each Xij by a family
of Boolean variables yijA, for each possible state or tape
symbol A.
• yijA is true if and only if Xij = A.

60
Points to Remember

1. The boolean function has components that depend on n.

• These must be of size polynomial in n.
2. Other pieces depend only on M.
• No matter how many states/symbols m has, these are of
constant size.
3. Any logical formula about a set of variables whose size is
independent of n can be written somehow.

61
Designing the Function

• We want the Boolean function that describes the Xij’s to be

satisfiable if and only if the NTM M accepts w.
• Four conditions:
1. Unique: only one symbol per position.
2. Starts right: initial ID is q0w.
3. Moves right: each ID follows from the next by a move of M.
4. Finishes right: M accepts.

62
Unique

• Take the AND over all i, j, Y, and Z of (-yijY+ -yijZ).

• That is, it is not possible for Xij to be both symbols Y and Z.

63
Starts Right
• The Boolean Function needs to assert that the first ID is the
correct one with w = a1…an as input.
1. X00 = q0.
2. X0i = ai for i = 1,…, n.
3. X0i = B (blank) for i = n+1,…, p(n).
• Formula is the AND of y0iZ for all i, where Z is the symbol in
position i.

64
Finishes Right
• Somewhere, there must be an accepting state.
• Form the OR of Boolean variables yijq, where i and j are arbitrary and
q is an accepting state.
• Note: differs from text.

65
Running Time So Far

• Unique requires O(p2(n)) symbols be written.

• Parentheses, signs, propositional variables.
• Algorithm is easy, so it takes no more time than O(p2(n)).
• Starts Right takes O(p(n)) time.
• Finishes Right takes O(p2(n)) time.

Variation over symbols Y

and Z is independent of n, Variation
so covered by constant. over i and j 66
Running Time (2)
• Caveat: Technically, the propositions that are output of the
transducer must be coded in a fixed alphabet, e.g., x10011 rather
than yijA.
• Thus, the time and output length have an additional factor O(log
n) because there are O(p2(n)) variables.
• But log factors do not affect polynomials

67
 Works because
Moves Right Unique assures
only one yijX true.

• Xij = Xi-1,j whenever the state is none of Xi-1,j-1, Xi-1,j, or Xi-1,j+1.

• For each i and j, construct a formula that says (in propositional
variables) the OR of “Xij = Xi-1,j” and all yi-1,k,A where A is a state
symbol (k = i-1, i, or i+1).
• Note: Xij = Xi-1,j is the OR of yijA.yi-1,jA for all symbols A.

68
Constraining the Next Symbol

…A B C… …A q C…

B ? ? ?

Easy case; Hard case; all

must be B three may depend
on the move of M

69
Moves Right (2)

• In the case where the state is nearby, we need to write an

expression that:
1. Picks one of the possible moves of the NTM M.
2. Enforces the condition that when Xi-1,j is the state, the values of Xi,j-1,
Xi,j, and Xi,j+1. are related to Xi-1,j-1, Xi-1,j, and Xi-1,j+1 in a way that
reflects the move.

70
Example: Moves Right

Suppose δ(q, A) contains (p, B, L).

Then one option for any i, j, and C is:
C q A
p C B
If δ(q, A) contains (p, B, R), then an
option for any i, j, and C is:
C q A
C B p

71
Moves Right (3)

• For each possible move, express the constraints on the six X’s by a
Boolean formula.
• For each i and j, take the OR over all possible moves.
• Take the AND over all i and j.
• Small point: for edges (e.g., state at 0), assume invisible symbols are
blank.

72
Running Time

• We have to generate O(p2(n)) Boolean formulas, but each is

constructed from the moves of the NTM M, which is fixed in size,
independent of the input w.
• Takes time O(p2(n)) and generates an output of that length.
• Times log n, because variables must be coded in a fixed alphabet.

73
Cook’s Theorem – Finale

• In time O(p2(n) log n) the transducer produces a boolean formula,

the AND of the four components: Unique, Starts, Finishes, and
Moves Right.
• If M accepts w, the ID sequence gives us a satisfying truth
assignment.
• If satisfiable, the truth values tell us an accepting computation of M.

74
 Picture So Far

• We have one NP-complete problem: SAT.

• In the future, we shall do polytime reductions of SAT to other problems,
thereby showing them NP-complete.
• Why? If we polytime reduce SAT to X, and X is in P, then so is SAT, and
therefore so is all of NP.

75
Main Reference
1. Post’s Correspondence Problem
2. Other Undecidable Problems
3. The Classes P and NP
4. An NP-Complete Problem
(Introduction to Automata Theory, Languages, and Computation
(2013) Global Edition 3rd Edition)
Additional References

https://www.divaportal.org/smash/get/diva2:1087096/FULLTEXT03.pdf
http://infolab.stanford.edu/~ullman/focs/ch03.pdf

This Presentation is mainly dependent on the textbook: Introduction to Automata Theory, Languages, and Computation: Global Edition, 3rd edition (2013) PHI
by John E. Hopcroft, Rajeev Motwani and Jeffrey D. Ullman
Thank You