Week 4

Bayesian Decision Theory

Dr. Nehad Ramaha,

Computer Engineering Department
Karabük Universities 1
The class notes are a compilation and edition from many sources. The instructor does not claim intellectual property or ownership of the lecture notes.
 Week 3

 Bayesian Decision Theory is a fundamental

statistical approach that quantifies the tradeoffs
between various decisions using probabilities
and costs that accompany such decisions.
 First, we will assume that all probabilities are
known.
 Then, we will study the cases where the
probabilistic structure is not completely known.

2
 Week 3

 Design classifiers to recommend decisions

that minimize some total expected ”risk”.
 The simplest risk is the classification error
(i.e., costs are equal).
 Typically, the risk includes the cost
associated with different decisions.

3
Week 3

4
Week 3

5
Week 3

6
Week 3

7
Week 3

8
 P(B A)P( A)
P( A B ) =
P(B)

P ( B A) P ( A)
P ( A B) =
å P(B A )P( A )
n n n

9
10
11
using feature x
make it more
accurate

12
13
14
15
 Loss for taking
specific wrong
action (it's given
to you)

16
17
18
19
 Let's think of a market. Two different brands of
eggs come to the market. Information about
eggs from the experience gained and the records
kept is as follows:
◦ Brands: Br1 Egg and Br2 Egg
◦ Daily supply amount: Br1 800, Br2 600
◦ Broken egg(K) rate: 05%
 The question here is: what is the probability that
an egg coming from Br2 will be broken in one
day?

20
 A total of 1400 eggs,
 600 of them come from Br2
 70 broken eggs per day (total * broken rate)
 If we assume equal distribution of the broken
parts according to the brands, there are 35
Br2
 Then the probability of Br2 being broken is
35/600 = 0.058333

21
 P(Br2|K) * P (K)
P(K|Br2)=
P(Br2)

22
 P(Br2|K) * P (K)
P(K|Br2)=
P(Br2)

23
 Members of a consulting company rent a car at a rate of 60%
from the 1st enterprise, 30% from the 2nd enterprise and 10%
from the 3rd enterprise. If 9% of the vehicles coming from the
first enterprise, 20% of the vehicles coming from the second
enterprise and 6% of the vehicles coming from the third
enterprise require maintenance;

◦ a) What is the probability that a vehicle rented to the company will require
maintenance?
◦ b) What is the probability that the vehicle requiring maintenance came
from the second enterprise?
 B: A car requires maintenance.
 Ai: Let the car come from the 1st, 2nd or 3rd enterprise With i=
1, 2, 3.

24
  P(B)—> the probability that the car will require
maintenance.
 From the total probability is found using:

 Then 12% of the vehicles rented by this company will require maintenance.

25
 the probability that the vehicle requiring
maintenance came from the second enterprise:

26
The Classification Problem Katydids
(informal definition)

Given a collection of annotated

data. In this case 5 instances
Katydids of and five of
Grasshoppers, decide what type of
insect the unlabeled example is. Grasshoppers

Katydid or Grasshopper?
 For any domain of interest, we can measure features

Color {Green, Brown, Gray, Other} Has Wings?

Abdomen Thorax
Length Length Antennae
Length

Mandible
Size
Spiracle
Diameter Leg Length
Grasshoppers Katydids

10
9
8
7

Antenna Length
6
5
4
3
2
1

1 2 3 4 5 6 7 8 9 10
Abdomen Length

Let’s get lots more data…

With a lot of data, we can build a histogram. Let
us just build one for “Antenna Length” for now…
10
9
8
7

Antenna Length
6
5
4
3
2
1

1 2 3 4 5 6 7 8 9 10

Katydids
Grasshoppers
We can leave the
histograms as they are,
or we can summarize
them with two normal
distributions.

Let us us two normal

distributions for ease
of visualization in the
following slides…
• We want to classify an insect we have found. Its antennae are 3 units long.
How can we classify it?

• We can just ask ourselves, give the distributions of antennae lengths we have
seen, is it more probable that our insect is a Grasshopper or a Katydid.
• There is a formal way to discuss the most probable classification…

p(cj | d) = probability of class cj, given that we have observed d

Antennae length is 3
p(cj | d) = probability of class cj, given that we have observed
d
P(Grasshopper | 3 ) = 10 / (10 + 2) = 0.833
P(Katydid | 3 ) = 2 / (10 + 2) = 0.166

10

Antennae length is 3
p(cj | d) = probability of class cj, given that we have observed
d
P(Grasshopper | 7 ) = 3 / (3 + 9) = 0.250
P(Katydid | 7 ) = 9 / (3 + 9) = 0.750

9
3

Antennae length is 7
p(cj | d) = probability of class cj, given that we have observed
d
P(Grasshopper | 5 ) = 6 / (6 + 6) = 0.500
P(Katydid | 5 ) = 6 / (6 + 6) = 0.500

66

Antennae length is
5
That was a visual intuition for a simple case of the Bayes
classifier, also called:

• Idiot Bayes
• Naïve Bayes
• Simple Bayes

We are about to see some of the mathematical formalisms, and

more examples, but keep in mind the basic idea.

Find out the probability of the previously unseen instance

belonging to each class, then simply pick the most probable class.
Assume that we have two classes
(Note: “Drew
c1 = male, and c2 = female. can be a male Drew Barrymore
or female
We have a person whose sex we do not know, say name”)
“drew” or d.
Classifying drew as male or female is equivalent to
asking is it more probable that drew is male or female,
i.e which is greater p(male | drew) or p(female | drew)

What is the probability of being Drew Carey

called “drew” given that you are a

male?
What is the
probability of being
a male?
p(male | drew) = p(drew | male ) p(male)
p(drew) What is the probability of being
named “drew”? (actually irrelevant,
since it is that same for all classes) 37
 This is Officer Drew (who arrested me in
1997). Is Officer Drew a Male or Female?
Luckily, we have a small
database with names and sex.

We can use it to apply Bayes Name Sex

rule…
Drew Male
Officer Drew Claudia Female
Drew Female
Drew Female
p(cj | d) = p(d | cj ) p(cj) Alberto Male
p(d) Karin Female
Nina Female
Sergio Male
 Name Sex
Drew Male
Claudia Female
Drew Female
Drew Female
p(cj | d) = p(d | cj ) p(cj) Alberto Male
p(d) Karin Female
Officer Drew Nina Female
Sergio Male
p(male | drew) = 1/3 * 3/8 = 0.125
3/8 3/8 Officer Drew is
more likely to be
p(female | drew) = 2/5 * 5/8 = 0.250 a Female.
3/8 3/8
 Officer Drew IS a female!

Officer Drew

p(male | drew) = 1/3 * 3/8 = 0.125

3/8 3/8

p(female | drew) = 2/5 * 5/8 = 0.250

3/8 3/8
So far we have only considered Bayes p(cj | d) = p(d | cj ) p(cj)
Classification when we have one
attribute (the “antennae length”, or the p(d)
“name”). But we may have many
features.
How do we use all the features?
Name Over 170CM Eye Hair length Sex
Drew No Blue Short Male
Claudia Yes Brown Long Female
Drew No Blue Long Female
Drew No Blue Long Female
Alberto Yes Brown Short Male
Karin No Blue Long Female
Nina Yes Brown Short Female
Sergio Yes Blue Long Male
 To simplify the task, naïve Bayesian classifiers assume
attributes have independent distributions, and thereby
estimate

p(d|cj) = p(d1|cj) * p(d2|cj) * ….* p(dn|cj)

The probability of
class cj generating
instance d, equals….
The probability of class cj
generating the observed
value for feature 1,
multiplied by..
The probability of class cj
generating the observed
value for feature 2,
multiplied by..
 To simplify the task, naïve Bayesian classifiers
assume attributes have independent
distributions, and thereby estimate
p(d|cj) = p(d1|cj) * p(d2|cj) * ….*
p(dn|cj)

p(officer drew|cj) = p(over_170cm = yes|cj) * p(eye =blue|cj) * ….

Officer
Drew is
blue-eyed,
over 170cm
tall, and has p(officer drew| Female) = 2/5 * 3/5 * ….
long hair
p(officer drew| Male) = 2/3 * 2/3 * ….
The Naive Bayes classifiers is
often represented as this type of cj
graph…

Note the direction of the arrows,

which state that each class causes
certain features, with a certain
probability

p(d1|cj) p(d2|cj) … p(dn|cj)

Naïve Bayes is fast and cj
space efficient

We can look up all the probabilities

with a single scan of the database
and store them in a (small) table…

p(d1|cj) p(d2|cj)
… p(dn|cj)

Sex Over190c Sex Long Sex

m Hair Male
Male Yes 0.15 Male Yes 0.05
No 0.85 No 0.95 Female
Female Yes 0.01 Female Yes 0.70
No 0.99 No 0.30
An obvious point. I have used a
simple two class problem, and
cj
two possible values for each
example, for my previous
examples. However we can have
an arbitrary number of classes, or
feature values

p(d1|cj) p(d2|cj) p(dn|cj)

…
Animal Mass >10kg Animal Color Animal
Cat Yes 0.15 Cat Black 0.33 Cat
No 0.85 White 0.23
Dog Yes 0.91 Dog
Brown 0.44
No 0.09 Dog Black 0.97
Pig
Pig Yes 0.99 White 0.03
No 0.01 Brown 0.90
Pig Black 0.04
White 0.01
47