Bayes’ Rule

There is a clever trick that lets us relate P (A|B) to P (B|A). It is

called Bayes’ rule, and it is one of the most important results in all
of science and engineering.

Let A, B be events with P (B) > 0. Then Bayes’ rule states

P (B|A) P (A)
P (A|B) = .
P (B)

This relationship is easy to verify. We know that

P (A \ B) P (A \ B)
= P (A|B) , and = P (B|A) ,
P (B) P (A)

and so
P (A \ B) = P (A|B) P (B) = P (B|A) P (A) ,
and Bayes’ rule follows immediately.

One of the main applications of Bayes’ rule is for inference from in-
direct observations. The following example is a classic illustration of
this.

39
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
Example. The incidence rate for a certain disease is 15/100,000.
There is a test for the disease which is 95% accurate: if a person has
the disease, the test comes back positive with probability 0.95; if a
person does not have the disease, the test comes back negative with
probability 0.95.
Emma goes to the doctor and is given the test as part of a routine
checkup. It comes back positive. What is the probability that Emma
has the disease?
To answer the question, define the events
A = {Emma has the disease}, B = {Emma tested positive}.
We know that
P (A) = 0.00015
P (B|A) = 0.95
P (B c |Ac ) = 0.95, (and so P (B|Ac ) = 1 P (B c |Ac ) = 0.05).

We can calculate P (B) using the Total Probability Theorem:

P (B) = P (A) P (B|A) + P (Ac ) P (B|Ac ) ,
where Ac is the complement of A:
Ac = {Emma does not have the disease}.
We have
P (B) = (0.00015)(0.95) + (0.99985)(0.05) ⇡ 0.0501,
and so
P (B|A) P (A)
P (A|B) =
P (B)
(0.00015)(0.95)
= ⇡ 0.0028,
(0.00015)(0.95) + (0.99985)(0.05)
or about 0.28%.

40
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
The preceding example illustrates some general principles that are
valuable to take note of. First, for some probabilities it is some-
times easier to specify conditional probabilities than unconditional
ones (e.g., P (B|A) is easier to specify than P (B)). Second, we often
have easier access to some conditional probabilities than others (e.g.,
P (B|A) is easier to get than P (A|B) because we designed the test
and can evaluate it), but we need Bayes’ rule because the other con-
ditional probability we can’t observe is what we need to answer our
question. Finally, using Bayes’ rule often relies on an unconditional
probability (e.g., P (B)) that we don’t have access to. But, using our
conditional probabilities we can get it by applying the law of total
probability.

Exercise:
Let’s go back to the problem of Robert, the injured star quarterback.
Suppose you could not bear to watch the game, but you find out later
that your team won. What is the probability that Robert played?
Ans. As before we define A to be the event that Robert plays, and
B to be the event that the team wins. We know that P (A) =
0.4, P (Ac ) = 1 P (A) = 0.6, P (B|A) = 0.75 and P (B|Ac ) = 0.35.
Since A and Ac are disjoint sets, we can use the total probability
theorem to write:

P (B) = P (B|A) P (A)+P (B|Ac ) P (Ac ) = 0.75·0.4+0.35·0.6 = 0.51

In this problem we are interested in P (A|B), so we must turn to

Bayes’ rule:

P (B|A) P (A) 0.75 · 0.4 30

P (A|B) = = = ⇡ 0.588.
P (B) 0.51 51

41
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
Methods for Solving Problems
Let’s take a step back for the moment and review some of the tech-
niques we’ve developed for calculating probabilities.

Enumeration
One technique we have used to compute the probability of an event is
enumerate (i.e., write an explicit list of) all possible outcomes of the
experiment. If the outcomes are equally likely, then the probability
of the event occurring is the ratio of the number of “successes” (i.e.,
the number of outcomes leading to the event) to the total number of
possible outcomes. If the elementary outcomes have di↵erent proba-
bilities, then the probability of the event occurring is the sum of the
probabilities of the outcomes that correspond to a “success”. This is
the method we have used the most so far.

Divide-and-conquer
Another approach (that we discussed at the beginning of this lec-
ture) is the divide-and-conquer approach of splitting up the sample
space into more manageable subsets and calculating the conditional
probabilities for each subset (possibly using Bayes’ rule if necessary),
combining the results using the total probability theorem.

Sequential
When dealing with a sequence of outcomes, it can be useful to break
up the calculation of a particular probability into more manageable
pieces by following along the “history” of the sequence of outcomes.
We have already done this a few times (our use of the multiplication
rule is a prime example), but a couple more examples will hopefully
clarify how this technique can be used to help organize and sometimes
simplify our calculations.

42
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
Example. We draw 4 cards from an ordinary 52 card deck. What is
the probability that we draw at four “Aces”?
To answer this, let
A1 = {the 1st card is an Ace}
...

A4 = {the 4th card is an Ace}

We are interested in
P (A1 \ A2 \ A3 \ A4 )
which we can write (using the multiplication rule) as
P (A1 ) · P (A2 |A1 ) · P (A3 |A1 \ A2 ) · P (A4 |A1 \ A2 \ A3 )
Calculating each of these probabilities then yields
4 3 2 1
P (A1 \ A2 \ A3 \ A4 ) = · · · ⇡ 0.0000037.
52 51 50 49

Example. You are in Las Vegas and decide to bet 10 on each of

3 football games. For each game, if you win, you will double your
money (so that if you predict correctly you make 10 and if you are
wrong you lose 10). Your knowledge gives you di↵erent degrees of
confidence. You estimate your chances of correctly predicting each
outcome as:
P (1st game correct) = 2
3
P (2nd game correct) = 1
2
P (3rd game correct) = 3
4

What is the probability that you come out ahead? (That is, what is
the probability that you win at least two of the bets?)

43
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
You can think about this by drawing a tree with 3 levels, where the
edges depict you winning or losing certain bets, and we label the
nodes with the probability that we arrive there.

1 st game 2 nd game 3 rd game PCW Payday

Put
1/4 WWW
P wi 1/3
W
W
L 1/12 WWL
2/3

L W 1/4 WLW
W
1/3
L 1/12 WLL

W 1/8 LWW
1/6
L W
L 1/24 LWL
1/3
W 1/8 LLW
L
1/6
L 1/24 LLL

1 1
Thus, our probability of winning at least two of the bets is: 4 + 12 +
1 1 17
4 + 8 = 24 ⇡ 0.71.
PLWWWUWWLUWLWULWW
Now, what happens if we only have 10 to wager (so that if we lose
the first bet, we no longer have enough money to continue)?
The probability is now 14 + 12 + 14 = 12
7
⇡ 0.58. Note how easy this
I diagram.
calculation was given the tree

44
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
In this example, each outcome is independent of the other, but they
occur with di↵erent probabilities (as opposed to a sequence of fair
coin tosses) and the order in which they occur can be important. In
such cases the sequential method, sometimes with accompanying dia-
grams, can be a useful way to keep track of the relevant probabilities.
Ultimately, this is not really dramatically di↵erent from just enumer-
ating the possible outcomes and adding up the relevant possibilities,
but it often provides a useful way to organize the calculations (and
can also be very useful in situations where direct enumeration is not
possible).

The next example shows that the same approach can also be very
handy when the sequence of outcomes are not independent.

Exercise:
We draw 3 cards from an ordinary 52 card deck. What is the proba-
bility that we draw at least two hearts?

Using a similar tree structure as before one can calculate that the
probability of three hearts is given by
13 12 11
· ·
52 51 50
and the probability for any of the three sequences resulting in exactly
two hearts is
13 12 39
· · .
52 51 50
Adding these up results in a probability of approximately 0.15.

45
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
Counting
You may have noticed that all the problems we’ve been talking about
so far have involved experiments with only a small number of out-
comes (e.g., experiments with only two outcomes, repeated only two
or three times). Strategies where we explicitly enumerate all possible
outcomes (or draw a tree showing all possible outcomes) are no longer
feasible when the number of outcomes gets much larger than this. To
handle cases where the number of outcomes is rather large, we typi-
cally have to simplify our calculations carefully counting the number
of ways a certain event can occur without explicitly listing them all.
We’ve already done this a few times (e.g., we have used the fact that
there are 210 possible outcomes if we toss a coin ten times without
ever listing all 210 possibilities) but we haven’t discussed principled
methods for handling more complicated scenarios yet.
There is an entire branch of mathematics — called combinatorics —
that deals with problems of this type. We will of course only scratch
the surface in this course, but we will end up acquiring some powerful
tools.

46
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
The counting principle
Many counting problems can be naturally broken down into multiple
stages. If the outcomes at one stage do not a↵ect the number of
possibilities at the subsequent stages, we can just multiply the number
of possibilities at each stage together. To formalize this, suppose that
our problem has S stages, and that:
1. there are n1 possibilities at stage 1;
2. for every possible result at the first stage, there are n2 possible
results at the second stage;
3. for any sequence of results at the first i 1 stages there are ni
possibilities at stage i;
then the total number of possible results is
n1 · n2 · · · nS

This idea can be demonstrated graphically with a tree, where the

main goal is to count the number of “leaves” on the right side.

n1 choices n2 choices n3 choices

47
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
Example. How many three letter strings are there using the letters
“a”–“z”?.

Ans. 26 · 26 · 26 = 17, 576.

Example. How many three letter strings are there where the second
letter is a vowel?

Ans. 26 · 5 · 26 = 3380.

Example. How many three letter strings are there where the first
letter is a consonant, the second letter is a vowel, and the third letter
is a consonant?

Ans. 21 · 5 · 21 = 2205.

Example. How many three letter strings are there so that no two
consecutive letters are the same?

Ans. 26 · 25 · 25 = 16, 250.

Example. How many three letter strings are there that do not in-
clude any repeated letters?

Ans. 26 · 25 · 24 = 15, 600.

Example. How many five letter strings are there that start with a
consonant and end with “y”?

Ans. 21 · 26 · 26 · 26 · 1 = 369, 096.

48
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
Sometimes we are interested in a sequence of events where the num-
ber of possibilities at the next stage depends on the outcome of the
previous stages. While the goal is still to count the number of leaves,
we now need to enumerate the di↵erent possible paths for getting
from left to right in the tree.

Example. How many three letter strings are there that do not have
consecutive vowels?

Ans. 5 · 21 · 26 + 21 · 5 · 21 + 21 · 21 · 26 = 16, 401.

Example. How many three letter strings have two or fewer conso-
nants?

Ans. 5 · 26 · 26 + 21 · 5 · 26 + 21 · 21 · 5 = 8315.

Example. There are five parking spots arranged in a row. How

many ways can two di↵erent cars park so that they are not next to
one another?

Ans. 3 + 3 · 2 + 3 = 12.

Example. How many five letter strings start with “t”, end with “n”,
and have no two consecutive letters the same?

Ans. This one is pretty tricky. There are several way to try to get the
answer, but the easiest way to think about this involves breaking the
problem into three cases: “t” in the middle, “n” in the middle, and
neither “t” nor “n” in the middle. By counting up these possibilities
we get 25 · 24 + 24 · 25 + 243 = 15, 024.

We can use the counting principle to derive expressions for four gen-
eral types of calculations that often come up: permutations, k-
permutations, combinations and partitions.

49
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
Permutations
How many ways can I order a set of n objects?
There are n choices for the first position, then n 1 choices for the
second, then n 2 for the third, etc:

n(n 1)(n 2) · · · (1) = n!

Example. You want to visit Nashville, Charlotte, Tampa, and Birm-

ingham. How many ways are there to arrange your travel?

Ans. 4! = 4 · 3 · 2 · 1 = 24.

Example. You want to visit a city in all 50 states (starting and

ending in Atlanta, Georgia). How many possible ways are there to
arrange your travel?

Ans. 49! ⇡ 6.083 · 1062 .

50
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
k-permutations
More generally, we might want to count the number of ways we can
pick an ordered subset of size k from a larger set of size n.

As before, there are n choices for the first stage, n 1 for the second,
and so on, but there are only k stages:
n!
n(n 1)(n 2) · · · (n k + 1) =
(n k)!

Example. Suppose that 20 horses compete in the Kentucky Derby.

How many ways can 1st , 2nd , and 3rd place be awarded?

Ans. There are 20! ways the horses can be ordered. Given the first
three slots, we do not care about the ordering of the last 17, so we
divide by 17!. This yields
20!
= 20 · 19 · 18 = 6840.
17!

Combinations
We have a set of size n. How many subsets of size k are there?

In this case, the ordering of the subset does not matter. For
example, all of the 2-permutations of the letters A,B,C are
AB, BA, AC, CA, BC, CB (6 total),
while all the 2-letter combinations are
AB, AC, BC (3 total)

51
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
The number of k-combinations is simply the number of k-permutations
divided by the number of “duplicates” ... since we can order a set of
size k in k! di↵erent ways:
✓ ◆
n n!
“n choose k” = =
k (n k)!k!

This quantity is also known as the binomial coefficient.

Example. There are 12 people on a basketball team, but only 5

people play at a time. How many di↵erent 5 person lineups are there?

Ans. ✓ ◆
12 12! 12 · 11 · 10 · 9 · 8
= = = 11 · 9 · 8 = 792.
5 7!5! 5·4·3·2

Example. How many binary strings of length 10 contain exactly 4

ones?

Ans. You can think of this as the number of ways you can choose the
4 locations (out of 10 possible) for the ones, and
✓ ◆
10 10! 10 · 9 · 8 · 7
= = = 210.
4 6!4! 4·3·2·1

52
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
 Think of it as many
Partitions combinations

The number of combinations nk can be interpreted as the number of

ways we can partition the set into two subsets of sizes k and n k.
We can generalize the expression for the more general problem:
How many ways can a set of size n be partitioned into r non-
overlapping subsets of size n1 , n2 , . . . , nr ?
(The constraint n1 + n2 + · · · + nr = n is implicit.)

We answer this by forming the subsets in stages. The first subset can
be formed in nn1 ways. For the second, n n1 elements remain, and
so there are n n2n1 possibilities, etc.

So the total number of partitions is

✓ ◆✓ ◆✓ ◆ ✓ ◆
n n n1 n n1 n2 n n1 ··· nr 1
···
n1 n2 n3 nr

We can expand then simplify this as

n! (n n1 )! (n n1 · · · nr 1 )!
· ···
n!(n n1 )! n2 !(n n1 n2 )! nr !(n n1 · · · nr )!
n!
= .
n1 !n2 ! · · · nr !
This has the special notation
✓ ◆
n n!
= ,
n 1 , n2 , . . . , n r n1 ! · · · nr !

which is also sometimes called the multinomial coefficient. Note that

this is the binomial coefficient when r = 2.

53
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
Example (Anagrams). How many di↵erent words can be obtained
by rearranging the letters in the word BANANA?
Ans. We can think of having six slots for letters that we want to
partition into subsets of size 3 (for the As), 2 (for the Ns) and 1 (for
the B). The number of such partitions is
✓ ◆

I
6 6!
= = 60.
3, 2, 1 3!2!1!

Summary
To summarize all in one place, we can use the counting principle to
derive expressions for four types of calculations.
Permutations. How many ways can I order a set of n objects?
Answer: n! = n(n 1)(n 2) · · · 2 · 1

k-permutations. How many ways can I select k out of n objects

and arrange them in a sequence?
n!
Answer:
(n k)!
Combinations. How many ways can I select k objects out of n when
order does not matter?
n!
Answer:
k!(n k)!
Partitions. How many ways can I divide n objects into r groups
where the ith group has ni members? (So n1 + n2 + · · · + nr = n.)
n!
Answer:
n1 !n2 ! · · · nr !

54
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
Bernoulli trials
A Bernoulli trial is a random experiment with two outcomes, generi-
cally labeled “success” and “failure” (or “heads”/“tails”, “yes”/“no”,
“1”/“0”, “win”/“loss”, etc.) We will use p to denote the probability
of success:
p = P (success) and so 1 p = P (failure) .
The phrase Bernoulli trials refers to a sequence of independent ex-
periments of this type.

We are interested in calculating the probability of k successes in n

trials. As you will see, this is closely related to the combinatorics
ideas we’ve been discussing.

Example. I am flipping a fair coin. What is the probability that

exactly 2 out of the next 3 flips will be “heads”?

There are 8 possible outcomes and each outcome is equally likely.

Since there are three ways to get a successful event, the probably of
interest is
3
P ({HHT,THH,HTH}) = .
8
But, let’s make explicit the other way we can think about this. The
three successful outcomes are disjoint events, so we can write
P ({HHT,THH,HTH}) = P (HHT) + P (THH) + P (HTH) .
For each outcome, since it is the result of independent coin tosses, we
can write it using the multiplication rule: P (HHT) = P (H) · P (H) ·
P (T) = ( 12 )3 = 18 = P (THH) = P (HTH). So, the total probability is
again P ({HHT,THH,HTH}) = 38 .

55
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
In general, for any specific sequence with k successes in n trials, the
probability of that sequence is pk (1 p)n k . Moreover, there are nk
possible sequences, and thus
✓ ◆
n k
P (exactly k successes out of n) = p (1 p)n k ,
k

and
n ✓ ◆
X n
P (at least k successes out of n) = pi (1 p)n i .
i
i=k

Example. I am a 70% free throw shooter. What is the probability

that I make exactly 6 out of my next 10 free throws?

Ans. The probability of any given sequence of 10 free throws with

6 makes and 4 misses is (.7)6 (.3)4 . To find the total probability, we
simply have to add up this probability for each of these sequences.
So, how many ways are there to get 6 success in 10 trials? There are
10
6 , giving an answer of
✓ ◆
10
P (make exactly 6 out of 10) = (.7)6 (.3)4 ⇡ 0.2001.
6

We can also calculate the probability that I make at least 6 out of

ten:
X10 ✓ ◆
10
P (make at least 6 out of 10) = (.7)i (0.3)10 i ⇡ 0.8497.
i=6
i

56
ECE 3077 Notes by M. Davenport, J. Romberg, C. Rozell, and M. Wakin. Last updated 11:55, August 30, 2023
 Blue 2 213 7

1167 i

I
1

41