Statistics for Economics Handout 1: Overview of Probability Theory

CHAPTER ONE: OVERVIEW OF BASIC PROBABILITY THEORY

1. INTRODUCTION

Statistics is one of the most important and useful tools for many scientific investigations. This
course on statistics begins with the fundamentals of Probability on the assumption that
students have taken introductory course on statistics.
The field of statistics has two aspects (or broad sub-divisions): Descriptive statistics and
inferential (analytical) statistics.
Descriptive statistics – is concerned with the collection, processing, summarizing and
describing important features of the data without going beyond (i.e. without any attempt to
infer from the data).
Inferential (Analytical) statistics – is concerned with the process of using data obtained from
sample to make estimates or test hypotheses about the characteristics of a population. It
consists of a host of techniques that help decision makers to arrive at rational decisions under
uncertainty.
Actually, data are sought for a large group of elements (individuals, households, products,
etc). However, due to time, cost and other considerations, data are collected from only a
small portion of the group. Thus, economists, managers and other decision makers draw
conclusions, make estimates and test hypotheses about the characteristics of population from
the data for a small portion of the group. This process is referred to as statistical inference.
In descriptive statistics, we cannot generalize the results it was outside the observed data
under consideration. Any question relating to the population from which the observed data
were drawn cannot be answered within the descriptive statistics framework. In order to be
able to do that we need the theoretical framework offered by probability theory. In effect,
probability theory develops a mathematical model that provides the logical foundation of
statistical inference procedures for analyzing observed data.

Page 1 of 15
Statistics for Economics Handout 1: Overview of Probability Theory

2.PROBABILITY THEORY
2.1 Introduction:
Personally, in our daily lives we are faced with many decision-making situations that involve
uncertainty. Perhaps you may ask yourself to analyze one of the following situations:
- What is the chance for me to score “A” in statistics?
- What is the likelihood that your (our/my) weekend picnic be successful?
In such situations, we use the concept of probability in our daily life without detailed and
actual knowledge of the concept in other words we use it intuitively.
Professionally, much of statistical theory and practice rests on the concept of probability,
since conclusions concerning population are drawn from samples and this is subject to certain
amount of uncertainty. Besides, you may be asked one of the following:
As business economist: What is the chance that sales or quantity demand will increase if the
price of the product reduced?
As project analyst: What likely is the project will be completed on time?
The subject matter most useful in effectively dealing with such uncertainties is enclosed
under the heading probability.Probability can be thought as a numerical measure of the
chance of likelihood that a particular event will occur.
2.2 Some basic concepts in probability theory
In probability theory, we are concerned with an experiment with an outcome depending on
chance, which is called a random experiment.
Experiment / Trial: An experiment or trial is an act that can be repeated under given identical
conditions.
Example: Throwing a die, tossing a coin are the examples of experiment or trial.
Outcome:An outcome is the result of an experiment.
Examples
 If we throw a die once, we get 1 or 2 or 3 or 4 or 5 or 6. So that individually 1 is an
outcome or sample point, 2 is an outcome or sample point and etc.
 If we toss a coin once, we get head or tail. Individually head and trial are two
outcomes.

Page 2 of 15
Statistics for Economics Handout 1: Overview of Probability Theory

Sample space: A sample space is the set of all outcomes/sample points.

Example
 If we throw a die once, the outcomes are 1, 2, 3, 4, 5 and 6. Then, S  {1, 2,3, 4,5, 6} is
a sample space.
 If we toss a coin once, then the outcomes are head (H) and tail (T). Then, S  {H , T } is
a sample space.
Event: Each possible outcome is called a sample point. An event is the collection of one or
more sample points of an experiment. It is a subset of the sample space having one or more
sample points as its elements.
Example
 If we throw a die, the outcomes are 1, 2, 3, 4, 5, and 6. Then, the outcomes of even
numbers are 2, 4, 6. Then, A  {2, 4, 6} is called an event of even numbers.
Events may be Simple or Compound events
Simple Event: is a subset of sample space that has exactly one sample point. It can also be
called as element or fundamental event.
Compound Event: is a subset of sample space that has two or more
sample points.
Complement Event – the complement of event A is denoted by A’. A’ is the event that has all
the points in a sample space that are not in A.
Example: Rolling a die: S = {1,2,3,4,5,6}
Event A = {1, 3, 5}
Complement event of A, A’ = {2, 4, 6}.
Impossible Event: is a subset of sample space that contains none of the sample points.
E.g. Rolling a die: S = {1, 2, 3, 4, 5, 6}
E = {7} or E = {0} are impossible events
Independent Events: Two events are said to be independent when the happening of one
event does not affect the happening of the other.

Page 3 of 15
Statistics for Economics Handout 1: Overview of Probability Theory

Example: In tossing a coin twice, the happening of Head or Tail in the first toss does
not affect the happening of head or tail in the second toss. The two tosses are
therefore independent.
Dependent Events- Two events are said to be dependent when the happening (or occurrence)
and non-occurrence of an event affects the happening of another event.
Example: If a person randomly selects a ball in two rounds from a box containing two
black and two white balls without replacing the first ball from the first round selection,
the probability of selecting a white/black ball in the second round depends on the
probability of selecting a white/black ball in the first round. Then, we say that the
probability of selecting a white/black ball in the second round depends on the
happening of a white/black ball in the first round.
Mutually Exclusive Event: Events are said to be mutually exclusive if one and only of them can
take place at a time.
Example: In tossing a coin once, the outcomes Head and Tail are mutually exclusive in
that only one of them can happen and not both at the same time.
Collectively Exhaustive Lists: When a set of events for an experiment includes every possible
outcome, the set is said to be collectively exhaustive event/list.
Example: In flipping a fair coin twice if the list or set has all the possible outcomes
(that is {HH, HT, TH, TT}), then it is said to be collectively exhaustive.
2.3 Definition of Probability
Experts disagree about the concept of probability, since there are various conceptual
approaches in defining probability. The most common are discussed below:
i. Classical Approach (or A priori Definition of Probability)
This approach is based on the assumption that each of the possible outcomes must be
mutually exclusive and equally likely. If a trial results in n exhaustive, mutually exclusive and
equally likely cases and m of them are favorable to the happening of an event E, then the
probability of event E happening, i.e. P (E), is given by:

( )
( )

Page 4 of 15
Statistics for Economics Handout 1: Overview of Probability Theory

 Equally likely means that each outcome of an experiment has the same chance of
happening as others have.
Example: In rolling a fair dice once, the probability of observing the outcome 1 is:
* + ( )
Shortcoming of the Classical Approach
The classical method was originally developed in the analysis of gambling problems, where
the assumption of equally likely and mutually exclusive outcomes is often reasonable. It does
not provide ways to compute probabilities for events that are not equally likely.
ii. Relative Frequency / Statistical or Empirical Approach
According to this approach, the probability of an event is the proportion of times that this
event occurs over the long run if the experiment is repeated many times under uniform
conditions.
If a trial is repeated a number of times under essentially homogeneous and identical
conditions, then the limiting value of the ratio of the number of times the event happens to
the number of trials as the number of trials become indefinitely large is called the probability
of happening of the event. It is assumed that the limit is finite and unique.
Symbolically, if in n trials an event E happens m times, then the probability p of the
happening of event E is given by
m
P(E)= lim
n  n

Sometimes this approach is referred to as objective probability since experiments should be

conducted or recorded data must be there in order to compute probability.
Shortcoming of the Relative Frequency Approach
o People often use it without evaluating a sufficient number of outcomes.
o This approach seems to imply that probability can play no part in situations
that occur only once.
iii. Subjective (Personal) Approach
Subjective probability can be defined as the probability assigned to an event by an individual,
based on whatever evidence is available. This evidence may be in the form of relative
frequency of past occurrences, or it may be just educated guess.

Page 5 of 15
Statistics for Economics Handout 1: Overview of Probability Theory

o Subjective probability assignments are frequently found when events occur

only once or at most a very few times.
o In fact, most high-level social and managerial decisions are concerned with
specific unique situations, rather than with a long series of identical situations,
decisions makers at this level make considerable use of subjective
probabilities.
This approach is used when outcomes are not mutually exclusive and there is no objective
data. Generally, though there are three approaches of probability, we can use any of the
aforementioned approach determined depending up on the problem under consideration.
2.4 Basic Axioms and Theorems of Probability
The steps to be followed in determining the probability of an event are as follows:
1. Define the experiment: that is, describe the process used to make an observation and
the type of observation that will be recorded.
2. List the sample points
3. Assign probabilities to the sample points
4. Determine the collection of sample points contained in the event of interest
5. Sum the sample point probabilities to get the event probability
The axioms or rules of probabilities provide the general framework in dealing with probability
of events. The axioms are common to every one of the three approaches to probability.
(a) The probability of an event must lie between 0 and 1, i.e. For any event A,
( )
The explanation is straightforward. If A is certain to occur, it occurs in 100% of all trials and so
its probability is 1. If A is certain not to occur then its probability is 0, since it never happens
however many trials there are. Since one cannot be more certain than certain, probabilities of
less than 0 or more than 1 can never occur.
(b) The sum of the probabilities associated with all the outcomes in the sample space is 1
Suppose S is a sample space associated with an experiment, the probability of the entire
sample space, i.e. P(S), is equal to one. This postulate states indirectly that certainty is
identified with a probability of 1; after all, it is always assumed that one of the possibilities in

Page 6 of 15
Statistics for Economics Handout 1: Overview of Probability Theory

S must occur, and it is to this certain event that we assign a probability of 1ur, and it is to this
certain event that we assign a probability of 1.
(c) Complement Rule: the complement of an event is everything in the sample space apart
from that event.
The complement of heads is tails, for example. If we write the complement of A as not-A then
it follows that ( ) ( ) and hence:
( ) ( )
Most practical problems require the calculation of the probability of a set of outcomes or
compound events rather than just a single one, or the probability of a series of outcomes in
separate trials. These compound events (whose probability are called Joint Probability) are
made up of simple events compounded by the words “Or” and “And” which act as operators.
The following rules for manipulating probabilities show how to handle these operators and
thus how to calculate the probability of a compound event.
(d) Rules of Addition
This rule is associated with the operator OR which is designed by the sign “U”. When we want
the probability of one outcome or another occurring, we add the probabilities of each.
Special rule of Addition: If two events A and B are mutually exclusive, the special rule of
addition states that the probability of one or the other events occurring equals the sum of
their probabilities. That is,
( ) ( ) ( ) ( )
For three mutually exclusive events designated, A, B and C the rule is written as,
( ) ( ) ( ) ( ) ( )
Example: If we roll a fair die once, what is the probability of a 5 or a 6?
Solution: Here there are two events, namely event and event . So that,
( ) ( ) ( ) ( )
The general rule for addition: When two or more events are not mutually exclusive then we
use the general rule for addition. The rule is
( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )

Page 7 of 15
Statistics for Economics Handout 1: Overview of Probability Theory

Example: Mr. X feels that the probability that he will pass Mathematics is 2/3 and Statistics is
5/6. If the probability that he will pass both the course is 3/5. What is the probability that he
will pass at least one of the courses?
Solution: Let M and S be the events that he will pass the courses Mathematics and Statistics
respectively. The event means that at least one of occurs. Therefore,
P( ) ( ) ( )
( ) ( ) ( )

Joint Probability: When two events overlap, the probability is called a joint probability. In the
exercise below, the probability that Mr. Y gets A in both Calculus and Statistics is an example
of a joint probability. It is a probability that measures the likelihood that two or more events
(getting A in both Calculus and Statistics) will happen concurrently (jointly).The individual
probabilities are referred to as Marginal Probabilities.
In general, the general rule of addition is used to combine and determine Joint Probabilities
for events that are not mutually exclusive.
Exercise: A student feels that the probability that he will get A in Calculus is 3/4, A in
Statistics is 4/5 and A in both courses is 3/5. What is the probability that the student will get
i. At least one A
ii. No A ’s?
(e) Multiplication Rules
The multiplication rule is associated with use of the word ‘And’(∩) to combine events.
The special rule of multiplication: requires that two events A and B be Independent. Two
events are independent if the occurrence of one does not alter the probability of the other.
For two independent events, A and B, the probability that A and B will both occur (written as

P(A ∩B)) is found by multiplying the two probabilities, i.e.

( ) ( ) ( )
For three independent events, A, B, and C, the probability of occurrence of all is given by:
( ) ( ) ( ) ( )

Page 8 of 15
Statistics for Economics Handout 1: Overview of Probability Theory

Example: In terms of the fair coin example, the probability of heads on two successive tosses
is the probability of heads on the first toss (which we shall call H 1) times the probability of
heads on the second toss (H2). We have shown that the events are statistically independent,

because the probability of heads on any toss is 0.5, and P (H1∩ H2) = 0.5 x 0.5 = 0.25. Thus, the

probability of heads on two successive tosses is 0.25.

Exercises: If a fair coin is tossed three times, what is the probability of getting
 Tails, Heads and Tails in that order?
 At least two tails? At least one tail? Three tails? No tails?
General Rule of Multiplication: The general rule of multiplication is used to find the joint
probability that two events will occur if they are dependent. If the probability of event B
occurring depends on the condition that event A has occurred, the probability of event B
happening is called the Conditional Probability of event B and is written as P(B/A). The
General rule states that the joint probability that both events A and B (event B depends on
event A)will happen is found by multiplying the probability that event A will happen by the
Conditional Probability of event B occurring.
Symbolically, the joint probability P(A and B) is found by:
( ) ( ) ( ) ( )
For three dependent events A, B and C, such that C depends on B and Bin turn depends on A,
the joint probability that A, B and C will all occur is given by:
( ) ( ) ( ) ( ⁄ ) ( ⁄( ))
Example 1: There are 10 rolls of films in a box, 3 of which are defective. Two rolls are to be
selected one after another without replacement. What is the probability of selecting a
defective roll followed by another defective roll?
Solution:A roll of film is selected from the box. Obviously, the probability of selecting a
defective roll is 3/10, and the probability of selecting a good roll is 7/10. Then, a second roll is
selected from the box without the first one being returned to the box. The probability that it
is defective depends on whether the first roll selected was defective or good. The probability
that the second roll is defective is:

Page 9 of 15
Statistics for Economics Handout 1: Overview of Probability Theory

2/9 if the first roll was defective. (Only two defective rolls remain in the box containing
nine rolls)
3/9 if the first roll selected was good. (All three defective rolls are still in the box
containing nine rolls.)
The fraction 2/9(or 3/9) is aptly called a conditional probability because its value is
conditional on (dependent on) whether a defective or a good roll of film is chosen in the first
selection from the box. If selecting a defective and good rolls are named as event A and B
respectively, and subscripts 1 and 2 to denote the selection rounds, the probability of
selecting a defective roll followed by another defective roll is computed as:
( ) ( ) ( ) ( )

Example 2: Samples of executives were surveyed about loyalty to their company. One of the
questions was, “if you were given an offer by another company equal to or slightly better
than your present position, would you remain with the company or take the other position?”
The responses of the 200 executives in the survey were cross-classified with their length of
service with the company as shown in the following table.
Length of Service
Loyalty Less than 1 year, 1-5 years, 6-10 years, More than 10 years, Total
B1 B2 B3 B4
Would remain, A1 10 30 5 75 120
Would not remain, A2 25 15 10 30 80
Sub Total 35 45 5 105 200
Now, using the table determine the probability of randomly selecting an executive who is
loyal to the company (would remain) and who has more than 10 years of service?
Solution: note that two events occur at the same time- the executive would remain with the
company (A1), and he or she has more than 20 years of service (B4).
I. Event A1 happens if a randomly selected executive will remain with the company
despite an equal or slightly better offer from another company. Out of the 200
executives in the survey, 120 of them would remain with the company. So, the
probability that event A1 will happen is ( )

Page 10 of 15
Statistics for Economics Handout 1: Overview of Probability Theory

II. Event B4happens if a randomly selected executive has more than 10 years of service.
Thus, ( ) is the conditional probability that an executive with more than 10
years of service would remain with the company despite an equal or slightly better
offer from another company. 75 of the 120 executives who would remain have more
than 10 years of service, so ( ⁄ ) .
III. Thus, the probability that a randomly selected executive will be one who would
remain in the company and who has more than 10 years of service will be:
( ) ( ) ( ( ⁄ ) ( ⁄ ) ( ⁄ )
Exercise: Using the above table determine ( ) and ( )
2.5 Counting Procedures
We have learned that one of the steps in assigning probabilities to events is to list and count
the related events of interest from the sample space. However, when the number of possible
events in an experiment is large, it would be tedious to count all the possibilities. For example
is it easier to count the possible outcomes in rolling a single die. But, it requires more time
and effort to count the possible arrangements if the die is rolled say 10 times. However, we
can avoid the tedious task involved in counting large number of possible arrangement by
using the rules of counting. We will start with the multiplication formula.
a. Multiplication Formula:states that if there are m ways of doing one thing and n ways of
doing another thing, there area total of m x n ways of doing both things. It is applied to
find the number of possible arrangements for two or more groups.
Example:If a TV dealer can offer to sell 10 different brands of TVs and each brand of TV comes
with 6 different price levels, how many different TV sets and prices can the dealer offer?
Solution: Each brand of TV can be sold at 6 different prices. There are a total of 10 TV brands
and as a result the total number of TV sets the dealer can offer is given by 6 x 10 = 60.
b. Permutation: is any arrangement of r objects selected from a single group of n possible
objects. It is denoted by . The permutation formula is applied to find the possible
number of arrangements when there is only one group of objects. It is given by the
formula:

( )

Page 11 of 15
Statistics for Economics Handout 1: Overview of Probability Theory

Where n is the total number of objects and r is the number of objects selected. n! is read as n
factorialand is used to denote the product of n(n-1)(n-2)(n-3)…(1). For example;5!
=5*4*3*2*1 = 120 and 5!/3! = 120/6=20. By definition, zero factorial (0!) is equal to one.
Example 1: If there are three seats where three persons, A, B and C, can seat, in how many
different ways can the three persons seat considering that the order of arrangement matters?
List: ABC, ACB, BCA, BAC, CAB, CBA. We can also determine it using the permutation formula.
There are three persons so n = 3 and because there are only 3 seats r= 3

( ) ( )
Note that in permutation the arrangement ABC is different from BCA.
In the previous example we selected and arranged all the people n = r. However, in many
cases only some objects are selected and arranged from n possible objects.
Example 2:Assume that there are only three seats but there are 8 people. In how many
different ways can the 8 people be arranged or seated?

( ) ( )
Exercise: How many 4 digit numbers can be formed using the numbers 0 to 9? 1 to 9?4 to 8?
c. Combination: if the order of the selected objects is not important, any selection is called
combination. The formula to count the number of r objects from a set of n objects is:
( ) ( )
( ) * ( ) + *( ) ( ) +
For example, if Ketema, Chaltu and Tirhas are to be chosen as a committee to negotiate a
merger, there is only one possible combination of these three: the committee Ketema, Chaltu
and Tirhas is the same as the committee Tirhas Chaltu, Ketema. The Order of the
arrangement has no importance so we use combination:

( ) ( )
Example: The marketing department has been given the assignment of designing color codes
for the 42 different lines of compact disks sold by Google Recods. Three colors are to be used
on each CD, but a combination of 3 colors used for one CD cannot be arranged and used to
identify a different CD. I.e. if green, yellow and violet were to identify one line, then yellow,

Page 12 of 15
Statistics for Economics Handout 1: Overview of Probability Theory

green and violet or any other combination of these three colors cannot be used to identify
another line. Would seven colors taken three at a time be adequate to color-code the 42
lines?
Solution: There are 35 different combinations found by

( ) ( )
The seven colors taken three at a time (i.e. three colors to a line) would not be adequate to
color-code the 42 different lines because they would provide only 35 combinations.
Exercise 1: Would 8 colors taken three at a time be adequate to the above example?
Exercise 2: A family has 10 children. How many different possible combinations of boys and
girls might this family have?

2.6 Bayes’ Theorem(Revising prior probabilities and estimating posterior ones)

In our discussion of conditional probability, we indicated that revising probabilities when new
information is obtained is an important phase of probability analysis. Often, we begin our
analysis with initial or prior probability estimates for specific events of interest. Then, from
sources such as a sample, a special report, or some other means, we obtain some additional
information about the events. Given this new information, we update the prior probability
values by calculating revised probabilities, referred as posterior probabilities, Bayes’ theorem
provides a means for making these probability calculations. The steps in this probability
revision process are shown in figure below.

Prior New Application of Posterior

Probabilities Information Bayes’ Theorem probabilities

Example: An application Of Bayes’ Theorem

Consider a manufacturing firm that receives shipments of parts from two different suppliers.
Let A1 denote the event that a part is from supplier 1 and A2 denote the event that a part is
from supplier 2. Currently, 65% of the parts purchased by the company are from supplier 1
and the remaining 35% are from supplier 2. Hence, if a part is selected at random, we would
assign the prior probabilities P (A1) = .65 and P (A2) = .35.The quality of the purchased parts

Page 13 of 15
Statistics for Economics Handout 1: Overview of Probability Theory

varies with the source of supply. Historical data suggest that the quality ratings of the two
suppliers are as shown in the table below.
Historical quality Levels of Two Suppliers
Suppliers Percentage of Good parts (G) Percentage of Bad parts ( B)
Supplier 1 (A1) 98 2
Supplier 2 (A2) 95 5
If we let G denote the event that a part is good, and B denote the event that a part is bad, the
information in the above table provides the following conditional probability values.
P (G/A1) = 0.98 P (B/A1) = 0.02
P (G/A2) = 0.95 P (B/A2) = 0.05
Based on the above information we can compute the joint probabilities of a part being good
and comes from supplier 1, good and A2, a part being bad and supplied by A1; and bad and
supplied by A2.
P (A1G) = P (A1)* P (G/A1) = 0.637
P (A1B) = P (A1)* P (B/A1) = 0.013
P (A2G) = P (A2)*P (G/A2) = 0.3325
P (A2B) = P (A2) *P(G/A2) = 0175
Suppose now that the parts from the two suppliers are used in the firm’s manufacturing
process and that a machine breaks down because it attempts to process a bad part. Given
the information that the part is a bad, what is the probability that it came from supplier 1 and
what is the probability that it came from supplier 2?
 With the prior probabilities and the join probabilities, Bayes’ theorem can be used to
answer these questions.
Letting B denote the event that the part is bad, we are looking for the posterior probabilities
P (A1/B) and P (A2/B). From the law of conditional probability and marginal probability, we
know that:
( )
( ⁄ )
( )
( ) ( ) ( ) ( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( ) ( ) ( )

Page 14 of 15
Statistics for Economics Handout 1: Overview of Probability Theory

Substituting the above equations, we obtain Bayes’ theorem for the case of two events.
( ) ( )
( ⁄ ) ( ⁄ )
( ) ( ) ( )
( ) ( )
( ⁄ )
( ) ( ) ( ) ( )
Similarly, we can determine P(A2/B) as follows
( ) ( )
( ⁄ )
( ) ( ) ( )
( ) ( )
( ⁄ )
( ) ( ) ( ) ( )
Using the above formula:
0.65x0.02 0.0130
P( A1 / B)    0.4262
(0.65x0.02)  (0.35x0.05) 0.0305
0.35x0.05 0.0175
P( A2 / B)    0.5738
(0.65x0.02)  (0.35x0.05) 0.0305
Note that in this application we started with a probability of .65 that a part selected at
random was from supplier 1. However, given information that the part is bad, the probability
that the part is from supplier 1 drops to .4262. In fact, if the part is bad, there is a better than
50-50 chance that the part came from supplier 2; that is, P (A2/B) = .5738.
Bayes’ theorem is applicable when the events for which we want to compute posterior
probabilities are mutually exclusive and their union is the entire sample space. Bayes’
theorem can be extended to the case where there are n mutually exclusive events A1, A2,…, An
whose union is the entire sample space. In such case, Bayes’ theorem for computing
posterior probability P (Ai/B) can be written symbolically as:
P( Ai ) P( B / Ai )
P( Ai / B) 
P( A1 ) P( B / A1 )  P( A2 ) P( B / A2 )  ...  P( An ) P( B / An )

Page 15 of 15