IE335 Stochastic Models Fall 23

Study Set 2 for Final Exam

1. A white rat is put into the maze of Figure 1. There are four compartments with connections

between the compartments as indicated. The rat moves through the compartments at random

at each time step. That is, if there are k ways to leave a compartment, it chooses each of

these with equal probability. However, a piece of cheese is placed in the center of the maze

(compartment 4) and once the rat finds the food, he stays to enjoy it making the compartment

an absorbing state. Define the Markov Chain by the travels of the rat. Calculate the

transition matrix. Find the expected time to reach the food when starting at compartment

i. What is the expected amount of time the rat is at compartment 3 when it starts from

compartment 1?

Figure 1:

Solution :

Lets define Xn as the compartment the rat is at, at time instance n. So the transition

probability matrix is given below for this Markov Chain.

1 2 3 4
 
1 0 1/2 1/2 0
 
 
2  1/2 0 0 1/2 
P =
 
 
 1/2 0 0 1/2 
 
3
 
4 0 0 0 1

Expected to time reach food is expected time to absorption. Lets solve this. we have µ4 = 0

µ1 = 1 + µ2 /2 + µ3 /2

1
 IE335 Stochastic Models Fall 23

µ2 = 1 + µ1 /2 + µ4 /2

µ3 = 1 + µ1 /2 + µ4 /2

If we solve these equations, we get µ1 = 3

µ2 = µ3 = 2

In order to find the expected amount of time the rat is at compartment 3 when it starts from

compartment 1, we have to solve Mean Time Spent in Transient States.

1 2 3
 
1
 0 1/2 1/2 
 
So, we have PT = 2
 1/2
 0 0 
 
3 1/2 0 0

Lets solve S = (I − PT )−1

1 2 3
 
1
 2 1 1 
 
We get S = 2
 1
 3/2 1/2 

 
3 1 1/2 3/2
The solution is 1.

2. A small barbershop operated by a single barber, has room for at most two customers. Po-

tential customers arrive at a Poisson rate of three per hour, and the successive service times

are independent exponential random variables with mean 1/4 hour.

i. Find the average number of customers in the shop.

Solution : Lets define the states as as the number of customers in the shop. We have 3

states, 0, 1, and 2 customers. The transition diagram is given below.

In order to solve the problem, we have to find the limiting probabilities, (stationary

dist.) Lets write all the equations regarding limiting probabilities.

2
 IE335 Stochastic Models Fall 23

3 3
0 1 2
4 4

3π0 = 4π1

3π1 = 4π2

π0 + π1 + π2 = 1

Here, we get π0 = 16/37,

π1 = 12/37,

π2 = 9/37

Expected number of customers =0 ∗ π0 + 1 ∗ π1 + 2 ∗ π2 = 30/37

ii. What is the proportion of potential customers that enter the shop?

That is obviously, 1 − π2 = 28/37, because when we have 2 customers in the system,

the customer coming to the shop cannot enter.

iii. If the barber could work twice as fast, how much more business would he do?

If the customer work twice as fast, then he would have the service rate 8 per hour.

3 3
0 1 2
8 8

Please do the relevant work to find π0 = 64/97,

π1 = 24/97,

π2 = 9/97

That is making 88/97 of the customers enter the shop. So his business went up from

serving 28/37 of the customers to 88/97 of the customers.

3. Each time a machine is repaired it remains up for an exponentially distributed time with rate

3
 IE335 Stochastic Models Fall 23

λ. It then fails, and its failure is either of two types. If it is a type 1 failure, then the time to

repair the machine is exponential with rate µ1 ; if it is a type 2 failure, then the repair time

is exponential with rate µ2 . Each failure is, independently of the time it took the machine to

fail, a type 1 failure with probability p and a type 2 failure with probability 1 − p. a) What

proportion of time is the machine down due to a type 1 failure? b) What proportion of time

is it down due to a type 2 failure? c) What proportion of time is it up?

Solution : Lets define the states as :

State 0 : Machine is working.

State 1 : Machine is under repair after Type 1 failure.

State 2 : Machine is under repair after Type 2 failure.

pλ 0
(1 − p)λ
µ2
µ1
1 2

Please note that, the machine failure is split into two Poisson events, with probabilities p and

1-p. Therefore, the rates from state 0 to both states 1 and 2 are defined by these probabilities.

After stating the rates of the Markov Chain, we can easily find the limiting probabilities.

Lets write the equations.

π0 + π1 + π2 = 1

pλπ0 = µ1 π1

(1 − p)λπ0 = µ2 π2

Solving these equations together will result in

µ1 µ2
π0 = µ1 µ2 +µ2 pλ+µ1 (1−p)λ , (Answer to c)
µ2 pλ
π1 = µ1 µ2 +µ2 pλ+µ1 (1−p)λ , (Answer to a)
µ1 (1−p)λ
π2 = µ1 µ2 +µ2 pλ+µ1 (1−p)λ , (Answer to b)

4
 IE335 Stochastic Models Fall 23

4. Customers arrive at a two-server station in accordance with a Poisson process having rate λ.

Upon arriving, they join a single queue. Whenever a server completes a service, the person

first in line enters service. The service times of server i are exponential with rate µi , i = 1,2,

where µ1 + µ2 > λ . An arrival finding both servers free is equally likely to go to either one.

i. Define an appropriate continuous-time Markov chain for this model, draw transition

diagram.

First of all, if both of the servers’ rates were equal, our job would be easy, we would

have an M/M/2 system, we could just use the formula of M/M/c systems. So why cant

we use M/M/c formula? (Please check M/M/c slides) That is because, the rate from

state 1 to state 0 (from only one server is occupied to state no server is occupied) we

do not know the rate, it is dependent on which server is actually working! Therefore,

we need to define the Markov Chain, via stating the rates which are dependent on the

server that is occupied. So lets define the states as:

State 0 : Where no customers are present.

State 1.1 : Only one customer, he is being served at the first server.

State 1.2 : Only one customer, he is being served at the second server.

State 2 : Two customers ( one customer at each of the servers)

State 3 : Three custemers (two customers are being served, one customer at the queue)

...

State n : n customers (two customers are being served, n-2 customers at the queue)

λ/2 1.1
λ
µ2 λ λ λ
µ1
0 2 3 4 n
µ1
µ2 µ1 + µ2 µ1 + µ2 µ1 + µ2
λ
λ/2
1.2

5
 IE335 Stochastic Models Fall 23

Note that the number of states can go infinity, there would be many states connected to

this graph (Unfortunately i could not find a way to draw that in Latex :) ) Now, lets go

over the values, When a customer comes to an empty system, he can choose any server,

therefore, the rates to each of the server would be λ/2. From state 1.1 to state 0, server

1 has to finish its job, so, the rate would be µ1 . The same applies to the rate from state

1.2 to state 0. From state 2 to state 1.1, server 2 has to complete its job, therefore the

rate is µ2 . The rates from state 3 to state 2 is just like competing exponentials, the

rates are summed up. The rest is easy.

ii. Find the limiting probabilities.

Lets write the equations for limiting probabilities.

Balance eq for state 0 :

(λ/2)π0 + (λ/2)π0 = µ1 π1.1 + µ2 π1.2

Balance eq for state 1.1:

π1 (µ1 + λ) = µ2 π2 + (λ/2)π0

Balance eq for state 1.2:

π2 (µ2 + λ) = µ1 π2 + (λ/2)π0

Balance eq between state 2 and 3: (Dividing the state set to 2 parts with the ones that

are on the left of state 2 (including state 2) and the rest)

π2 (λ) = π3 (µ1 + µ2 )

Similar Balance eq between state 3 and 4:

π3 (λ) = π4 (µ1 + µ2 )

etc...

Well here, we have couple of equations to solve. You might try it yourself. But dont

forget the normalization equation which is π0 + π1.1 + π1.2 + π2 + π3 + .... = 1