Lecture: starts Tuesday 9:35am

Course website: introml.odl.mit.edu

Who's talking? Prof. Iddo Drori
Questions? Piazza
Materials: Will be available on course website
Today’s Plan: State Machines and
Markov Decision Processes (MDPs)

• State machine
• Observation vs. State
• Markov decision process (MDP)
• Policy: value, optimal
• Finite horizon value iteration, optimal policy
• Return
• Bellman equations: expectation, optimality
• Infinite horizon value iteration
State Machine

• S = set of possible states = {standing, moving}

standing moving
State Machine

• S = set of possible states = {standing, moving}

• s0 ∈ S = initial state = standing

standing moving
State Machine

• S = set of possible states = {standing, moving}

slow
• " = set of possible inputs = {slow, fast}
• s0 ∈ S = initial state = standing
standing moving
fast

slow fast
State Machine

• S = set of possible states = {standing, moving}

slow
• " = set of possible inputs = {slow, fast}
• #: S x " ➞ S transition function
• s0 ∈ S = initial state = standing standing moving
fast

slow fast
State Machine

• S = set of possible states = {standing, moving}

slow
• " = set of possible inputs = {slow, fast}
• #: S x " ➞ S transition function
• s0 ∈ S = initial state = standing standing moving
fast
• s1 = #(s0, fast) = ?
slow fast
State Machine

• S = set of possible states = {standing, moving}

slow
• " = set of possible inputs = {slow, fast}
• #: S x " ➞ S transition function
• s0 ∈ S = initial state = standing standing moving
fast
• s1 = #(s0, fast) = moving
slow fast
State Machine

• S = set of possible states = {standing, moving}

slow
• " = set of possible inputs = {slow, fast}
• #: S x " ➞ S transition function
• s0 ∈ S = initial state = standing standing moving
fast
• s1 = #(s0, fast) = moving
slow fast

• May not observe states {standing, moving} directly

for example a sensor measurement
Example: Observation vs. State

observation O state S
State Machine

• S = set of possible states = {fallen, standing, moving}

slow
• " = set of possible inputs = {slow, fast}
• #: S x " ➞ S transition function
• s0 ∈ S = initial state = standing standing moving
fast
• s1 = #(s0, fast) = moving
slow fast
• Not observe states
• $: set of possible outputs
• %: S ➞ $
• $1 = g(s1) = moving
State Machine

• S = set of possible states = {fallen, standing, moving}

slow
• " = set of possible inputs = {slow, fast}
• #: S x " ➞ S transition function
• $: set of possible outputs standing moving

• %: S ➞ $ fast

• s0 ∈ S = initial state = standing slow fast

• Iteratively computer for t ⥸ 1:

State Machine Example

• Reads binary string

• String has even number of zeros iff ends at state S1

State and Reward

?
Markov Model
Markov Process
Markov Process
Markov Process
Markov Process
Markov Process

• set of possible states S = {fallen, standing, moving}

• set of possible actions A = {slow, fast} p=1

• transition model T: S x A x S ➞ ℝ p=1

p=3/5
• transition function is stochastic standing moving

defines probability distribution p=3/4

over next state given

p=2/5 p=2/5 p=1/5
previous state and action
• Output is state (g is identity)
fallen
p=3/5 p=1
Transition P(s,a,s’)
fallen standing moving s = {standing, moving, fallen} a = {slow, fast} probability

p=1
fallen
p=1
standing p=3/5
standing moving
moving
p=3/4

p=2/5 p=2/5 p=1/5

fallen
p=3/5 p=1

21
Transition P(s,a,s’)
fallen standing moving s = {standing, moving, fallen} a = {slow, fast} probability

p=1
fallen
p=1
standing p=3/5
standing moving
moving
p=3/4

fallen standing moving

p=2/5 p=2/5 p=1/5
fallen

standing

fallen
moving p=3/5 p=1

22
Markov Decision Process (MDP)

• set of possible states S = {fallen, standing, moving}

• set of possible actions A = {slow, fast} p=1

• transition model T: S x A x S ➞ ℝ p=1

• reward function R: S x A ➞ ℝ standing p=3/5
moving

reward based on state & action p=3/4

R(fallen, slow) = 1
R(fallen, fast) = 0 p=2/5 p=2/5 p=1/5
R(standing, slow) = 1
R(standing, fast) = 2
R(moving, slow) = 1 fallen
R(moving, fast) = -1 p=3/5 p=1
Markov Decision Process (MDP)

• S = set of possible states {fallen, standing, moving}

• A = set of possible actions {slow, fast} p=1

• T: S x A x S ➞ ℝ transition model p=1

• R: S x A ➞ ℝ reward function standing p=3/5
moving

• ( discount factor p=3/4

p=2/5 p=2/5 p=1/5

fallen
p=3/5 p=1
Markov Decision Process (MDP)

• set of possible states S = {fallen, standing, moving}

• set of possible actions A = {slow, fast} p=1 r=1
• transition model T: S x A x S ➞ ℝ p=1 r=1
• rewards ℝ may be probabilistic standing p=3/5 r=2 moving

with transition function p=3/4 r=2

• ( discount factor
p=2/5 r=1 p=2/5 r=-1 p=1/5 r=-1

fallen
p=3/5 r=-1 p=1 r=0
Reward R(s,a)
slow fast s = {standing, moving, fallen} a = {slow, fast} probability
p=1 r=1 reward
fallen
p=1 r=1
p=3/5 r=2
standing standing moving
p=3/4 r=2
moving

p=2/5 r=1 p=2/5 r=-1 p=1/5 r=-1

horizon 1 fallen
optimal myopic policy
p=3/5 r=-1 p=1 r=0

26
Markov Decision Process (MDP)
• Defined by tuple (S, A, T, R, ))
state space S, action space A, transition function T, reward
function R, discount factor )

• At every time step t agent finds itself in state s ∈ S and

selects action a ∈ A
Agent transitions to next state s’ and selects new action.

• In contrast, in reinforcement learning the agent does not

know T and R, learns by sampling environment
27
Policy

• *: S ➞ A
p=1
• Rule book:
p=1
what action to take in each state? standing
p=3/5
moving
• in state s ∈ {fallen, standing, moving} p=3/4

take action a ∈ {slow, fast}

p=2/5 p=2/5 p=1/5

fallen
p=3/5 p=1
Policy
whichpolicy is best
• *: S ➞ A
p=1
• Example robot policies:
p=1
– *A: always slow standing
p=3/5
moving
p=3/4

– *B: always fast

p=2/5 p=2/5 p=1/5

– *C: if fallen slow, else fast

fallen
p=3/5 p=1
– *D: if moving fast, else slow
Stochastic Policy

• *: S ➞ A
p=1
• Policy may be stochastic
p=1
randomness in agent actions standing
p=3/5
moving

for example p=3/4

– *E: from all states

p=2/5 p=2/5 p=1/5
with probability 0.3 do slow
with probability 0.7 do fast
fallen
p=3/5
p=1
In addition to transitions being stochastic.
State-Action Diagram

• In state S
• Take action a by policy *: S ➞ A
• Transition to state s’ s

Repeat take action

• Tree of (s,a,r,s’)
s, a

where the transition function

will take us

s’
State-Action Diagram

• In state S
• Take action a by policy *: S ➞ A s

• Transition to state s’ take action

Repeat

take action
What is the Value of a Policy

• *: S ➞ A
• What is the value of a policy? p=1 r=1

p=1 r=1
• Depends on number of steps p=3/5 r=2
standing moving
• Renting robot for h time steps p=3/4 r=2

then will be destroyed

p=2/5 r=1 p=2/5 r=-1 p=1/5 r=-1

fallen
p=3/5 r=-1 p=1 r=0
What is the Value of a Policy?

• *: S ➞ A
• +: horizon, number of time steps left p=1 r=1

p=1 r=1
• : value (expected reward) p=3/5 r=2
standing moving
with policy * starting at s p=3/4 r=2

• Example robot policies

– *A: always slow p=2/5 r=1 p=2/5 r=-1 p=1/5 r=-1

– *B: always fast

– *C: if fallen slow, else fast fallen
p=1 r=0
– *D: if moving fast, else slow p=3/5 r=-1
What is the Value of a Policy?

• *: S ➞ A
• +: horizon, number of time steps left
• : value (expected reward) with policy * starting at s
• By induction on the number of steps left to go h.

• Base case: no steps remaining, no matter what state

we’re in the value is:
What is the Value of a Policy?

• *: S ➞ A
• +: horizon, number of time steps left
• : value (expected reward) with policy * starting at s
• Value of policy at state s at horizon h is
reward in s + next state’s expected horizon h-1 value.
• For h = 1
What is the Value of a Policy?

• *: S ➞ A
• +: horizon, number of time steps left
• : value (expected reward) with policy * starting at s
• Value of policy at state s at horizon h is
reward in s + next state’s expected horizon h-1 value.
• For h = 2
What is the Value of a Policy?

• *: S ➞ A
• +: horizon, number of time steps left
• : value (expected reward) with policy * starting at s
• Value of policy at state s at horizon h is
reward in s + next state’s expected horizon h-1 value.
• For any h
What is the Value of a Policy?
Updating State Value Function

40
Finite Horizon Value Iteration Algorithm

• Compute : start from horizon 0, store values

• Use to compute
• For n = |S|, m = |A|, horizon h, computation time O(nmh)
Updating Action Value Function

42
Finite Horizon Optimal Policy

• Given the optimal finite horizon policy is:

• May be multiple optimal policies

State

44
Action

45
Policy

46
Policy

47
Return and Discount Factor

(0,1)

if ! = 0 then agent is myopic maximizing only immediate rewards

as ! → 1 agent becomes foresighted
48
State Value Function

49
Action-Value Function

50
Value Functions

51
Value Functions

• For all states s :

52
Returns at Successive Time Steps

• Recursive relationship

53
4 Bellman Equations

• Expectation for state value

linear
• Expectation for action value

• Optimality for state value function

non linear
• Optimality for action value function

54
Infinite Horizon

• Don’t know when game will be over.

• Problem: may be infinite, cannot select one over other
• Solution: find policy that maximizes infinite horizon
discounted value

• t denotes number of steps from starting state

Policy Evaluation

• Expected infinite horizon value of state s under policy ,

• t denotes number of steps from starting state

• n = |S| linear equations

system of linear equations
State-Value Function for Policy
• Expected return starting in s and following policy ! satisfies recursive relationship:

Bellman equation for V!

• Relationship between value of state and values of successors
57
 Bellman Equation for State-Value Function
• V! is unique solution to its Bellman equation s
Linear equation take action average
Vector notation
s, a
• Value of start state s
is discounted value of expected r

next state + expected reward where the transition function

will take us

s’
• Bellman equation averages over all possibilities
weighing each by its probability to occur
58
Bellman Equation for Action-Value Function

linear equation

where the transition

function will take us

what action we will take

59
Finding Optimal Policy

• In infinite horizon there exists a stationary optimal policy ,*

(at least one) such that for all s ∈ S and all other policies ,

• Stationary: does not change over time

Infinite Horizon Value Iteration

• expected infinite horizon discounted value of being in

state s, taking action a, then executing optimal policy ,*

• n = |S|, m = |A|, nm non-linear equations with unique solution

value
Bellman Optimality Equation for Q*

non linear equation

expectation over where the transition

function will take us

maximize over the actions we can take

max max

once we have Q* we can act optimally

62
Finding Optimal Policy

• If we know optimal action-value function then

derive optimal policy:

• Optimal policy is not unique

Infinite Horizon Value Iteration Algorithm

Beltran
I