Difference-in-Differences

Difference-in-Differences Model

Gumilang Aryo Sahadewo

Universitas Gadjah Mada

September 27, 2020

Difference-in-Differences

Motivation

1 Difference-in-Differences
Motivation
A Case Study: Kiel and McClain (1995)
Difference-in-differences estimator
Issues with DiD Estimation
Issues with the Estimation of Standard Errors
Triple Differences Model
References
Difference-in-Differences

Motivation

Motivation

One of the challenges in microeconometrics is to obtain a

valid counterfactual.
It is usually difficult to obtain a valid counterfactual using
observational data such as national surveys.
Treated units are more likely to have different characteristics
than non-treated units
One way of getting around this is to use pre-treatment
outcomes.
Difference-in-differences is a strategy to model the role of
pre-treatment outcomes (Pischke, 2005)
Difference-in-Differences

Motivation

Motivation

One of the challenges in microeconometrics is to obtain a

valid counterfactual.
It is usually difficult to obtain a valid counterfactual using
observational data such as national surveys.
Treated units are more likely to have different characteristics
than non-treated units
One way of getting around this is to use pre-treatment
outcomes.
Difference-in-differences is a strategy to model the role of
pre-treatment outcomes (Pischke, 2005)
Difference-in-Differences

Motivation

Motivation

One of the challenges in microeconometrics is to obtain a

valid counterfactual.
It is usually difficult to obtain a valid counterfactual using
observational data such as national surveys.
Treated units are more likely to have different characteristics
than non-treated units
One way of getting around this is to use pre-treatment
outcomes.
Difference-in-differences is a strategy to model the role of
pre-treatment outcomes (Pischke, 2005)
Difference-in-Differences

Motivation

Motivation

One of the challenges in microeconometrics is to obtain a

valid counterfactual.
It is usually difficult to obtain a valid counterfactual using
observational data such as national surveys.
Treated units are more likely to have different characteristics
than non-treated units
One way of getting around this is to use pre-treatment
outcomes.
Difference-in-differences is a strategy to model the role of
pre-treatment outcomes (Pischke, 2005)
Difference-in-Differences

Motivation

Motivation

One of the challenges in microeconometrics is to obtain a

valid counterfactual.
It is usually difficult to obtain a valid counterfactual using
observational data such as national surveys.
Treated units are more likely to have different characteristics
than non-treated units
One way of getting around this is to use pre-treatment
outcomes.
Difference-in-differences is a strategy to model the role of
pre-treatment outcomes (Pischke, 2005)
Difference-in-Differences

Motivation

Data Structure

Data structure:
Two waves of cross section data or
Panel data
Cross-sectional units: i ∈ {1, . . . , N}
Time periods: Post ∈ {0 (pre-treatment) , 1 (post-treatment)}
(
1 treatment group
Group indicator: Treat =
0 control group
Difference-in-Differences

Motivation

Data Structure

Data structure:
Two waves of cross section data or
Panel data
Cross-sectional units: i ∈ {1, . . . , N}
Time periods: Post ∈ {0 (pre-treatment) , 1 (post-treatment)}
(
1 treatment group
Group indicator: Treat =
0 control group
Difference-in-Differences

Motivation

Data Structure

Data structure:
Two waves of cross section data or
Panel data
Cross-sectional units: i ∈ {1, . . . , N}
Time periods: Post ∈ {0 (pre-treatment) , 1 (post-treatment)}
(
1 treatment group
Group indicator: Treat =
0 control group
Difference-in-Differences

Motivation

Data Structure

Data structure:
Two waves of cross section data or
Panel data
Cross-sectional units: i ∈ {1, . . . , N}
Time periods: Post ∈ {0 (pre-treatment) , 1 (post-treatment)}
(
1 treatment group
Group indicator: Treat =
0 control group
Difference-in-Differences

Motivation

Data Structure

Data structure:
Two waves of cross section data or
Panel data
Cross-sectional units: i ∈ {1, . . . , N}
Time periods: Post ∈ {0 (pre-treatment) , 1 (post-treatment)}
(
1 treatment group
Group indicator: Treat =
0 control group
Difference-in-Differences

Motivation

Data Structure

Data structure:
Two waves of cross section data or
Panel data
Cross-sectional units: i ∈ {1, . . . , N}
Time periods: Post ∈ {0 (pre-treatment) , 1 (post-treatment)}
(
1 treatment group
Group indicator: Treat =
0 control group
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

1 Difference-in-Differences
Motivation
A Case Study: Kiel and McClain (1995)
Difference-in-differences estimator
Issues with DiD Estimation
Issues with the Estimation of Standard Errors
Triple Differences Model
References
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Case study

What is the effect of the location of a garbage incinerator on

house prices?
Kiel and McClain (1995)
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Case study

What is the effect of the location of a garbage incinerator on

house prices?
Kiel and McClain (1995)
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Illustration of a garbage incinerator

Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Case study

What is the effect of the location of a garbage incinerator on

house prices?
Kiel and McClain (1995)
Hypothesis: having an incinerator nearby lowers the price of a
house.
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Case study

What is the effect of the location of a garbage incinerator on

house prices?
Kiel and McClain (1995)
Hypothesis: having an incinerator nearby lowers the price of a
house.
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Case study

What is the effect of the location of a garbage incinerator on

house prices?
Kiel and McClain (1995)
Hypothesis: having an incinerator nearby lowers the price of a
house.
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Data Structure

Data: Prices and characteristics of houses in different

distances to the incinerator.
Two cross-sections data:
1978: pre-treatment
1981: post-treatment
Two groups based on house i distance to the incinerator:
(
1 if i is near from the incinerator
nearinci =
0 if i is far from the incinerator
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Data Structure

Data: Prices and characteristics of houses in different

distances to the incinerator.
Two cross-sections data:
1978: pre-treatment
1981: post-treatment
Two groups based on house i distance to the incinerator:
(
1 if i is near from the incinerator
nearinci =
0 if i is far from the incinerator
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Data Structure

Data: Prices and characteristics of houses in different

distances to the incinerator.
Two cross-sections data:
1978: pre-treatment
1981: post-treatment
Two groups based on house i distance to the incinerator:
(
1 if i is near from the incinerator
nearinci =
0 if i is far from the incinerator
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Data Structure

Data: Prices and characteristics of houses in different

distances to the incinerator.
Two cross-sections data:
1978: pre-treatment
1981: post-treatment
Two groups based on house i distance to the incinerator:
(
1 if i is near from the incinerator
nearinci =
0 if i is far from the incinerator
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Data Structure

Data: Prices and characteristics of houses in different

distances to the incinerator.
Two cross-sections data:
1978: pre-treatment
1981: post-treatment
Two groups based on house i distance to the incinerator:
(
1 if i is near from the incinerator
nearinci =
0 if i is far from the incinerator
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Data Structure

Data: Prices and characteristics of houses in different

distances to the incinerator.
Two cross-sections data:
1978: pre-treatment
1981: post-treatment
Two groups based on house i distance to the incinerator:
(
1 if i is near from the incinerator
nearinci =
0 if i is far from the incinerator
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Case study

A naive approach: use 1981 cross section to estimate the

model:
lpricei = γ0 + γ1 nearinci + ui
price is the price of a house, nearinc is a dummy variable that
takes the value 1 if the house is located near the incinerator.
Is this a good estimate of the causal effect on house prices of
locating the incinerator nearby?
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Case study

A naive approach: use 1981 cross section to estimate the

model:
lpricei = γ0 + γ1 nearinci + ui
price is the price of a house, nearinc is a dummy variable that
takes the value 1 if the house is located near the incinerator.
Is this a good estimate of the causal effect on house prices of
locating the incinerator nearby?
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Case study

A naive approach: use 1981 cross section to estimate the

model:
lpricei = γ0 + γ1 nearinci + ui
price is the price of a house, nearinc is a dummy variable that
takes the value 1 if the house is located near the incinerator.
Is this a good estimate of the causal effect on house prices of
locating the incinerator nearby?
Difference-in-Differences

A Case Study: Kiel and McClain (1995)

Policy analysis with pooled cross sections

No! Why?
Difference-in-Differences

Difference-in-differences estimator

1 Difference-in-Differences
Motivation
A Case Study: Kiel and McClain (1995)
Difference-in-differences estimator
Issues with DiD Estimation
Issues with the Estimation of Standard Errors
Triple Differences Model
References
Difference-in-Differences

Difference-in-differences estimator

Policy analysis with pooled cross sections

House prices have gone up between 1978 and 1981 for most
houses.
Whether nearby and far away from the location of the
incinerator.
Let’s see this in Stata
Difference-in-Differences

Difference-in-differences estimator

Policy analysis with pooled cross sections

House prices have gone up between 1978 and 1981 for most
houses.
Whether nearby and far away from the location of the
incinerator.
Let’s see this in Stata
Difference-in-Differences

Difference-in-differences estimator

Policy analysis with pooled cross sections

House prices have gone up between 1978 and 1981 for most
houses.
Whether nearby and far away from the location of the
incinerator.
Let’s see this in Stata
Difference-in-Differences

Difference-in-differences estimator

Policy analysis with pooled cross sections

Relevant question: Has the change been bigger for houses far
from the incinerator?
Need to look at differences in space (nearby/far away) of
differences in time (between 1978 and 1981): Diff-in-diff
(DiD)
Difference-in-Differences

Difference-in-differences estimator

Policy analysis with pooled cross sections

Relevant question: Has the change been bigger for houses far
from the incinerator?
Need to look at differences in space (nearby/far away) of
differences in time (between 1978 and 1981): Diff-in-diff
(DiD)
Difference-in-Differences

Difference-in-differences estimator

Difference-in-differences approach

The difference-in-differences method compares the changes in

outcomes over time between:
a population that is enrolled in a program (the treatment
group) and;
a population that is not (the comparison group)
Why do we need to compare the changes over time?
Why not just compare post-treatment outcomes of the two
groups?
Difference-in-Differences

Difference-in-differences estimator

Difference-in-differences approach

The difference-in-differences method compares the changes in

outcomes over time between:
a population that is enrolled in a program (the treatment
group) and;
a population that is not (the comparison group)
Why do we need to compare the changes over time?
Why not just compare post-treatment outcomes of the two
groups?
Difference-in-Differences

Difference-in-differences estimator

Difference-in-differences approach

The difference-in-differences method compares the changes in

outcomes over time between:
a population that is enrolled in a program (the treatment
group) and;
a population that is not (the comparison group)
Why do we need to compare the changes over time?
Why not just compare post-treatment outcomes of the two
groups?
Difference-in-Differences

Difference-in-differences estimator

Difference-in-differences approach

The difference-in-differences method compares the changes in

outcomes over time between:
a population that is enrolled in a program (the treatment
group) and;
a population that is not (the comparison group)
Why do we need to compare the changes over time?
Why not just compare post-treatment outcomes of the two
groups?
Difference-in-Differences

Difference-in-differences estimator

Difference-in-differences approach

The difference-in-differences method compares the changes in

outcomes over time between:
a population that is enrolled in a program (the treatment
group) and;
a population that is not (the comparison group)
Why do we need to compare the changes over time?
Why not just compare post-treatment outcomes of the two
groups?
Difference-in-Differences

Difference-in-differences estimator

Model setup for DID

Let Treat be the treatment group and Control the control

group. Let Post be a time period dummy equal to one for a
unit in the second time period. Let Pre be a time period
dummy equal to one for a unit in the first time period. Write:

yit = β0 + β1 Treati + δ0 Postt + δ1 Treats · Postt + uit

where y is the outcome of interest.

The mean value of u is zero.
Difference-in-Differences

Difference-in-differences estimator

Model setup for DID

Let Treat be the treatment group and Control the control

group. Let Post be a time period dummy equal to one for a
unit in the second time period. Let Pre be a time period
dummy equal to one for a unit in the first time period. Write:

yit = β0 + β1 Treati + δ0 Postt + δ1 Treats · Postt + uit

where y is the outcome of interest.

The mean value of u is zero.
Difference-in-Differences

Difference-in-differences estimator

Model setup for DID

Recall:

yit = β0 + β1 Treati + δ0 Postt + δ1 Treats · Postt + uit

See the graph:

Pre (1) Post (2) Post - Pre

C β0 β0 + δ0 δ0
T β0 + β1 β0 + δ0 + β1 + δ1 δ0 + δ1
T-C β1 β1 + δ1 δ1
Difference-in-Differences

Difference-in-differences estimator

Model setup for DID

Recall:

yit = β0 + β1 Treati + δ0 Postt + δ1 Treats · Postt + uit

See the graph:

Pre (1) Post (2) Post - Pre

C β0 β0 + δ0 δ0
T β0 + β1 β0 + δ0 + β1 + δ1 δ0 + δ1
T-C β1 β1 + δ1 δ1
Difference-in-Differences

Difference-in-differences estimator

Model setup for DID

Recall:

yit = β0 + β1 Treati + δ0 Postt + δ1 Treats · Postt + uit

See the graph:

Pre (1) Post (2) Post - Pre

C β0 β0 + δ0 δ0
T β0 + β1 β0 + δ0 + β1 + δ1 δ0 + δ1
T-C β1 β1 + δ1 δ1
Difference-in-Differences

Difference-in-differences estimator

Model setup for DID

Recall:

yit = β0 + β1 Treati + δ0 Postt + δ1 Treats · Postt + uit

See the graph:

Pre (1) Post (2) Post - Pre

C β0 β0 + δ0 δ0
T β0 + β1 β0 + δ0 + β1 + δ1 δ0 + δ1
T-C β1 β1 + δ1 δ1
Difference-in-Differences

Difference-in-differences estimator

Intuition DiD

The first difference:

before-and-after outcomes for the treated group
control for characteristics that do not vary over time within a
group
The second difference:
control for time-varying factors that affect both groups
use before-and-after outcomes for the control group to “clean”
the first difference.
Difference-in-Differences

Difference-in-differences estimator

What if we just compare the price of the treated group

before and after?

Recall:

yit = β0 + β1 Treati + δ0 Postt + δ1 Treats · Postt + uit

See the graph:

Pre (1) Post (2) Post - Pre

C β0 β0 + δ0 δ0
T β0 + β1 β0 + δ0 + β1 + δ1 δ0 + δ1
T-C β1 β1 + δ1 δ1
Difference-in-Differences

Difference-in-differences estimator

What if we just compare the price of the treated and the

control group post intervention?

Recall:

yit = β0 + β1 Treati + δ0 Postt + δ1 Treats · Postt + uit

Or:

yit = β0 + β1 Treati + δ0 Postt + δ1 Treats · Postt + ΓXit + uit

See the graph:

Pre (1) Post (2) Post - Pre

C β0 β0 + δ0 δ0
T β0 + β1 β0 + δ0 + β1 + δ1 δ0 + δ1
T-C β1 β1 + δ1 δ1
Difference-in-Differences

Difference-in-differences estimator

What if we just compare the price of the treated and the

control group post intervention?

Recall:

yit = β0 + β1 Treati + δ0 Postt + δ1 Treats · Postt + uit

Or:

yit = β0 + β1 Treati + δ0 Postt + δ1 Treats · Postt + ΓXit + uit

See the graph:

Pre (1) Post (2) Post - Pre

C β0 β0 + δ0 δ0
T β0 + β1 β0 + δ0 + β1 + δ1 δ0 + δ1
T-C β1 β1 + δ1 δ1
Difference-in-Differences

Difference-in-differences estimator

Graphical illustration: if there were no treatment, the

outcome would have been...

Outcome of Interest

Comparison group

Treatment group

t-2 t-1 t t+1 t+2 Time

Difference-in-Differences

Difference-in-differences estimator

Graphical illustration: but with treatment....

Outcome of Interest

Comparison group

Treatment group

t-2 t-1 t t+1 t+2 Time

Difference-in-Differences

Difference-in-differences estimator

Back to the analysis of the “incinerator effect”

Treated group:
Control group:
How should we write the model:

lpriceit = β0 + δ0 y 81t + β1 nearinci + δ1 y 81t · nearinci + uit

What is the hypothesis?

This framework mimic controlled experiments in science by
finding something that happened naturally to one group of
people, but not to another: quasi-experiments.
This is a tool for an impact evaluation
What is the counterfactual?
Difference-in-Differences

Difference-in-differences estimator

Should we include control variables?

Yes, we should include control variables if possible:

lpriceit = β0 +δ0 y 81t +β1 nearinci +δ1 y 81t ·nearinci +γXit +uit

In this case, we can include age of the house, square of age,

distance to interstate (in log), land (in log), area (in log) in
the specification model
Difference-in-Differences

Issues with DiD Estimation

1 Difference-in-Differences
Motivation
A Case Study: Kiel and McClain (1995)
Difference-in-differences estimator
Issues with DiD Estimation
Issues with the Estimation of Standard Errors
Triple Differences Model
References
Difference-in-Differences

Issues with DiD Estimation

Parallel Trend Assumption

A potential problem with using only two periods is that the

control and treatment groups may be trending at different
rates having nothing to do with the intervention.
In the incinerator example, what if the prices of house nearby
the incinerator was decreasing faster than those far from the
incinerator?
In this case (non-parallel trend), out estimate is biased.
To solve this problem, we either get another control group or
more years of data (preferably at least two before the
intervention, to make sure there is a parallel trend prior to the
treatment ).
The seminal paper by Card and Krueger (1994) suffers from
this problem.
Difference-in-Differences

Issues with DiD Estimation

Parallel Trend Assumption

Outcome of Interest

Comparison group

Treatment group

t-2 t-1 t t+1 t+2 Time

Difference-in-Differences

Issues with DiD Estimation

Parallel Trend Assumption

Outcome of Interest

Comparison group

Estimated impact

Comparison group
Treatment group
trend

t-2 t-2 t-1 t t+1 t+2 Time

Difference-in-Differences

Issues with DiD Estimation

Assumptions in DID framework

Recall:

yit = β0 + β1 Treati + δ0 Postt + δ1 Treats · Postt + uit

To have unbiased estimators, we need to have the following

assumptions:
Cov (u, Treat) = 0
Cov (u, Post) = 0
Cov (u, treat · post) = 0
The last of these assumptions is also known as parallel-trend
assumption, is the most critical.
Difference-in-Differences

Issues with DiD Estimation

Paralell Trend Assumption

The parallel trend assumption is crucial. Why?

Recall the main problem of counterfactuals.
We cannot observe what would have happened to the
treatment group in the absence of the treatment.
We must assume that, in the absence of the program, the
outcomes in both treatment and comparison group must
move in parallel.
Difference-in-Differences

Issues with DiD Estimation

Is the estimated effects of the program unbiased?

Outcome of Interest

Comparison group

True impact

True (unobserved)
counterfactual

Comparison group
Treatment group
trend

t-2 t-2 t-1 t t+1 t+2 Time

Difference-in-Differences

Issues with the Estimation of Standard Errors

1 Difference-in-Differences
Motivation
A Case Study: Kiel and McClain (1995)
Difference-in-differences estimator
Issues with DiD Estimation
Issues with the Estimation of Standard Errors
Triple Differences Model
References
Difference-in-Differences

Issues with the Estimation of Standard Errors

Correlated standard errors

Due to Bertrand, Duflo, and Mullainathan (2006), “How

much should we trust difference-in-differences estimates?”
Studies using DiD uses many years of data.
Thus, within a “cluster”:
outcomes are correlated across years
error terms or unobservables are correlated across years as well
Estimation of standard errors should take into account this
correlation.
Difference-in-Differences

Triple Differences Model

1 Difference-in-Differences
Motivation
A Case Study: Kiel and McClain (1995)
Difference-in-differences estimator
Issues with DiD Estimation
Issues with the Estimation of Standard Errors
Triple Differences Model
References
Difference-in-Differences

Triple Differences Model

DDD Model

Suppose that a particular local government implements a

policy that affects a sub-population e.g. the elderly or only
the poors
How can we evaluate the effect of this policy?
Think of two possible DD analyses:
Alternative 1:
Alternative 2:
Difference-in-Differences

Triple Differences Model

DDD Model

Why not combine both?

We can include:
another local government that did not implement the policy
a treatment group within the treated local government
Difference-in-Differences

Triple Differences Model

DDD Model

Let T represents the local government that implements the

policy, C the other local government
Let G represents the group affected by the policy, NG the
group not affected
Let i be the group, in local government s, at time t:

yist = β0 + β1 Ts + β2 Gist + β3 Ts Gist +

δ0 Postst + δ1 Postst Ts + δ2 Postst Gist + δ3 Postst Ts Gist + uist

Difference-in-Differences

Triple Differences Model

DDD Model

The OLS estimate is:

δ̂3 = (ȳT ,G,2 − ȳT ,G,1 )−(ȳT ,NG,2 − ȳT ,NG,1 )−(ȳC ,G,2 − ȳC ,G,1 )

Focus on each component:

(ȳT ,G,2 − ȳT ,G,1 ) :
(ȳC ,G,2 − ȳC ,G,1 ) :
(ȳT ,NG,2 − ȳT ,NG,1 ) :
This estimate is called the DDD estimate
Difference-in-Differences

References

1 Difference-in-Differences
Motivation
A Case Study: Kiel and McClain (1995)
Difference-in-differences estimator
Issues with DiD Estimation
Issues with the Estimation of Standard Errors
Triple Differences Model
References
Difference-in-Differences

References

References

De Janvry, Alain, Frederico Finan, and Elisabeth Sadoulet.

2011. “Local Electoral Incentives and Decentralized Program
Performance.” Review of Economics and Statistics 94 (3):
672–85.
Duflo, Esther. 2001. “Schooling and Labor Market
Consequences of School Construction in Indonesia: Evidence
from an Unusual Policy Experiment.” American Economic
Review 91 (4): 795–813.
Galiani, Sebastian, Paul Gertler, and Ernesto Schargrodsky.
2005. “Water for Life: The Impact of the Privatization of
Water Services on Child Mortality.” Journal of Political
Economy 113 (1): 83–120.