Introduction

The Difference Method

The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

Chapter 3 Difference in Differences and

Policy Evaluation

Advanced Financial Empirical Research

Lu Liu

1/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

Introduction

I Financial markets and market participants are subject to

government policies, rules and regulations.
I Policy makers need to evaluate the effect of the policies to
improve the implementation process.

2/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

Introduction

Banking concerns a lot with government policies and regulations.

I What is the impact of (de)regulation of the banking industry
on market structure, bank conduct, bank strategy, financial
stability and development, etc?
I The recent decade has witnessed governments’ capital
injection into banking industry to tackle the financial crisis. Is
the primary purpose of the intervention fulfilled? Are there
any side-effects?

3/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

Introduction

Market microstructure
I How do regulatory changes in the market design (e.g. changes
in tick size, introduction of midday auction, MiFID II by
ESMA) shift the nature of trading, the behaviours of market
participants, market quality, etc.?

4/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

Introduction

We need to choose appropriate evaluation methods

I The difference approach: a simple method.
I The difference-in-differences (DD) approach: better
identifying the impact of a policy program that is not
attributable to something else.
I The difference-in-differences-in-differences (DDD)

5/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

The Difference Method

Provided with data on a bunch of entities right before the policy is

enacted and on the same group of entities after it is enacted.
I The policy is enacted at time t = τ .
I We observe the outcome variable y before and after the policy.
I Identify the effect simply as E[y |t ≥ τ ] − E[y |t < τ ].

6/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

The Difference Method

Formally present this method with a panel data setup

yit = α + βTt + uit , (1)

(
0 if t < τ
Tt = ,
1 if t ≥ τ
The expected value of y before and after the policy, respectively,
are
E[y |t < τ ] = α + β × 0 = α (2)
E[y |t ≥ τ ] = α + β × 1 = α + β. (3)
Taking the difference between the two expected values, we get the
effect of the policy, β. The model in (1) can be estimated using
OLS.
7/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

The Difference Method

An example about the effect of the bank deregulation on
bank performance and local economic growth.
I In the US, intrastate branching regulations imposed by state
legislatures used to restrict a bank from making state-wide
branching expansions, and a bank holding company from
folding its subsidiaries in different counties into a single
operation entity. Beginning in the mid-1970s, individual states
started to lift these restrictions.
I The dependent variable yit is the performance (measured as
profitability, operating costs, loan losses, etc.) of bank i at
year t.
I Tit is a dummy variable with a value of one if state i is
deregulated at t.
8/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

Drawback
The problem with the difference method is that it attributes any
changes in time to the policy.

9/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

The Difference-in-Differences Method

The difference-in-differences (DD) method removes the time trend

that is not attributed to the policy.
Compare the before and after difference between the treated group
and the control group .
I The treated group: the group that the policy is enacted on.
I The control or untreated group: the group that are not
affected by the policy.

10/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

The Difference-in-Differences Method

An important assumption of DD: the treated group would follow

the same time trend as the control group if the treated group had
not been treated (i.e. affected by the policy).
I The difference before and after the policy enactment for the
control group can be used to pick up the time change of the
treated group.
I Correct the difference before and after for the treatment
group by subtracting the difference before and after for the
control group.
The name of the method comes from taking the difference in the
before and after differences.

11/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

The Difference-in-Differences Method

Figure: The difference estimator

12/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

The Difference-in-Differences Method

Figure: The DD estimator

13/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

The Difference-in-Differences Method

The formal formulation of the DD estimate

   
β̂ = E[y T |t ≥ τ ] − E[y T |t < τ ] − E[y C |t ≥ τ ] − E[y C |t < τ ]
(4)

where y T is the outcome of the treated group, and y C is the

outcome of the control group.

14/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

The Difference-in-Differences Method

DD estimation in a regression

yit = α + λDi + δTt + βDi Tt + uit , (5)

(
0 if i is in the control group
Di =
1 if i is in the treated group
(
0 if t < τ
Tt =
1 if t ≥ τ

15/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

The Difference-in-Differences Method

The OLS estimate of β in (5) is equivalent to that given in (4).
According to (5):
E[y T |t ≥ τ ] = α + λ + δ + β (6)
T
E[y |t < τ ] = α + λ
E[y C |t ≥ τ ] = α + δ
E[y C |t < τ ] = α
   
E[y T |t ≥ τ ] − E[y T |t < τ ] − E[y C |t ≥ τ ] − E[y C |t < τ ] = β

I λ = E[y T |t < τ ] − E[y C |t < τ ]: reflecting the difference

between the treated and the control groups before τ .
I δ = E[y C |t ≥ τ ] − E[y C |t < τ ]: common time change.
I β: treatment effect.
16/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

The Difference-in-Differences Method

Example revisited: the effect of the bank deregulation on local

economic growth.
I State A relaxed restrictions on intrastate branching, whereas
the adjacent state B did not.
I A DD strategy:
I treated group: banks in state A
I non-tread group: banks in state B

17/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

The Difference-in-Differences Method

Extensions
I Di and Tt on the RHS of eq(5) control for heterogeneity
between groups and between before and after.
I A natural extension to the DD model is including both
individual- and time- fixed effects. The fixed effects control
for state specific and time specific heterogeneity, which
encompass heterogenity between groups and between before
and after.
I Add other variables Xit .
The DD model with fixed effects is

yit = βDi Tt + Xit θ + µi + κt + uit (7)

18/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

The Difference-in-Differences Method

Extensions Allow the policy to enact on individuals in the

treatment group at different t.

yit = βTit + µi + κt + uit (8)

where Tit equals zero if the policy does not enact on i at t and
one otherwise. Tit is identical to Di Tt if policy enacts on all the
individuals of the treated group at the same time.

19/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

The Difference-in-Differences Method

Example: the effect of the bank deregulation (Jayaratne and
Strahan, 1998)

Perfit = β1 Branchit + β2 Bankit + ηXit + κt + µi + uit , (9)

I Perfit is a bank performance measure in state i at t.

I Branchit equals one if state i does not impose restrictions on
branching via M&A at t and zero otherwise.
I Bankit equals one if state i is in an interstate banking
agreement at t and zero otherwise.
I β̂1 (β̂2 ) is the DD estimator for the effect of removing
restrictions on branching (having interstate banking
agreement).
20/38
 Introduction
The Difference Method Graphical analysis
The Difference-in-Differences Method Placebo test
Checking Internal Validity Matching
Example: Troubled Asset Relief Program (TARP) Diff-in-Diff-in-Diff
References

Checking Internal Validity

I Internal validity is the extent to which you can be confident
that the causal relationship established in a study is not
explained by other factors.
I Internal validity in DD fails if the control group is not a valid
estimate of the counter-factual for the treatment group.
I Confounding factors: factors that correlate with the treatment
and drive the outcome variable.
I Selection bias: if Individuals are assigned to the treated and
the control group due to differences in confounding factors
prior to policy enactment.
I As a result, DD estimator will reflect differences in the
confounding factors of the treated and the control group
rather than the effect of the policy per se.

21/38
 Introduction
The Difference Method Graphical analysis
The Difference-in-Differences Method Placebo test
Checking Internal Validity Matching
Example: Troubled Asset Relief Program (TARP) Diff-in-Diff-in-Diff
References

Example of selection bias

Kroszner and Strahan (1999) find that the relative strength of

potential winners (large banks and small, bank-dependent firms)
and losers (small banks and the rival insurance firms) in bank
deregulation can explain the timing of branching deregulation
across states.
Therefore, the DD estimator of bank deregulation may reflect the
winner-loser difference rather than the effect of bank deregulation
per se.

22/38
 Introduction
The Difference Method Graphical analysis
The Difference-in-Differences Method Placebo test
Checking Internal Validity Matching
Example: Troubled Asset Relief Program (TARP) Diff-in-Diff-in-Diff
References

Graphical analysis
Trend plot of the outcome variable across the treatment and
control groups during pre- and post- treatment periods.

Figure: Trend plot

23/38
 Introduction
The Difference Method Graphical analysis
The Difference-in-Differences Method Placebo test
Checking Internal Validity Matching
Example: Troubled Asset Relief Program (TARP) Diff-in-Diff-in-Diff
References

Placebo test
Use data in the pre-treatment era and perform DD with a placebo
rather than the actual treatment.

Figure: Placebo test

24/38
 Introduction
The Difference Method Graphical analysis
The Difference-in-Differences Method Placebo test
Checking Internal Validity Matching
Example: Troubled Asset Relief Program (TARP) Diff-in-Diff-in-Diff
References

Matching

For every treated individual, find one (or more) control

individual(s) with similar observable characteristics against
whom the effect of the treatment can be assessed. By matching
treated individuals to similar non-treated individuals based on
observables, we reduce the potential selection bias.
I The first generation of matching methods: pair the treated
with the controls based on either a single variable or the
weighted average several variables. However, when there are
many variables, it is difficult to determine along which
dimensions to match units or which weighting scheme to
adopt.

25/38
 Introduction
The Difference Method Graphical analysis
The Difference-in-Differences Method Placebo test
Checking Internal Validity Matching
Example: Troubled Asset Relief Program (TARP) Diff-in-Diff-in-Diff
References

Matching

I Propensity score matching: a widely used matching

technique based on observables.
I A propensity score is the probability of a individual being
assigned to a policy program given a set of observed covariates.
I The first step: estimate the propensity score using a
logit/probit model.
I The second step: match a treated individual to one (or) more
control individual(s) whose propensity score is the closest to
the treated individual’s.

26/38
 Introduction
The Difference Method Graphical analysis
The Difference-in-Differences Method Placebo test
Checking Internal Validity Matching
Example: Troubled Asset Relief Program (TARP) Diff-in-Diff-in-Diff
References

Matching
I Unobserved confounding variables
I Example: geographic-matching method for examining the
effect of intrastate branching restriction on local economic
growth (Huang, 2008)
I ”Banking deregulation could be induced by an expectation of
future growth opportunities (unobservable to
econometricians), which could create a spurious correlation
between banking deregulation and future growth
accelerations.”
I The geographic-matching method compares economic
performance of contiguous counties on opposite side of
state borders, where only one state experienced the
deregulation. This procedure improves the control group in
both observable and unobservable characteristics as
contiguous countries are less likely to differ from each other.

27/38
 Introduction
The Difference Method Graphical analysis
The Difference-in-Differences Method Placebo test
Checking Internal Validity Matching
Example: Troubled Asset Relief Program (TARP) Diff-in-Diff-in-Diff
References

Difference-in-Difference-in-Differences

A state implements a program that aims at regulating big banks’

risk taking behavior.
I The DD strategy uses big banks in the program-state as the
treated group and big banks in non-program states as the
control group.
 
β̂BDD = E[yBP |t ≥ τ ] − E[yBP |t < τ ]
 
− E[yBNP |t ≥ τ ] − E[yBNP |t < τ ]

28/38
 Introduction
The Difference Method Graphical analysis
The Difference-in-Differences Method Placebo test
Checking Internal Validity Matching
Example: Troubled Asset Relief Program (TARP) Diff-in-Diff-in-Diff
References

Difference-in-Difference-in-Differences

I β̂BDD adjusts for general trends affecting all big banks.

I But it does not take account of non-program factors that
affect banks in different states differentially. E.g.,
different housing price across states may induce different risk
taking behaviors.

29/38
 Introduction
The Difference Method Graphical analysis
The Difference-in-Differences Method Placebo test
Checking Internal Validity Matching
Example: Troubled Asset Relief Program (TARP) Diff-in-Diff-in-Diff
References

Difference-in-Difference-in-Differences
I How can we adjust for the non-program factors that affect
banks in different states differentially?
I Small banks are not affected by the program but by the
non-program factors.
I The DD estimate for small banks in the program state and
the non-program state provides an estimate of the
non-program factors that affect states differentially.
 
β̂SDD = E[ySP |t ≥ τ ] − E[ySP |t < τ ]
 
− E[ySNP |t ≥ τ ] − E[ySNP |t < τ ]

30/38
 Introduction
The Difference Method Graphical analysis
The Difference-in-Differences Method Placebo test
Checking Internal Validity Matching
Example: Troubled Asset Relief Program (TARP) Diff-in-Diff-in-Diff
References

Difference-in-Difference-in-Differences

The DDD estimate

β̂ DDD = β̂BDD − β̂SDD

The first difference adjusts for general trends affecting all big
banks. The second difference adjusts for non-treatment factors
affecting states differentially.

31/38
 Introduction
The Difference Method Graphical analysis
The Difference-in-Differences Method Placebo test
Checking Internal Validity Matching
Example: Troubled Asset Relief Program (TARP) Diff-in-Diff-in-Diff
References

Difference-in-Difference-in-Differences
Regression
I P: a dummy variable equal to one for the banks in the
program states and zero otherwise.
I T : a dummy variable equal to one for time after the
treatment and zero otherwise.
I B: a dummy variable equal to one for big banks and zero
otherwise.

yit =β0 + β1 Pi + β2 Bi + β3 Tt
+ β4 Pi Bi + β5 Pi Tt + β6 Bi Tt
+ β7 Pi Bi Tt + uit . (10)
The coefficient β7 on the trip interaction term Pi Bi Tt is the DDD
estimate.
32/38
 Introduction
The Difference Method Graphical analysis
The Difference-in-Differences Method Placebo test
Checking Internal Validity Matching
Example: Troubled Asset Relief Program (TARP) Diff-in-Diff-in-Diff
References

Difference-in-Difference-in-Differences

P,B
ȳt≥τ = β0 + β1 + β2 + β3 + β4 + β5 + β6 + β7
P,B
ȳt<τ = β0 + β1 + β2 + β4
NP,B
ȳt≥τ = β0 + β2 + β3 + β6
NP,B
ȳt<τ = β0 + β2
P,S
ȳt≥τ = β0 + β1 + β3 + β5
P,S
ȳt<τ = β0 + β1
NP,S
ȳt≥τ = β0 + β3
NP,S
ȳt<τ = β0

33/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

Example: Troubled Asset Relief Program (TARP) I

Troubled Asset Relief Program (TARP), one of the largest
intervention by the US government to tackle the subprime
mortgage crisis.
I The Capital Purchase Program (CPP) is the main component
of TARP. Rather than purchasing ”troubled assets”, the CPP
authorized the U.S. Treasury to invest up to $250 billion in
preferred equity of selected financial institutions to enhance
their capital ratios.
I Initial receipts of the investment were nine large involuntary
participants (Citigroup, Bank of America, JPMorgan
Chase,Wells Fargo, Goldman Sachs Group, Morgan Stanley,
Wachovia Corporation, State Street Corporation, and Merrill
Lynch) with $125 billion invested.
34/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

Example: Troubled Asset Relief Program (TARP) II

I The rest of the receipts followed the formal CPP process and
applied for CPP funds from the U.S. Treasury.
I Approval to receive TARP took into account the health of the
banking organizations, with the viable, healthier ones being
more likely to receive capital.
I During 2008:Q4-2009:Q4, TARP infused $204.9 billion into
709 banking organizations.
I The primary purposes of TARP were to improve financial
stability and to encourage banks to increase lending.
Economists have paid much attention on evaluating the
effects of TARP.

35/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

Example: Troubled Asset Relief Program (TARP) III

I Berger and Roman (2015): find that TARP recipients received
competitive advantages and increased both their market
shares and market power.

yit = λ · TARP_recipienti + β · TARP_recipienti × POST_TARPt

+ Xit θ + κt + uit (11)

yit : a competitive advantage indicator (market share or market

power).
I TARP_recipient: a dummy variable equal to 1 if the bank was
provided TARP capital support.
I POST_TARPt a dummy equal to 1 after the capital injection.
I Xit : control variables
I κt : time fixed effects.
36/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

Example: Troubled Asset Relief Program (TARP) IV

I Black and Hazelwood (2013): large TARP banks did not

increase lending but increased their risk of loan relative to
non-TARP peers.This is suggestive of moral hazard. Further,
TARP may also have had unintended effects on bank
competition and resource allocation.

37/38
 Introduction
The Difference Method
The Difference-in-Differences Method
Checking Internal Validity
Example: Troubled Asset Relief Program (TARP)
References

References
Berger, A. N. and Roman, R. A. (2015). Did TARP banks get
competitive advantages? Journal of Financial and Quantitative
Analysis, 50(06):1199–1236.
Black, L. K. and Hazelwood, L. N. (2013). The effect of TARP on
bank risk-taking. Journal of Financial Stability, 9(4):790–803.
Huang, R. R. (2008). Evaluating the real effect of bank branching
deregulation: Comparing contiguous counties across us state
borders. Journal of Financial Economics, 87(3):678–705.
Jayaratne, J. and Strahan, P. E. (1998). Entry restrictions, industry
evolution, and dynamic efficiency: Evidence from commercial
banking. The Journal of Law and Economics, 41(1):239–274.
Kroszner, R. S. and Strahan, P. E. (1999). What drives
deregulation? economics and politics of the relaxation of bank
branching restrictions. The Quarterly Journal of Economics,
114(4):1437–1467. 38/38