International Journal of Data Science and Analysis

2017; 3(1): 5-12

http://www.sciencepublishinggroup.com/j/ijdsa
doi: 10.11648/j.ijdsa.20170301.12

Project Evaluation from Application to Econometric

Theory: A Qualitative Explanation of Difference in
Difference (DiD) Approach
Muhammad Ateeq-ur-Rehman1, 2
1
Gutenberg School of Management and Economics, Johannes Gutenberg University Mainz, Mainz, Germany
2
Lahore Business School, the University of Lahore, Lahore, Pakistan

Email address:
[email protected]

To cite this article:

Muhammad Ateeq-ur-Rehman, Project Evaluation from Application to Econometric Theory: A Qualitative Explanation of Difference in
Difference (DiD) Approach. International Journal of Data Science and Analysis. Vol. 3, No. 1, 2017, pp. 5-12.
doi: 10.11648/j.ijdsa.20170301.12

Received: September 1, 2016; Accepted: April 14, 2017; Published: April 26, 2017

Abstract: The main objective of this study is to show the importance of the Difference in Difference (DiD) method and its
applicability in the field of human and social sciences. The DiD method is one of the famous tools in econometrics to
investigate the causal effect of the policy before and after treatment or policy. Why difference in difference method is most
important in these days? Because the traditional methods requires more instructions as compare to DiD method which is easier
and applicable without randomization of the data. The difference is compared with treated and non-treated group in two time’s
period model with the same unit of data. The first difference removes the time-invariant factors while Difference in Difference
removes the time-variant factors of the model and the remaining statistic shows the original impact of the treatment or policy.
Keywords: Econometrics, Difference in Difference, Policy

1. Introduction
The main objective of the development policies are to those are not treated while the one treated or enrolled group
change the economic outcomes injecting different economic is analyzed to investigate the policy effect [2]. By
shocks to the economy. Which factor should be changed to assumption, the required average outcomes cannot be
achieve the required output and which factor should be achieved without treatment effect. The controlled (untreated)
controlled?, is the major concern of the strategic policy and treated groups are analyzed critically according to pre
makers. However, performance-based policy plays an and post-treatment period. The linear parametric model
important role in the economy and economic differences after shows the different time-variant and time-invariant
policy shocks can be measured through econometric tools. coefficients of interest but it is assumed that the relative
Difference-in-Difference (DiD) is one of the important coefficient of interest is not correlated with other individual
methods to evaluate the policy result and is being mostly observation. In addition, the mean difference of treated and
used in social and economic sciences. In general, Difference- untreated group before and after policy decides the impact of
in-Difference approach measures the average difference of intervention. Difference in difference approach measures the
two comparable groups before and after policy efficiency and affectivity of the policy in repeated cross-
implementation while the policy is considered effective if section panel data type [3].
comparative group difference is decreased. Specifically, in Difference in difference approach is not applicable on the
the situation of nonrandom and discontinuous data type, the nonreplicable data over time. The unit of the observations
difference in difference method can be more applicable and remains same before and after policy intervention while the
favorable to generate the required results [1]. Generally, pre-intervention data is a necessary condition for comparison.
different four groups are taken into analysis and three of In general, the treated and untreated data is analyzed with
6 Muhammad Ateeq-ur-Rehman and Andrey Launov: Project Evaluation from Application to Econometric
Theory: A Qualitative Explanation of Difference in Difference (DiD) Approach

binary nature to check the coefficient of interest known as [10]. In addition, the same method is applied in
difference-in-difference estimator. Why difference in management field and finds the impact of merger on the
difference method is most important in these days? Because stock price using the data of gasoline companies in Spain
the traditional methods requires complex procedures as [11]. Another study also uses the same approach to
compare to DiD method which is easier and applicable investigate the relation of environmental policy with the
without randomization of the data. In the first difference the carbon level in the industry [12]. The study finds the
time invariant behavior, within the same group, is removed relation of water with child mortality using the difference in
while in the double difference, the time variant behavior is difference approach. By comparing the group using publicly
removed from the both group. Meanwhile, the main result provided water, with the group, using privately water
shows the original change because of the policy intervention. supply, a less mortality child rate is found with the group
Thus the DiD removes not only the selection bias but also using privatized water scheme as compare to the controlled
eliminate the time variant bias and the rest of the change group not using privatized water [4]. Last but not least,
shows the counterfactual [1]. another study applies the difference in difference method
The contemporary study presents a brief view of DiD and analyzing the impact of financial assistance (Progresa) on
explain the common approach and methodology of the the child enrollment at primary and secondary school in
method. In addition, it explains the importance of the DiD Southern Mexico. The study finds that the impact of the
and the related issues as well referring the paper written by intervention is more on girls as compare to boys [13]. The
[4]. The major strengths and weaknesses of the method are literature shows the importance of difference in difference
also the relative part of the contemporary research work. approach academically and practically in different fields.
The outline of the paper is as follow; Section 2 discusses
the historical understanding with some interesting application 2.1. The Method
and also explains the reasons and the importance of the DiD Difference in Difference method is not only specific to the
approach. Section 3 gives a detailed view of the method with field of economics but also can be applied to the other field
example and some specific issues. The critical assessment of of sciences as discussed in the literature. It is the technique to
the method is also taken into account in the same section. estimates the mean differences between two groups keeping
Section 4 concludes. in view that one of the group members is exposed with
treatment while the other one is not. The method does not
2. Theoretical Background follow the random sampling techniques and therefore known
as the ‘quasi experiment method’ or ‘natural experiment
Difference in Difference method is a popular tool to find method’. Difference in Difference approach just explains the
the causal effect of the public intervention on the treated effect of the outer shocks on relative outcome comparing
group. Firstly, in order to find the reason of death because with the unaffected observations having same characteristics.
of impure water in the center of London. By applying The following steps are taken into account when the method
Difference-in-Difference method, the study discovers the is applied.
contaminated water was the main reason of high death rate
[5]
. The same method is also used to investigate the impact 2.2. Definition of the Variables
of the change in the law regime on the working stoppage in The two groups are defined having somewhat similar
the United States [6]. Another study applied the same characteristics before treatment and this way of selection can
techniques in economics and investigates the impact of remove the biasness from the model while one of them is
minimum wage law on the employees in Oregon comparing exposed for the treatment. In addition, the similarity between
with other employees [7]. Consequently, difference in both groups before treatment is necessary for the validity.
difference approach has also been applied to discover the
impact of the price on the sandwich sale comparing the 2.3. Time Specification
treated and non-treated group. In this regard, the method Time specification is the more important element in this
removes all trend and constant behavior of the sale and the method and time period with same unit is replicated when
remaining part clearly states the real impact of the price on treatment is exposed to one of two groups. The time
the sale [8]. Subsequently the same method is applied to period when intervention is not exposed, is known as
explore the effect of trade openness on the productivity and ‘before’ or ‘baseline’ while the ending time of the
wages in Uruguay. By introducing ‘’Southern Common intervention is known as ‘after’. Both time periods are
Market (MERCOSUR)’’ agreement, the trade openness compared relative to policy matter and the difference is
increases the productivity of the firms in Uruguay and captured through DiD.
reallocation strategy of the resources also increases the 2.4. Computation of the First Difference
wage level due to MERCOSUR law of trade in Uruguay In this step, the mean difference is taken with in same
[9]. Difference in difference approach is used to investigate group with different time period known as before and after
the illegal behavior of the ‘Marshfield clinics’ in period. By this way, the constant effect of the same group
Wisconsin. The comparison of two groups, involved in between two period is removed which is also known as
illegal conspiracy and not involved in unlawful behavior ‘time-invariant’ effect.
 International Journal of Data Science and Analysis 2017; 3(1): 5-12 7

2.5. The Final Outcome (DiD) the dummy variable and it takes value 1 if the dummy
The ending part of the method is a ‘’counterfactual’’ of the variable is in the treated group and ‘0’ otherwise. is the
two groups and again the difference is calculated from the time dummy and it takes the value 1 in the post treatment
first differences of the treated and untreated group. In first stage and ‘0’ in the pre-treatment stage . The most important
difference, the time invariant factor is removed and in the coefficient of interest in the equation (2) is which shows
double difference the time variant factor is removed with in the result of interaction term of time dummy ( ) with
two groups. The rest of the part shows the original effect of treated group ( ) after intervention. In addition how is
the policy on the treatment group. So these steps show that calculated is also shown in equation (3) which shows the
how the method work and what are the steps necessary to run back end procedure of the coefficient of the interest and it is
the method. just difference of the mean difference of controlled and
In Appendix (Figure 1), a conceptual explanation is given exposed group. Thus it is called the difference in difference
to illuminate the difference in difference approach briefly calculated by simple linear regression where the individual
where the outcome of the group (A and C) is on the baseline sample takes only two value ‘0’ or ‘1’ which prove the
period. This scenario shows the outcome of the both group linearity of the model.
before the treatment while D and C show the outcomes of
treated and untreated group after a specific time period. The = , − , − , − , (3)
line between point A and B clearly shows the trend in
3.1.2. Non-Linear Formulation
comparison with untreated group which shows that the
There are some outcomes which cannot be analyzed using
difference between B and D is less as compare to A and C. In
linear model and then a nonlinear model is applied to
addition, Appendix (Figure 2) shows in detail understanding
investigate the coefficient of interest. Thus DiD method is
regarding DiD. In the first column the variables names are
applied combining linear index with nonlinear function
categorized and second column shows the post-intervention
through probit and logit model but the common trend is not
facts of the two different periods while the third column
same. In simple linear model the unobserved heterogeneity
shows the difference within the group. The DiD shows the
remain constant and consistent over time while in nonlinear
actual change because of intervention.
model it remains inconsistent because of variation in group
The first difference of treatment and controlled are given
specific differences. In addition, the nonlinear model is
as (B-A= 0.14) and (D-C= 0.03) respectively in the last
applied when experimental data is not available and the
column of the Appendix (Figure 2). In the end of the last
treatment effect is not same with in the sample group as
column, Difference in Difference is calculated as shown in
shown in Appendix (Figure 4). It shows the trend of the
the equation (1).
treated child ratio compared with controlled group as an
= − − − (1) example of non-linear group.
Different statistical models are applied to find the
= 0.74 − 0.60 − 0.81 − 0.78 = 0.11 distributional form of the model and the most popular models
are probit and logit.
This is the approach of the method through which the
In probit model, standard normal distribution function is
actual change is calculated and also known as mean causal
estimated through linear index of the dummy variables while
effect.
in logit model, cumulative distribution function is derived
from different subsamples. The general idea is to find the
3. Assumptions of the DiD Model impact of the treatment or policy on the different subgroups
of the sample as shown in the Appendix (Figure 5). The
Cameron and Trivedi (2005) state the following
probability score is between 0 and 1while the common trend
assumptions of the DiD model.
or fixed effect is removed by taking difference and the
Assumption 1: ‘Common trend’ or ‘time effect’ remain
remaining difference of the distribution is known as the
same for treated and control group
policy effect.
Assumption 2: ‘Bias stability’ or ‘Composition stability’ in
treated and untreated group remains same if cross-sectional
data is used 4. Evidence
3.1. Common Approach The study uses the difference in difference technique to
find the impact of privatization policy of water system on
3.1.1. Linear Formulation death rate. The main question of study was whether the
The linear regression model is widely used to perform the intervention improves the health in Argintina or not. In 1995,
same result as shown in Appendix (Figure 3) and linear form the Govt. made a plan to privatize 30% of the local water
of the model is shown in equation (2) municipalities providing water 60% of the population. Before
1995 the mortality rate was same all over the country, after
= + + + . + (2)
intervention, the mortality rate decreased faster in the
Here, Y shows the final outcomes of the model while is privatized municipality as compare to nonprivatized
8 Muhammad Ateeq-ur-Rehman and Andrey Launov: Project Evaluation from Application to Econometric
Theory: A Qualitative Explanation of Difference in Difference (DiD) Approach

municipality in Argintina. [4]. Another study identified the method is applied by reweighting the area of the distribution
change using difference in difference approach proving that as shown in Appendix (Figure 8) from column 4 to 7.In the
the change is not related to another unobserved time-variant last column, the generalized difference in difference shows
and time-invariant heterogeneity which was the main that the intervention cuses to decrease the mortality around
objective of the study. The difference was observed around 9%.
8% which clearly states the impact factor of the intervention Difference in difference method is a famous tool now days
on the mortality rate. First of all the study shows that overall to find the impact of the policy but it is impossible to fully
people have become more health conscious and they prefer to disclose an uncertainty by the same method. The attractive
get pure water in order to protect the life. In Appendix point of this method is just to analyze the situation before and
(Figure 5 and Figure 7) show the general facts about the after intervention. The method is also a good tool to tackle
share of households connected to the water and sewerage all the non-experimental data type problems to investigate the
around the world and it shows that the poorest part of the causal effect of the policy. Contrary to that, it is not possible
household is shifted to the pure water area or privatized water to control the time variant and covariant factors which is a
supply area as compare to the richest ones. In the context of debatable issue related to that method. Because of this
Argentina the same case is with poor people and they shifted problem, the parameters become inconsistent with in the
themselves to the area where private companies offer the model. Therefore the best and consistent coefficient depends
pure water. Difference in difference results in Appendix upon the true value of the uncertainty with in the model.
(Figure 6) shows the significance of the method and a Consequently, the heterogeneous time trend of different
positive effect (0.018 and 0.042) shows that people shifted region can become a reason of inconsistent coefficient of
from local water municipalities to privitazied water uncertainty. In addition, if the common trend is not constant
municipalities in Argentina. and it has also a lag relation of more than two time’s period
The study also checks the trend of the mortality rate before then coefficient of interest does not show the true picture of
the privatization policy as shown in Appendix (Figure 3). the scenario. So the counterfactual part of the outcome
Before 1995, the average trend shows that mortality rate in cannot show the true value of the parameters because of
privitized and non privitized municipalities was same but mentioned reason and then the assumption can be tested for
after the intervention, the death rate in privatized further analysis.
municipalities decreased faster than non privatized one in
Argintina which perfectly remove the common trend and 5. Conclusion
provide a solid background in order to genreate the result
using DiD. It also uses the data to check the economic shock The difference in difference method is one of the famous
(time variant factors) on the mortaliy rate but the results tools in econometrics to investigate the causal effect of the
show that other factors have no cause to mortality rate as policy. Why difference in difference method is most
shown in Appendix (Figure 8). The only one reason to important in these days? Because the traditional methods
decrease the mortality rate was the intervention as shown in requires more rules as compare to DiD method which is
the first row of the Figure 8 with different time variant and easier and applicable without randomization of the data. In
covariant factors [4]. the first difference the time invariant behavior, within the
By analyzing the result shown in Appendix (Figure 8), it is same group, is removed while in the double difference, the
clear that the interevention played an important role to time variant behavior is removed from the both group and the
decrease the mortality rate if the other factors are not result shows a pure change because of the policy
controlled. Because of hetrogenous response from different intervention.
municipalities, comparison problem and different observation The historical background shows the importance of the
create a biasness in the model. The matching method is used method and its applicability in the different field of human
to resolve the problem of selection and covariant source of and social sciences. DiD method follows some steps to
biasness from the model. The common support region generate required outcomes in which two groups are
eliminate the selection biasness and reweighting the compared to check the effectiveness of the policy. The
controlled group after the selection biasness eliminates the difference is compared with treated and non-treated group in
second source of biasness which is also known as two time’s period model with the same unit of data. The first
distributional biasness. The reweighting techniques again difference removes the time-invariant factors while
find the difference in difference which is also known as the Difference in Difference removes the time-variant factors of
generalized difference in difference technique. In this the model and the remaining fact shows the original impact
technique, the coefficient of interest is calculated given the of the policy while the method is applicable to both nonlinear
time invariant factor of the treatment group in the mode and and linear functional form of the model.
the similar controlled group is matched with treatment group Evidence from Argentina, shows the positive impact of the
before intervention [4]. water policy which shows that the mortality rate is decreased
The scientist establish the probability distribution function about 5% after the local water supply is privatized. Matching
using logit model and find the common area of the techniques and generalized difference technique confirm the
distribution. At last the genrealized difference in difference impact of the intervention and the model also describes the
 International Journal of Data Science and Analysis 2017; 3(1): 5-12 9

ineffectiveness of the other shocks with in the model as the assumption of the model can be tested in order to
shown in Appendix (Figure 8). At last further investigation of generate the consistent parameters of the model.

Appendix

Source: Gertler, et. al., (2011)

Figure 1. Graphical Explanation of DiD.

Source: Gertler, et. al., (2011)

Figure 2. DiD Method.
10 Muhammad Ateeq-ur-Rehman and Andrey Launov: Project Evaluation from Application to Econometric
Theory: A Qualitative Explanation of Difference in Difference (DiD) Approach

Source: Havnes and Mogstad (2010)

Figure 3. Child Coverage Rate (1972-1985).

Source: Gertler and Martinez accessed from (www.google.com (2012)).

Figure 4. Nonlinear Approach.

Source: Galiani et. al., (2005)

Figure 5. Share of Household connected to water and sewerage 1992-2002.
 International Journal of Data Science and Analysis 2017; 3(1): 5-12 11

Source: Galiani et. al., (2005)

Figure 6. DiD (Privatization and mortality rate connected to the water system).

Source: Galiani et. al., (2005)

Figure 7. Comparison of mortality rate with privatized vs nonprivatized water companies.

Source: Galiani et. al., (2005)

Figure 8. Privatization of water services on Mortality Rate.

12 Muhammad Ateeq-ur-Rehman and Andrey Launov: Project Evaluation from Application to Econometric
Theory: A Qualitative Explanation of Difference in Difference (DiD) Approach

[8] Simon, J. L. (1966), The Price Elasticity of Liquor in the US

and a simple Method of Determination , Econometrica, No.
References 34, pp. 193-205.
[1] Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., and [9] Peluffo, A. (2010), Trade Liberalization, Productivity,
Vermeersch, C. M. J., (2011), Impact evaluation in practice, Employment and Wages: A Difference-in-Difference
The World Bank, DOI: 10.1596/978-0-8213-8541-8. Approach , Instituto de Economía, FCEA, Universidad de la
República, Uruguay.
[2] Lechner, M., (2011), the Estimation of Causal Effects by
Difference-in-Difference Methods’’, Department of [10] Mcclure, R. C. and Starr, M. A. (2012), Using difference in
Economics, University of St. Gallen, Discussion Paper, No. differences to estimate damages in healthcare antitrust: A case
2010-28. study of Marshfield Clinic, Working Paper Series, American
University, WD.
[3] Abadie, A. (2005), Semiparametric Difference-in-Differences
Estimators, Review of Economic Studies, No. 72, pp. 1–19. [11] Jiménez, J. L. and Perdiguero, J. (2012), Mergers, Difference -
In -D Ifference And Concentrated Markets: Why Firms Do
[4] Galiani, S., Gertler, P. and Schargrodsky, E. (2005), Water for Not Increase Prices? University of Barcelona, IREA working
Life: The Impact of the Privatization of Water Services on paper, No. 201205.
Child Mortality, Journal of Political Economy, Vol. 113, No. 1,
pp. 83-120. [12] Matisoff, D. C. (2011), Privatizing Climate Change Policy: is
there a public benefit? Working Paper, No. 60, Georgia
[5] Snow, J. (1855), On the Mode of Communication of Cholera, Institute of Technology, Atlanta.
2nd edition, London.
[13] Schultz, T. P. (2004), School subsidies for the poor: evaluating
[6] Rose, A. M. (1952), Needed Research on the Mediation of the Mexican Progresa poverty program, Journal of
Labour Disputes, Personal Psychology, No. 5, pp. 187-200. Development Economics, Vol. 74, pp. 199–250.
[7] Obenauer, M., Niebburg, B. M., (1915), Effect of Minimum-
wage Determinations in Oregon, Bulletin of the United States
Bureau of Labor Statistics, No. 176. Washington, D. C.