QUANTITATIVE EVALUATION OF

MANAGEMENT TECHNIQUES

Abstract

Watershed management projects are important for soil and water conservation but hard to
evaluate due to social and technical complexities. This paper promotes combining qualitative
and quantitative methods for more effective evaluations, using an Indian case study for
illustration. Real-World Example: Evaluating a Farming Subsidy Program
Context: A government rolls out a subsidy program to help farmers buy fertilizer in selected
rural districts. An evaluator wants to know if the program increased crop yields.

1. Introduction

Watershed projects are increasingly central to rural development, especially in India.

However, there's limited research on their real impacts. Evaluation is difficult because of the
complexity and context-specific nature of these projects.

2. Relevant Characteristics of Watershed Projects

Watersheds have three key features that complicate evaluation:

2.1. Spatial Interlinkages and Externalities

Upstream actions affect downstream outcomes. Benefits and costs are unevenly distributed,
requiring community coordination.

2.2. Multiple Objectives

Projects vary in goals (water quantity, sedimentation, biomass, etc.) and are influenced by
many local factors—making evaluation more complex.
2.3. Long-Term Impacts

Benefits like soil improvement or water recharge take years to show. Evaluating these subtle
changes is hard, especially without clear market value.

3. Evaluation Methods

3.1. Quantitative Techniques

These methods use statistical tools and controlled experiments (e.g., before-after, with-
without designs). They are ideal but often unfeasible in real-world watershed contexts due to
lack of randomization or baseline data.

1. Non-random treatment and control groups

In ideal situations (like randomized controlled trials), researchers randomly assign
participants to "treatment" (those who receive the intervention) and "control" (those who
don't). But in many real-world cases, such as watershed or environmental projects, this isn’t
practical or ethical, so evaluators must find alternative methods.

1. The Before/After Study

This is one of the simplest alternatives:
 What it is: You measure some outcomes (like water quality, crop yield, or income)
before a project is implemented, and then again afterward.
 Goal: To see if there’s been any change and attribute that change to the intervention.
Why it's weak
Although simple and often the only feasible option, this method has serious
limitations:
 Key assumption: It assumes nothing else changed during the time between "before"
and "after."
o That is rarely true. Many external factors (e.g., economic changes, weather,
policy shifts) could affect the outcome.
 Potential for bias: If the outcome improves (or worsens), you can’t be sure it’s
because of the intervention—it could be due to something else.
 This means causal attribution (saying the project caused the change) is questionable.
 A before/after study is like taking two snapshots and comparing them, assuming the
world stayed still in between. It's easy to do but prone to confounding—when outside
influences affect your results.
This is why more robust methods (e.g., difference-in-differences, matching, or using
instrumental variables) are preferred when possible, even though they require more
data and design effort.
Before/After
 Measure average crop yield in the treatment district before and after the program.
 Problem: If rainfall increased in the same period, the yield increase might be wrongly
attributed to the program.

2. "With/Without" Design
➤ What is it?
 Instead of measuring before and after an intervention in the same group, this method
compares participants (with the intervention) and non-participants (without it) at
one point in time, often after the project has already started.
➤ Why use it?
 Often used when no baseline (before) data were collected.
 Useful for retrospective evaluations (those done after the project has already started
or ended).
➤ Main challenge: Sample selection bias
 Treatment and control sites may differ systematically (e.g., richer villages might be
more likely to be selected).
 Any differences in outcomes could be due to these pre-existing differences, not the
intervention.
With/Without
 Compare yields between farmers who got the subsidy and similar ones who didn’t.
 Problem: Farmers who signed up might already be more motivated or have better
land.

3. Propensity Score Matching (PSM)

➤ What is it?
 A statistical method to reduce selection bias in with/without designs.
 It estimates the probability (or “propensity”) of each site or person being in the
treatment group, based on observable characteristics.
 Then, each treated unit is matched to a non-treated unit with a similar propensity
score.
➤ Goal:
 To make the comparison between treatment and control groups more fair by ensuring
they are similar in key observed ways.
➤ Limitation:
 Can’t account for unobservable factors (e.g., motivation, leadership quality), which
might still bias the results.
3. Propensity Score Matching
 Match each participating farmer with a non-participant who has similar land size,
income, education, etc.
 Still a problem: Cannot match on things like motivation or skills if not measured

4. Difference-in-Differences (DiD) or “Double Difference” Approach

➤ What is it?
 Combines before/after with with/without.
 Compares changes over time in both treatment and control groups.
➤ How it works:
 You calculate:
(Post-Pre in Treatment Group)−(Post - Pre in Control Group)
➤ Advantage:
 Helps cancel out any time-invariant unobservable differences between groups
(e.g., geography, long-term infrastructure).
➤ Limitation:
 Still relies on the assumption that unobservable factors do not change over time.
 Requires data before and after the intervention for both groups — which means the
evaluation must be planned ex ante (in advance).
Difference-in-Differences
 Measure yields for both groups before and after, and compare the changes.
 Strength: If both groups were on the same trend before, you can better isolate the
program’s effect

5. Instrumental Variables (IV)

➤ What is it?
 A more advanced statistical technique used when even DiD is not possible or
sufficient.
 IV is used to correct for selection bias caused by unobserved factors.
➤ How it works:
1. First stage: Model the probability of participation using an instrument — a variable
that affects whether someone gets treated but does not directly affect the outcome.
2. Second stage: Use the predicted treatment variable to estimate the actual outcome.
➤ Example:
 If access to a project depends on distance from a central office, and that distance
doesn't affect crop yields directly, then distance might be used as an instrument.
➤ Advantages:
 Can be used even after the fact (ex post).
 Adjusts for unobserved variables if the instrument is valid.
➤ Disadvantages:
 Finding valid instruments is difficult.
 If the instrument is not valid, the estimates can be more biased than not correcting at
all.
Instrumental Variables
 Suppose farmers closer to a fertilizer distribution center were more likely to
participate.
 Use distance to the center as an instrument (assuming it affects yield only through
participation).
 If valid, this can help correct for selection bias due to hidden traits like motivation.
🔸 Summary of Methods and Trade-Offs
Can Handle
Method Data Needs Pros Cons
Unobservables?

Pre and post data

Biased by time
Before/After for treatment ❌ Simple, feasible
trends
group

Can be used when Biased by group

With/Without Post data only ❌
baseline is missing differences

Post data + many Makes

Propensity Score Can't fix
observed ❌ treated/untreated
Matching unobserved bias
variables groups more similar

Difference-in- Pre and post data Needs good

✅ (some) Stronger causal claims
Differences for both groups baseline data

Instruments hard
Instrumental Post data + valid Controls for hidden
✅ to find, risky if
Variables instruments bias
wrong

3.1.1. Cost Analysis

Evaluates net gains by assigning values to costs and benefits. In watershed projects, this is
hard due to difficulty in measuring environmental services and uneven benefit distribution.
the limitations and challenges of using cost–benefit analysis (CBA) and cost-effectiveness
analysis (CEA) to evaluate agricultural development and natural resource management
(NRM) projects, particularly watershed projects in developing countries.
Cost–Benefit Analysis (CBA) Overview
 CBA evaluates whether a project produces net benefits for society by comparing the
costs and benefits with and without the project.
 This involves estimating adoption rates, attributing impacts to the project, and
evaluating how these affect production, prices, and incomes.
Example: If a new irrigation system increases rice yields, CBA would compare those gains to
the situation without the project, estimating the project's true impact.
Cost-Effectiveness Analysis (CEA)
 Similar to CBA, but focuses only on the cost of achieving a specific goal—not on the
benefits.
 Avoids the difficulty of assigning monetary values to environmental or intangible
benefits.
Example: If the goal is to reduce soil erosion, CEA compares the cost of different methods
(e.g., contour grass vs. stone bunds) without trying to quantify all resulting benefits.

🔹 Challenges in Evaluating Watershed Projects

1. Attribution of Benefits
 In watershed projects, multiple practices (some pre-existing) can improve outcomes.
 It's hard to know whether improvements are due to the project or independent
farmer actions.
Example: Farmers may already use traditional soil conservation methods, so
measuring only new project-promoted practices can overstate impact.
 2. Promoting Existing Practices
 If the project reinforces existing methods, it's difficult to measure how much the
project truly increased adoption.
 3. Data Accuracy
 These analyses depend on accurate data, which is hard to collect, especially in rural
or resource-limited settings.
 4. Valuing Environmental Benefits
 Many benefits (e.g., cleaner water, biodiversity) are hard to price.
 While environmental economists have methods to estimate such values, they often
require data that are unavailable or unreliable.
5. Uneven Distribution of Benefits
 Even if total benefits are high, not all users benefit equally.
 Those who don’t benefit may resist or sabotage the project, threatening its
sustainability.
Example: A project improves upstream irrigation but leaves downstream users with less
water—they may block further adoption or maintenance.
Key Takeaway
No evaluation method is perfect. The quality and credibility of the evaluation depend
on:
 The skills of the evaluator
 The type of project
 Data and resources available
 Timing of the evaluation
Evaluators must be transparent about assumptions and limitations, and choose
methods that fit the context and constraints of the project.

Summary Table: Evaluating Watershed Projects with CBA & CEA

Cost–Benefit Analysis Cost-Effectiveness Analysis
Aspect
(CBA) (CEA)

Compare costs of achieving

Objective Assess net societal benefits
a goal

Requires assigning
Does not require valuing
Benefit Valuation monetary values to all
benefits
benefits

Use in
Difficult due to hard-to- More practical in such
Environmental
value services cases
Projects

Attribution Hard to isolate project Same issue exists, but focus

Challenge impact from external factors is on cost comparison

High (detailed, reliable data Lower, but still needs

Data Requirement
on adoption, prices, etc.) accurate cost data
 Cost–Benefit Analysis Cost-Effectiveness Analysis
Aspect
(CBA) (CEA)

May hide unequal benefit Same – doesn’t address who

Equity Issues
distribution benefits or loses

Complexity / Often complex, may require Typically simpler, easier for

Transparency advanced techniques stakeholders to follow

3.2. Qualitative Approaches

1. Focus of Qualitative Research

 Emphasizes understanding "why" and "how" things happen.
 Seeks in-depth insights rather than broad generalizations.
 Values the perspectives of multiple stakeholders (e.g., farmers, local leaders).
 Focuses on the process of change, not just the results.
Example: Instead of asking “Did crop yield increase?”, a qualitative researcher might ask,
“Why did farmers adopt (or not adopt) the new practice, and how did they decide?”

2. Flexibility and Iteration

 Research design is flexible and adaptive.
 Uses open-ended questions and follows up as needed.
 Findings emerge gradually and evolve as more interviews or observations are
conducted.
 Not tied to pre-determined variables or hypotheses.
The researcher might change questions or revisit participants based on new insights gained
during the study.

3. Role of the Researcher

 The researcher is actively involved in data collection and analysis.
 Collects data directly (e.g., through interviews, observations).
 Interprets findings as they emerge, rather than waiting until all data is collected.
In contrast, quantitative researchers often analyse data collected by someone else using pre-
designed tools.
4. Understanding Over Generalisation
 Less concerned with statistical generalizability.
  Focus is on "transferable lessons learned" rather than predicting results elsewhere.
 Research is context-specific—findings are grounded in the local reality.

5. Validation Methods in Qualitative Research

 Because there's no statistical testing, qualitative research uses other techniques to
ensure validity:
Method Purpose
Uses multiple sources (people, settings, methods) to confirm
Triangulation
findings.
Member Checking Respondents review and comment on the researcher's interpretation.
Negative Case
Searches for examples that contradict early conclusions.
Analysis
Researcher reflects on how their presence and perspective affect
Reflexivity
results.
6. Sample Size and Scope
 Usually smaller in scale than quantitative studies.
 Focuses on depth rather than breadth—studies a few cases deeply rather than many
cases shallowly

Summary Table: Qualitative vs. Quantitative Research in Project Evaluation

Aspect Qualitative Quantitative

Measuring outcomes and

Main Focus Understanding processes and context
comparing values

Method Open-ended, flexible, inductive Structured, fixed, deductive

Interviews, observation, and direct Surveys, experiments, and

Data Collection
involvement secondary data

Role of Highly involved; interprets data Detached; analyses pre-collected

Researcher iteratively data

Scale Small, in-depth Large, generalizable

Statistical significance, numerical

Outcome Lessons learned, rich insights
outcomes

Validity Triangulation, member checking, Random sampling, statistical

Techniques negative cases corrections

Focus on statistically
Generalization Focus on lessons that may transfer
generalizable results

3.3. Mixed-Method Designs

Combines both methods to leverage the strengths of each. Quantitative data gives measurable
results; qualitative data adds context and insight into the mechanisms behind the outcomes.
Key Points Explained
1. Different Approaches Offer Complementary Insights
 Quantitative methods are best when:
o You need to measure how much something changed (e.g., yield, income).
o You have good data and comparable treatment groups.
 Qualitative methods are better when:
o You want to understand why or how a change happened.
o Impacts are unexpected, complex, or not easily measurable.
o You want to learn from the perspectives of beneficiaries.
No single method can capture the full picture—combining them helps to address each other’s
weaknesses.

2. Mixed-Method Evaluations
 Combine both qualitative and quantitative data in one study.
 Help deal with data limitations, time constraints, and complex questions.
USAID Example: Evaluating a child survival project in Indonesia under tight constraints.
Evaluators:
 Used existing data sets
 Reviewed project documents
 Conducted qualitative interviews
 Combined findings to cross-check results and rule out alternative explanations

3. Two Main Types of Mixed-Method Designs

Design Type Description Example

Qualitative and quantitative parts are done One method answers “what
Component
separately; integrated only in the final happened,” another answers
Design
interpretation. “how/why.”

Integrated Methods are interlinked and influence Qualitative findings might

Design each other during the study. modify a survey or guide data
Design Type Description Example

collection.
Integrated design allows the research process to be more dynamic, with ongoing feedback
loops between methods.

4. Why Integration Matters

 Sometimes, conflicting evidence between methods can highlight problems or blind
spots.
 This can improve the quality of the study, for example by:
o Revising flawed surveys
o Uncovering misinterpretations

4. Case Study: Indian Watershed Projects

4.1. Mixed-Method Approach

A study in Maharashtra and Andhra Pradesh evaluated five types of projects using surveys
and interviews. Though quantitative analysis was prioritized, qualitative data provided critical
insights.

4.2. Study Design

86 villages were studied using stratified random sampling, including NGO, government, and
mixed projects, plus control groups. The evaluation used perception-based and observed
indicators due to lack of baseline data.

4.3. Findings
Projects involving NGOs performed better in both social mobilization and agricultural
productivity. Qualitative data revealed how NGOs enabled collective action and helped poor
communities. It also highlighted who benefited or lost from projects (e.g., landless laborers
excluded from grazing lands).

5. Issues for Future Evaluations

Mixed methods are essential for capturing the complexity of watershed projects.
Focus not only on outcomes but also on processes like social organization.
Involve end-users in evaluation design for better relevance and acceptance.
Participatory and self-evaluation approaches can empower communities and improve project
implementation

PARTICIPATORY PARADIGM OF
QUANTITATIVE APPROACH

Participatory watershed development has become a global approach since the Earth Summit
(1992), gaining widespread acceptance due to its alignment with Agenda 21’s complex socio-
environmental demands. In India, watershed development programs shifted from a top-down
model to a bottom-up participatory approach after the Hanumantha Rao Committee
recommendations in 1995. This led to the formulation of Common Guidelines for Watershed
Development by the Government of India.
Despite large-scale implementation, no robust methodology existed to evaluate the
participatory components of watershed development.

Methodology Highlights
 A comprehensive list of participatory components was compiled from various policy
guidelines and programs.
 Feedback was collected from stakeholders (beneficiaries, implementing agencies,
etc.) and scored using a 3-point scale.
 Expert weights were assigned to each component, followed by classification into
major component groups.
 These were used to calculate:
o Participation Paradigm Index (PPI)
o Participatory Watershed Development Index (PWDI)
 These indices help in monitoring and evaluating stakeholder participation in WDPs.
 The methodology was validated using two real-world projects implemented by
different agencies.

3-Point Scale for Stakeholder Responses

Each stakeholder's response to a component is rated using a 3-point scale to assess the level
of participation or compliance:
Score Interpretation

1 Poor/Not Participatory/Not Implemented

2 Moderate/Partially Participatory

3 Good/Fully Participatory/Well Done

Key Components Used in Evaluation

The components are grouped into major participatory areas, each containing specific
evaluation points. Each component is later weighted by expert opinion.
1. Institutional Development
 Formation of Watershed Committees (WC), Self Help Groups (SHGs), User Groups
(UGs)
 Representation of women and marginalized groups
 Functioning and transparency of Village Institutions
2. Planning and Implementation Participation
 Community involvement in planning (PRA, micro-planning)
 Decision-making at village level
 Inclusion of traditional/local knowledge
3. Equity and Social Inclusion
 Participation of resource-poor, landless, SC/ST groups
 Distribution of benefits across community members
 Gender equity in benefits and responsibilities
4. Transparency and Accountability
 Regular social audits
 Public display of budgets, work schedules, and outcomes
 Grievance redressal mechanisms
5. Capacity Building and Awareness
 Training programs for local stakeholders and staff
 Awareness campaigns on watershed objectives and roles
 Leadership development among community members
6. Sustainability and Resource Management
 Maintenance of created assets (e.g., check dams, plantations)
 Community contribution to resource management
 Local-level mechanisms for CPR (Common Property Resource) management
7. Convergence and Linkages
 Coordination with line departments and development schemes
 Synergy with agriculture, forestry, and rural development sectors
 Role of NGOs and external agencies

MATERIALS AND METHODS: Step-by-Step Breakdown

Step 1: Component Identification

 Reviewed post-1995 watershed guidelines issued by various Indian
ministries/agencies.
  Utilised CSWCRTI’s (Central Water Conservation Research and Training Institute)
field experience.
 Resulted in a list of 80 components, covering all aspects of participatory watershed
development.
 Justification: Large financial investments warrant detailed and structured evaluation to
identify factors of success/failure.
Step 2: Questionnaire Design
 Questionnaire developed using these 80 components.
 Primary data was collected through personal interviews of:
o Internal stakeholders: WDT (Watershed Development Team) members, SHG
leaders, Watershed Association.
o External stakeholders: Beneficiaries, non-beneficiaries, community members.
 Responses were recorded using a 3-point scale:
Response Score

Yes 2

Do not know 1

No 0
 Verification was done via physical inspection of:
o Action plans, by-laws
o Meeting registers
o Bank passbooks
o Field execution of works
Step 3: Expert-Based Weight Assignment
 List of 41 experts (ICAR, NGOs, state depts., funding agencies including NABARD).
 15 experts responded and assigned importance weights to each of the 80 components.
 Final weight = Average weight per component based on expert input.

Step 4: Grouping into Major Components

 The 80 components were grouped into 10 Major Component Groups (MCGs) based
on thematic similarity.
  Table 1: Grouping and Weights
No. of Summed New Weight
Rank Major Component
Components Weight (1–10)
I Participation 15 66 10
II Transparency 15 49 9
III Watershed plan preparation 7 41 8
IV Watershed stakeholders 9 36 7
Institutions, meetings,
V 10 29 6
records
VI Monitoring and withdrawal 6 23 5
CPR (Common Property
VII 9 22 4
Resources)
PIA (Project Implementing
VIII 2 13 3
Agency)
WDT (Watershed
IX 3 12 2
Development Team)
X Equity 4 9 1


Step 5: Calculating Indices

For each watershed project, responses were used to compute two indices:
1. Participation Paradigm Index (PPdI) – For each major component group:

PPdI=(Max Weighted Score/Weighted Score ​) ×100

Weighted Score: Sum of stakeholder responses (0/1/2) × component weight × major

component group weight
 Max Weighted Score: Maximum possible score assuming perfect responses (all 2s)

2. Participatory Watershed Development Index (PWDI) – For overall project

evaluation:

( )
10⋅
Σ ⅈ=1 weighted score of 10 MCG
×100
Σ 10
ⅈ =1 max weighted score

Step 6: Classification of Watershed Projects

Based on PWDI, watershed performance is classified:
Category PWDI Range

Excellent > 90

Very Good 75 – 90

Good 50 – 75

Fair 20 – 50

Poor < 20

Background and Rationale

 Participatory watershed management gained momentum globally post-1992 and was
previously piloted in India (e.g., Sukhomajri 1975–80).
 India adopted participatory approaches after identifying the failures of centralized
models and the success of people-centric models like Ralegaon Siddhi.
 The 1995 Guidelines aimed to integrate:
o Productivity (optimal natural resource use)
o Social benefits (employment and development)
o Ecological sustainability
o Equity (focus on marginalized communities)

Challenges in Implementation
 Variability in bio-physical and socio-economic conditions across India complicates
standard implementation.
 Many implementing agencies lack capacity to adapt flexibly to local needs.
 Existing monitoring systems are administrative in nature, offering limited feedback
for strategic mid-course corrections.
 Achieving gender equity and inclusion of the poor remains a critical gap.
 There's a lack of national-level institutional mechanisms for long-term research and
evaluation.

Need for Quantitative Evaluation

 Traditional evaluations focused mainly on biophysical indicators (e.g., water table,
soil erosion).
 Newer evaluations include economic, social, and equity parameters.
  This study advances evaluation by integrating participatory criteria, creating indices
that can objectively measure the participatory success of watershed projects.
 Summary:
Step Description

1 Identified 80 key components of participation

2 Grouped into 10 major components

3 Experts assigned weights (1–10)

4 Stakeholder responses collected (0, 1, 2 scale)

5 Scores were calculated and normalized to form the PPdI

Conclusion
The methodology offers a structured, quantitative framework for assessing the participatory
effectiveness of watershed development programs. It helps in:
 Benchmarking participation
 Identifying weak areas in community involvement
 Supporting policy refinement and adaptive planning
 Institutionalizing participatory evaluation at the national level