Effective

rural
development

IFAD’s evidence-based
approach to managing
for results
The opinions expressed in this publication are those of the authors and do not necessarily represent
those of the International Fund for Agricultural Development (IFAD). The designations employed and the
presentation of material in this publication do not imply the expression of any opinion whatsoever on
the part of IFAD concerning the legal status of any country, territory, city or area or of its authorities, or
concerning the delimitation of its frontiers or boundaries. The designations “developed” and “developing”
countries are intended for statistical convenience and do not necessarily express a judgement about the
stage reached in the development process by a particular country or area.

This publication or any part thereof may be reproduced for non-commercial purposes without prior
permission from IFAD, provided that the publication or extract therefrom reproduced is attributed to IFAD
and the title of this publication is stated in any publication and that a copy thereof is sent to IFAD.

© IFAD 2018
All rights reserved

ISBN 978-92-9072-839-9
Printed June 2018
Effective rural development
IFAD’s evidence-based approach to managing for results
Table of contents

About the authors��������������������������������������������������������������������������������������������������������������������������������� 4

Abbreviations and acronyms ���������������������������������������������������������������������������������������������������������������� 9

Chapter 1
IFAD’s approach to promoting effective rural development������������������������������������������������������������������������10

PART I: Analysing corporate-level evidence

Chapter 2
Approach to attributing IFAD’s aggregate impact�������������������������������������������������������������������������������������30

Chapter 3
IFAD’s disbursement performance: trends and drivers������������������������������������������������������������������������������46

Chapter 4
Assessment of IFAD’s end-of-project documentation�������������������������������������������������������������������������������59

PART II: Assessing project-level impact

Chapter 5
Ethiopia’s Participatory Small‑Scale Irrigation Development Programme (PASIDP)�����������������������������������������64

Chapter 6
Kenya’s Smallholder Dairy Commercialization Programme������������������������������������������������������������������������70

Chapter 7
Madagascar’s AD2M-I Project��������������������������������������������������������������������������������������������������������������78

Chapter 8
The Philippines’s Irrigated Rice Production Enhancement Project���������������������������������������������������������������88

PART III: Analysing the evidence on thematic activities

Chapter 9
Land administration interventions: a systematic review��������������������������������������������������������������������������� 100

Chapter 10
Smallholder irrigation in a broader productivity and value chain framework: implications for IFAD investments��� 111

Chapter 11
Assessment of IFAD’s policy impact in four countries in Asia�������������������������������������������������������������������� 122

Chapter 12
The impact of adoption of CGIAR’s improved varieties on poverty: a systematic review������������������������������� 131

3
 About the authors

Aslihan Arslan Rui Benfica

is a senior agricultural development economist is a lead economist in RIA at IFAD. His work
(research) in the Research and Impact focuses on developing regions. Before joining
Assessment Division (RIA), Strategy and IFAD, he worked for the Department of
Knowledge Department, IFAD. Before joining Agricultural Food and Resource Economics
IFAD in January 2017, she worked at FAO as a at Michigan State University as an associate
natural resource economist, primarily focusing professor, and for the World Bank, where he held
on research and policy support on climate‑smart the positions of economist in the Gender and
agriculture. Prior to that, she was a post‑doctoral Development Group, and poverty economist in
the Africa Region, where he worked on poverty
research economist at the poverty reduction,
assessments and poverty and social impact
equity and development research group of the
analyses of public sector reforms. Prior to
Kiel Institute for the World Economy in Germany.
obtaining his PhD from Michigan State University,
She holds a PhD and an MSc in agricultural
he was a policy advisor at the USAID Office of
and resource economics from the University of
Agriculture and Food Resources in Maputo.
California, Davis, and a BSc in economics from
the Middle East Technical University, Ankara.
Juan Bonilla
Tim Balint is a senior economist at the International
Development, Evaluation and Research Program
is a technical specialist in RIA at IFAD. His work at American Institutes for Research. He conducts
focuses on analysing the development impact evaluations of social programmes in
effectiveness of IFAD’s investment projects developing countries, including agricultural
through quantitative and qualitative techniques, extension, cash transfers and value chain
including impact assessments and systematic interventions. Bonilla has served on the scientific
reviews. He also has expertise in strategic committee both for the Latin American and the
planning, results-based management and Caribbean Economic Association, and for the
organizational change. He holds a PhD Brazilian Econometric Society. Before joining
and master’s degree in politics and public American Institutes for Research, he worked for
administration from the University of Konstanz. the World Bank and was an assistant professor

4
at the University of São Paulo, where he taught agencies and international cooperation for WWF
graduate courses in development economics and Argentina, and project coordinator of the Center
applied microeconomics, including programme for the Implementation of Public Policies Promoting
evaluation. He holds an MA in economics from Equity and Growth (CIPPEC). She holds a
New York University, and a PhD in economics from master’s degree in public policy from the Harvard
the University of Maryland. Kennedy School of Government, a postgraduate
degree in management of social organizations
Peter Brueckmann from Universidad Torcuato Di Tella, and a BA
in international relations from the University of
is a specialist in impact assessment and
San Andrés.
development effectiveness research in RIA at
IFAD. His research interests are in the areas of
Alessandra Garbero
poverty, agriculture, infrastructure and global
health. He previously worked on a research project is a senior econometrician in RIA at IFAD. Her work
in Pakistan, jointly conducted by the Deutsche focuses on impact assessment methodologies
Gesellschaft für Internationale Zusammenarbeit and applied econometrics. Her prior work
and the University of Göttingen. He holds an MA experience includes working at the United Nations
in development economics from the University Population Division on population projections; in
of Göttingen, and a BSc in economics from the FAO, on the impact of HIV/AIDS on food security
Philipps-Universität Marburg. and agriculture; and at the International Institute
for Applied Systems Analysis, on population
forecasting, demographic modelling and
Romina Cavatassi
vulnerability to climate change. She holds a BA
is a senior technical specialist in RIA at IFAD. Her in economics and an MSc in statistics and social
areas of expertise are impact evaluation of projects research methods from the University of Rome
and programmes, survey design and data collection. La Sapienza, and an MSc in demography from the
Before joining IFAD, she worked for the Food and London School of Economics. She obtained her
Agriculture Organization of the United Nations (FAO) PhD in epidemiology and population health from
as project coordinator, natural resources economist, the London School of Hygiene & Tropical Medicine.
in its Agricultural Development Economics Division,
and prior to that as a consultant and also as a junior
Edward Heinemann
economist on several aspects of natural resources,
economic development and poverty reduction. She is a lead technical specialist in the Programme

holds a PhD in natural resources and development Management Department at IFAD, promoting

economics from Wageningen University, an MS in IFAD’s country-level policy engagement agenda.

environmental assessment and evaluation from the He is the principal author of the 2014 Consultation
Report for the Tenth Replenishment of IFAD’s
London School of Economics, and a bachelor’s
resources, IFAD’s 2011 Rural Poverty Report,
degree in economics from the University of Bologna.
and its Strategic Framework 2007-2010, policy
on sector-wide approaches, partnership strategy
Constanza Di Nucci and grants policy. Prior to 2007, he worked
is the senior technical advisor to the Associate first as a country programme manager in East
Vice-President of IFAD’s Strategy and Knowledge and Southern Africa, and later as the regional
Department. She coordinated IFAD’s Rural economist for that region. Before joining IFAD,
Development Report 2016, as well as other he worked at the African Development Bank as
initiatives on rural transformation and policy issues. a project officer. He holds a BSc in agricultural
She was formerly programme officer at the IFAD economics from the University of Reading, and
Environment and Climate Division. Before joining an MA in rural development from the University
IFAD, she was coordinator of governmental of East Anglia.

5
 Daniel Higgins Paola Mallia
is a specialist in impact assessment and is a specialist in impact assessment and
development effectiveness research in RIA at IFAD. development effectiveness research in RIA at
He previously conducted such work for UNESCO, IFAD. Her research interests are in the areas
the University of East Anglia, and a Ugandan of poverty, food security, agriculture, rural
education non‑governmental organization. He has development and migration. She previously
an MSc in impact assessment for international worked in FAO’s Agricultural Development
development, and a BA in development economics, Economics Division. She holds an MSc in
both from the University of East Anglia. economics from Bocconi University.

Harold Liversage Pierre Marion

is the lead land tenure technical specialist for is a quantitative research assistant in the
IFAD. Prior to joining IFAD in 2004, he worked Young Lives team in the Oxford Department
in the early 1990s for the association for a land of International Development at the University
rights NGO in KwaZulu-Natal, South Africa, and of Oxford. He previously worked as a research
then as a land reform programme coordinator analyst, specialized in impact assessments
for the South African Government from 1996 to and agriculture research in RIA at IFAD as well
1998. Subsequently he worked as a land tenure as in the FAO Statistics Division. His research
adviser for the Zambézia Agricultural Development interests are in the areas of poverty, vulnerability,
Programme, working with the Government of food security, agriculture and rural development.
Mozambique’s provincial land administration He holds an MSc in development economics
service and a national land rights NGO. From from the University of Sussex.
2002 to 2004, he worked as a land policy adviser
for the Government of Rwanda/United Kingdom Lisandro Martin
Department for International Development.
is Director of the West and Central Africa
He holds a master’s degree in town and regional
Division and Acting Director of the Operational
planning from the University of KwaZulu-Natal.
Policy and Results Division at IFAD. Prior to
joining IFAD, he worked for the World Bank in
Chandra A. Madramootoo Washington, D.C., where he served as senior
is a James McGill Professor of Bioresource operations officer in charge of its results agenda.
Engineering at McGill University, and Director of the At the World Bank, he also held other positions,
McGill Water Innovation Lab. He holds BSc, MSc both in headquarters and in the field, including
and PhD degrees from McGill. His areas of expertise as a senior results specialist for Bangladesh
are water management, irrigation and food security. and Nepal, based in Dhaka, and as a social
He is also a visiting professor and visiting scholar development specialist in the Southern Cone
at the Massachusetts Institute of Technology. of Latin America. Between 2008 and 2011, he
He served as dean of the Faculty of Agricultural worked for the African Development Bank in
and Environmental Sciences at McGill from 2005 Tunisia, where his last position was as principal
to 2015. During his tenure as dean, he created results specialist. He holds an MA in public
programmes in integrated water management, affairs from Princeton University, where he was a
food safety, food security, and innovation and Fulbright Scholar. He also holds an honours BA
entrepreneurship. He was founding director degree in international relations from Universidad
of McGill’s Brace Centre for Water Resources del Salvador.
Management. He holds several awards, including a
DSc (honoris causa) from the University of Guelph.

6
Nancy McCarthy Lauren Phillips
earned a PhD in agriculture and resource is a senior technical specialist in the Programme
economics from the University of California, Management Department at IFAD, promoting
Berkeley in 1996, and a JD from the George IFAD’s country-level policy engagement agenda.
Mason University School of Law in 2009. In 2010, From 2008 to 2014, she was an assistant
she founded LEAD Analytics, a consulting firm professor of international political economy in
that specializes in agricultural economics and the Department of International Relations at the
legal analyses to support economic development. London School of Economics, and also taught at
The firm’s clients have included the World Bank, Johns Hopkins School of Advanced International
FAO, Inter-American Development Bank, and 3ie. Studies Europe. She has published on a wide
Prior to founding LEAD Analytics, she worked at range of topics related to the political economy
the International Food Policy Research Institute of development and finance. She previously
(IFPRI) in Washington, D.C., and held a joint worked for the Overseas Development Institute,
position with IFPRI and the International Livestock and has additional experience in private finance
Research Institute based in Nairobi for four years. and international NGOs. She holds a PhD in
international political economy from the London
Anna McCord School of Economics, an MA in international policy
studies from Stanford University, and a BA in
is a research economist and sociologist,
international relations and Latin American studies
specializing in the design and evaluation of social
from Stanford University.
protection programmes, with a particular interest
in programme impact at household level, public
works and labour markets, and political economy.
Silvana Scalzo
She has over 25 years’ experience working is an administrative associate in RIA at IFAD
in international development, with experience responsible for coordinating impact assessment
throughout sub-Saharan Africa, as well as in activities. She has worked as both a research
South and South-East Asia; and she has worked associate and a statistical associate for various
with a range of governments, major donors, IFAD divisions since 1990, focusing on corporate
United Nations agencies and non-governmental data and databases, and combining these
agencies. She has an MA in social and political data into corporate statistical outputs. She
science from the University of Cambridge, a has worked closely with the Organisation for
diploma in economics from the Open University, Economic Co‑operation and Development,
and a master’s and PhD in economics from the reporting on IFAD’s official development assistance
University of Cape Town. statistics, to ensure IFAD’s historical data are
properly represented in the online Common
Mitchell Morey Reporting Standard.

is an economic researcher at American Institutes

for Research. He is responsible for evaluations of
Tisorn Songsermsawas
development programmes, with a special focus is a research analyst in RIA at IFAD. His research
on sub-Saharan Africa. He received his PhD in interest focuses on investigating how social
economics, with a minor in African studies, from interactions and agricultural interventions relate
the University of Wisconsin–Madison. Prior to that, to household welfare outcomes. He holds a PhD
he received a master’s in industrial and labour in agricultural and applied economics from the
relations from Cornell University. He also lived University of Illinois at Urbana-Champaign, and
and worked in Madagascar as a peace volunteer, a BS in economics and statistics from Bucknell
teaching English as a foreign language. University. His prior experience includes periods

7
 at the University of Illinois, teaching development
economics and environmental economics at
undergraduate level; FAO’s Investment Centre,
conducting research on rice production in
Southeast Asia; and the International Crops
Research Institute for the Semi-Arid Tropics,
conducting research on marketing barriers of
small‑scale farmers in rural India.

Paul Winters
is the Associate Vice-President of the Strategy
and Knowledge Department, and Director of the
Research and Impact Assessment Division at
IFAD. From 2004 to 2015, he was a professor in
the Department of Economics at the American
University in Washington, D.C., where he taught
courses on impact evaluation, development
economics and environmental economics.
He also worked at the International Potato
Center in Lima, the University of New England,
and the Inter-American Development Bank in
Washington, D.C. He holds a PhD in agricultural
and resource economics from the University of
California, Berkeley, an MA in economics from
the University of California, San Diego, and a BA
in non‑Western studies from the University of
San Diego.

8
Abbreviations and acronyms

AD2M-I Appui au Développement du Menabe et du Melaky

CGIAR Consultative Group on International Research
CIS communal irrigation system
CLPE country-level policy engagement
COSOP Country Strategic Opportunities Programme
CPM country programme manager
DP development partner
FFS farmer field school
IA irrigators association
IAI Impact Assessment Initiative
ICO IFAD country office
IFAD9 Ninth Replenishment of IFAD’s Resources
IFAD10 Tenth Replenishment of IFAD’s Resources
IFI international financial institution
IRPEP Irrigated Rice Production Enhancement Project
MOP-SEDP market-oriented participatory socio-economic
development planning
M&E monitoring and evaluation
NERCORMP North Eastern Region Community Resource
Management Project for Upland Areas
NGO non-governmental organization
NRD New Rural Development
NTP-NRD National Target Program on New Rural Development
ORMS Operational Results Management System
PASIDP Participatory Small-Scale Irrigation Development
Programme
PBAS performance-based allocation system
PCR project completion report
PNPM National Programme for Community Empowerment
PRiME Program in Rural Monitoring and Evaluation
RIMS Results and Impact Management System
RMF Results Measurement/Management Framework
SDCP Smallholder Dairy Commercialization Programme
SDG Sustainable Development Goal
SO strategic objective
VfM value for money
WUA water user association
4Es economy, efficiency, effectiveness and equity
4Ps public-private-producer partnerships

9
 Chapter 1

IFAD’s approach to promoting

effective rural development

by
Paul Winters
Lisandro Martin
Constanza Di Nucci

In September 2015, the United Nations General Furthermore, a consensus has emerged
Assembly adopted the Sustainable Development that generating evidence by monitoring and
Goals (SDGs) with the aim of ending poverty measuring results can play a strategic role in
and hunger, protecting the planet, and ensuring informing programme, policy and investment
prosperity for all, leaving no one behind. Building decisions, and in making them more likely to
on the experience of the Millennium Development achieve development objectives (Segone, 2008).
Goals, which accentuated the importance of Sound, relevant and frequent information about
thinking through indicators of success as early the progress of a given intervention allows
as possible, this agenda sought to develop decision makers to assess whether progress is
measurement frameworks that would allow being made, whether results are being achieved,
countries not only to report progress, but also and whether changes need to be introduced
to manage their implementation strategies, in order to reach expected impacts. Indeed,
allocate resources accordingly, and ensure the managing for results leads to better decisions,
accountability of all actors (SDSN, 2015). which in turn contribute to the achievement of
The need to focus on and measure development development results.
results has been recognized at each of the four
However, effectively supporting development is
high-level forums on aid effectiveness (in Rome,
not just about generating evidence on individual
Paris, Accra and Busan in 2003, 2005, 2008
interventions, it is also about effectively managing
and 2011, respectively). Efforts to modernize,
development institutions. The scarce resources
deepen and broaden development effectiveness
used for development must be employed as
have led to the formulation of frameworks,
efficiently and effectively as possible to ensure
strategies and plans in most international financial
value for money. Effective development thus
institutions (IFIs), including at IFAD. Development
requires examining the policies, procedures
effectiveness – understood as “the extent to
and use of resources of institutions working
which the development intervention’s objectives
in development.
were achieved, or are expected to be achieved,
taking into account their relative importance” Hence, at both the corporate and project level,
(OECD, 2002, 20) – has become a core the generation of evidence is central for learning
operational principle of international development and accountability. The rapid changes and
organizations (OECD, 2012). increasing complexity and uncertainty of the

10
development landscape create new and evolving additional commitments were made to enhance
challenges to reaching sustainable development IFAD’s development effectiveness and its value
results. Thus, it is essential for IFIs to monitor for money.1
and generate evidence from their own projects, This report reflects IFAD’s ongoing efforts to
learn from internally and externally generated generate evidence to inform decision-making at
knowledge, and build a culture of continuous the corporate and project level. The chapters in
learning and progress towards results. this volume include examples of: corporate-level

Chapter 1
Working towards achieving development evidence analysis (Part I); project-level impact
effectiveness in IFIs does not happen assessment (Part II); and evidence analysis for
automatically; it must be made to happen. thematic activities (Part III). They are based on

It requires a corporate agenda that involves, detailed quantitative and qualitative technical
papers that are being or will be published in
among other things: leadership and guidance to
appropriate technical products, including in
drive change; corporate frameworks to set up
IFAD’s Research Paper Series and in academic
a structure that facilitates the use of evidence in
journals. Overall, the report shows how creating
decisions about the design and implementation
a culture of results-based management operates
of projects; institutionalized systems that allow
in practice – namely, through a series of different
effective monitoring, evaluation and assessment
types of analyses that can inform decisions.
of interventions; concrete instruments and tools
to harmonize and aggregate results in order to To put these individual chapters into a broader
assess corporate impact; and continuous learning context, this introductory chapter provides
processes that enhance existing capacities and an overview of IFAD’s results architecture.
knowledge, and ensure that future operations This report is the first publication of this type

incorporate learning. For international development at IFAD, and it is important to articulate how
these activities fit together. Building on the
organizations to advance their agendas on
Development Effectiveness Framework, it
development effectiveness, all of these elements
presents an overview of the ongoing corporate
are necessary. However, they are not sufficient
efforts to develop a culture of results that aims
unless they also build a corporate-wide culture of
to go beyond the standard approaches of most
results-based management.
IFIs. Key components of such a system include:
With this background in mind, IFAD has (1) a strong and coherent self-evaluation and
systematically increased its efforts to create a impact assessment system that links project
culture of results-based management and to bring reporting to corporate reporting; (2) a systematic
innovative thinking on development effectiveness understanding of the portfolio that helps identify
to the organization. Building on previous corporate indicators and targets as well as
efforts to focus on results, IFAD’s Development analytics for assessing progress; (3) internal
Effectiveness Framework was presented to the systems that monitor the use of resources
Fund’s Executive Board in December 2016 with and link them to corporate outputs, outcomes
the objective of creating the structure needed and impact to enhance value for money; and
to facilitate the use of evidence in decisions on (4) trained staff and partners who understand
designing and implementing projects (IFAD, and are actively engaged in generating and
2016). It proposes a series of actions to overcome using results.
constraints on the generation and use of evidence The rest of this chapter covers each of these four
in decision-making and seeks to push forward areas in turn. It then provides an overview of the
a results-based agenda. In 2017, during the conclusions of the individual chapters as well as
Eleventh Replenishment of IFAD’s Resources, general conclusions and the challenges ahead.

1. These commitments can be found in the Commitment Matrix in Annex 1 of the Report of the Consultation on the
Eleventh Replenishment of IFAD’s Resources (IFAD, 2017).

11
 From project to corporate problem to allow for careful reflection on its
sources and to determine potential solutions.
measurement: IFAD’s corporate
The proposed solution should not only be
reporting system directly linked to the underlying causes of

An organization’s corporate-level indicators of underdevelopment, but also have a clear logic

success should reflect its strategic goals and and be based on the available evidence of
what has proved successful in overcoming the
objectives as well as the anticipated pathways
identified constraints.
through which it will reach these goals and
objectives. Identifying these indicators should be A development project is a series of inputs
relatively straightforward provided that: anticipated to lead to an output that should
(1) it is easy to quantify the stated goals and address the underlying causes limiting
objectives; and (2) it is possible to identify development under certain assumptions.
indicators that can be easily aggregated across These inputs and outputs embody the proposed
projects for corporate results reporting. solution. Assuming they adequately address the
underlying causes constraining beneficiaries, the
For IFIs such as IFAD, which provides
expectation is that beneficiaries will respond in a
investments primarily through loans to borrowing
manner that leads to anticipated outcomes and
countries, projects are the primary means
the hypothesized impact. In the pathway from
of achieving corporate strategic goals and
inputs and outputs to outcomes and impacts, a
objectives. They are thus the starting point for a
project should have an underlying logic, or theory
corporate results-based management system.
of change, that articulates how the proposed
To be effective in obtaining results, projects solution will bring about the desired result and
need to be designed to address a development what the assumptions are behind that logic.
problem linked to a corporate strategic objective If the proposed solution is successful and brings
or goal. Projects not linked to corporate about the desired results (outputs, outcomes
strategic objectives or goals are not fulfilling the and impact), the development problem should
corporate mandate. Projects that fail to address be addressed (as indicated by the arrow in
a development problem are not targeting a figure 1.1 showing that the result is linked to the
fundamental reason for underdevelopment. identified problem).
Figure 1.1 demonstrates the link between a
Thus, projects are hypotheses about how to
project and a development problem.
address a development problem. The success
Adequately addressing a development problem of the proposed solution should be verified
requires analysing the underlying causes of that through careful collection of data on key results

Figure 1.1 Effective projects

Development problem

Underlying causes

Proposed solution Result

Inputs Activities Outputs Outcomes Impact

Theory of Change

12
Figure 1.2 Corporate- and project-level reporting

PROJECT-LEVEL REPORTING (logframe)

Inputs Outputs Outcomes Impact

100% OF PROJECTS 15% OF PROJECTS

Chapter 1
Project indicators Core indicators Project-specific RMF indicators
Customized for projects Mandatory when relevant indicators


Outputs Impact
Core indicators RMF indicators
CORPORATE-LEVEL REPORTING

Note: RMF = Results Management Framework (previously, Results Measurement Framework).

indicators and analysis of those data. Each (M&E) plan developed at the initiation of the
project provides an opportunity to learn whether project. As projects necessarily differ because
and why a solution is effective. This work builds they address distinct development problems
an evidence base on what approaches are, and in differing contexts, these indicators vary by
are not, effective and the reasons for success project. This presents a challenge for aggregation
or limitations. to the corporate level because it is not possible

A sound project theory of change that to aggregate different indicators. However, there

addresses a development problem, combined are sufficient similarities among IFAD projects

with effective implementation and strong data to allow for some similar indicators – what IFAD
collection, constitutes the first step in assessing refers to as core indicators. These are discussed
development impacts at the project level. further below.
Assessing this performance at the corporate Second, a key part of measuring results
level requires aggregating these individual project involves attributing them to IFAD interventions.
results to the corporate level. This, in turn, As discussed more fully in Chapter 2, this is not
requires that projects include indicators that can challenging at the input and output level, but it is
be aggregated across interventions. a significant issue at the impact level, and in many
This is precisely the system IFAD has set up cases at the outcome level. Because attribution
for corporate reporting, which is summarized at the impact level is more complicated, it is
in figure 1.2. The figure shows how IFAD has also more costly. Therefore, it is difficult to justify
organized data collection in individual projects impact assessments for all projects. For this
to build a corporate results reporting system. reason, IFAD performs impact-level measurement
To understand this figure, some critical factors in only 15 per cent of projects but requires
must be considered. measurement of outputs and certain outcomes of
all projects. Chapter 2 describes IFAD’s approach
First, every project should have its own data
to measuring impact through attributable
collection system linked to the project’s theory of
impact assessments.
change. At IFAD, this is done through the logical
framework (logframe), which includes indicators IFAD’s approach is to ensure that all projects have
on results to be achieved. Data on these strong, high-quality logframes with indicators
indicators are collected throughout the project linked directly to its strategic objectives, which
cycle based on a monitoring and evaluation are in turn drawn from the SDGs. Project‑level

13
 Box 1.1 Operational Results Management System (ORMS)

To help facilitate a results culture and increase the probability of achieving development objectives,
IFAD has put in place the Operational Results Management System (ORMS), a single interconnected
system that provides real-time data on projects for evidence-based decision‑making.

ORMS is a fully fledged online platform that allows IFAD to aggregate outputs to the corporate
level. By collecting all the data from each project, the system links expected results (as per the
logframes in project design reports) with progress towards results (as documented in supervision
reports) and results actually achieved (as reported in project completion reports).

ORMS brings IFAD more in line with the standards of other international financial institutions
while promoting efficiencies. Efficiency gains include streamlined and harmonized reporting
processes, improved data generation and accessibility, and more-useful complete statistics and
reports. These improvements allow for timelier evidence-based decision-making, informed by
better‑quality, accessible and real-time reporting and analysis of results.

Figure 1.3 Core indicators reported in 2016

Natural resource management Marketing

22.2 million 4 million 3.6 million 16,460
voluntary savers active borrowers hectares of common-property-resource land kilometres of roads
under improved management practices contructed/rehabilitated
63:37 54:46
168,000 31,740
hectares under constructed/rehabilitated marketing groups
irrigation schemes formed/strengthened

Microenterprises Agricultural technologies

1 million 2.01 million
people trained in business people trained in crop production
and entrepreneurship practices/technologies
112.8 million
81:19 52:48
people
67,070 721,000
enterprises accessing business people trained in livestock
development services production practices/technologies

50:50
53:47

indicators are tracked over time using an depending on a project’s sectoral focus.
online system called the Operational Results For example, a project that includes a finance
Management System (ORMS). As explained in
2
component would include an indicator on
box 1.1, ORMS includes project data from all the numbers of voluntary savers and active
projects in the portfolio at the output level, and in borrowers. A project that promotes agricultural
some cases at the outcome level. technologies would include the number of
Included in the indicators identified in the people trained in crop production or livestock
logframes are core indicators – that is, a set production practices. And a project that improves
of indicators that are mandatory for projects an irrigation system would include hectares

2. ORMS replaces the previous system, known as the Results and Impact Management System (RIMS-online).
14
covered by constructed or rehabilitated irrigation Indicators, targets and analysis:
schemes. ORMS facilitates the aggregation
generating and using results
of these project-level core indicators to report
outputs at the corporate level as needed. While the previous section described IFAD’s
Figure 1.3 provides some examples of the use overall corporate approach to measuring results,
of core indicators to report on IFAD’s 2016 this section discusses how these results can
corporate output. be generated and used for decision-making.

Chapter 1
It describes how to identify the indicators and
As described in Chapter 2, a similar process
targets to measure in order to reflect corporate
occurs at the impact level. In this case,
performance, and how to analyse information about
15 per cent of IFAD projects completed within


the portfolio to determine whether IFAD is doing the
the three-year replenishment period undergo an
right things.
impact assessment in which a counterfactual is
identified to determine impact. This attributable IFAD’s Strategic Framework 2016–2025 establishes
impact estimate is combined with a strong its overarching goal – that is, to invest in rural people
understanding of IFAD’s portfolio of projects and to enable them to overcome poverty and achieve
used to estimate total corporate impact at the food security through remunerative, sustainable
end of each replenishment period. This process and resilient livelihoods. To support this goal, the
makes it possible to report on key indicators of framework identifies strategic objectives, outcomes,
corporate success at the impact level. and pillars of results delivery that guide IFAD’s
operations over the 2016–2025 period (figure 1.4).
Overall, IFAD’s approach involves careful planning
Because one key purpose of measuring results is to
and measurement to cascade results from
determine whether corporate goals and objectives
individual projects to corporate measurement.
are achieved, identified indicators of success and
It allows for corporate reporting that can be
targets for those indicators should be linked to
attributed to IFAD investment. Equally importantly,
stated corporate objectives. Following this logic,
it creates an opportunity to generate lessons
IFAD’s Results Measurement Framework (RMF) for
from individual projects, and from IFAD’s
IFAD10 has been revised to ensure alignment with
overall approach, that can provide insights for
the corporate strategy. A key principle of corporate
future programming and for the development
measurement should be alignment of RMFs and
community.
indicators used to collect data.
As noted, Chapter 2 provides an overview of
With this in mind, the impact indicators and the
the approach used for the impact assessments.
core indicators (reflecting outputs and outcomes)
Chapters 5, 6, 7 and 8 give examples of specific
have been mapped, respectively, to each of the
impact assessments for projects in Ethiopia,
strategic objectives and areas of thematic focus of
Kenya, Madagascar and the Philippines. These
IFAD’s Strategic Framework in such a way that each
chapters not only show IFAD’s impact resulting
project’s impact and outputs can be linked directly
from those projects but also present lessons
to the corporate strategy. Project-level data collected
from those projects. These individual impact
through ORMS and the impact assessment initiative
assessments will also be used as the basis for
can be used to determine whether IFAD is reaching
corporate-level impact estimates, in this case for
its corporate objectives. Examples of core indicators
the Tenth Replenishment of IFAD’s Resources
are shown in figure 1.3, and the impact indicators
(IFAD10) period. By allowing IFAD to draw
are discussed in Chapter 2.
significant lessons from individual projects as well
as aggregate lessons at the corporate level, this Once indicators have been identified, it is necessary
approach helps ensure that IFAD is “doing things to determine targets for those indicators in order to
right”. Of course, as discussed below, it is also express the scale of an organization’s ambition in a
important that IFAD is “doing the right things”. given period. Assuming (1) that corporate goals and

15
 Figure 1.4 Overview of IFAD’s Strategic Framework 2016–2025

STRATEGIC VISION
Inclusive and sustainable rural transformation

OVERARCHING GOAL
Poor rural people overcome poverty and achieve food security through remunerative,
sustainable and resilient livelihoods

STRATEGIC OBJECTIVES (SOs) PRINCIPLES OF

ENGAGEMENT
SO 1 SO 2 SO 3
Increase poor rural people’s Increase poor rural Strengthen the environmental
productive capacities people’s benefits from sustainability and climate Targeting
market participation resilience of poor rural
people’s economic activities
Empowerment
Areas of thematic focus

• Access to natural • Diversified rural • Environmental

resources enterprise and sustainability Gender equality
• Access to agricultural employment opportunities • Climate change
technologies and • Rural investment
production services environment Innovation, learning
• Inclusive financial services • Rural producers’ and scaling up
• Nutrition organizations
• Rural infrastructure
Partnerships

OUTCOMES
Enabling policy and regulatory Increased levels of Improved country-level
frameworks at national and investment in the capacity for rural policy
international levels rural sector and programme
development, implementation
and evaluation

PILLARS OF IFAD’S RESULTS DELIVERY

Country programme Knowledge-building, Financial capacity Institutional functions,

delivery dissemination and and instruments services and systems
policy engagement

project‑specific output and impact goals align, If this process can be carried out for every
and (2) that clear implementation plans identify project for the period in question, determining the
the investments and the number of beneficiaries corporate-level impact target for each indicator
affected by different types of investments, it involves simply adding the anticipated impacts on
should be possible to determine the anticipated that indicator for each project.
output and impact of the project, and the number This requires understanding the portfolio of
of people obtaining benefits associated with each investments being undertaken. There needs to
corporate indicator. be a sense of the types of activities to be carried

16
out across the institution, and the anticipated understanding and analysing the portfolio and
outputs, outcomes and impacts that will occur complementary activities allow for broader
as a result of those activities. Targets for IFAD introspection. IFAD’s approach to results-based
corporate indicators should reflect how IFAD is management is then to reflect on activities being
investing – not just in what activities but how undertaken and consider whether they represent
much in each activity – because this information the best way forward.
will determine the anticipated outputs. Any shift

Chapter 1
While understanding IFAD’s own work is central,
in overall strategy should generate a change in a
it is also necessary to look beyond the Fund and
target level going forward. For example, if IFAD
incorporate existing evidence on issues related
decides to invest in more irrigation as a pathway


to improve production and market access (two to IFAD’s portfolio. One way of doing this is
strategic objectives noted in figure 1.4), the target through systematic reviews – literature reviews
for beneficiaries receiving irrigation investment that use a transparent process to find, evaluate
should also increase. Chapter 2 illustrates how and synthesize evidence on a given topic.3
this can be done by discussing how impact-level This information, along with ongoing impact
targets were established for IFAD10. assessments, consolidates what IFAD knows and
does not know about what works in a particular
In short, measuring IFAD’s corporate
sector or subsector. By conducting regular
performance requires three steps:
systematic reviews that synthesize all existing
1. identifying indicators that reflect IFAD’s
high-quality evidence on a given intervention,
corporate strategic objectives;
an organization such as IFAD can ensure that
2. determining targets for those indicators that
“what works” is considered when it is time to
reflect IFAD’s approach to development as
represented in its project portfolio; design new operations, action plans, policies or

3. creating a system of measuring whether strategies on specific topics.

targets are reached. Examples of this type of analysis can be seen
This process creates significant quantities of data, in Part III of this report. Chapter 9 reflects on
both on individual projects and for aggregation the approaches IFAD uses to support land
across projects. Given the other data that are also administration and provides a systematic review
collected to manage projects and the institution, of the evidence on what works and what does
this means there is a wealth of data available on not in this area. Along similar lines, Chapter 10
corporate activities. Managing for results requires considers how to improve on IFAD’s approach
identifying and understanding the conditions to irrigation investment. Chapter 11 looks
and factors that determine project performance, beyond individual projects to consider IFAD’s
as well as the binding constraints behind those policy engagement in Asia to draw lessons for
factors. Results should be not just considered but improving how IFAD works with governments.
analysed. These data provide that opportunity. It provides critical insights on how to improve
For example, the Strategic Framework shows policy engagement just as IFAD is in the process
the potential pathways towards achieving of stepping up these efforts. Finally, Chapter 12
IFAD’s goals, but the collected data identify the provides a meta-analysis of the poverty‑reducing
relative emphasis of these pathways across effects of particular types of agricultural research
the investment portfolio. This, in turn, allows that IFAD has supported in the past. In general,
questions to be asked about whether IFAD’s focus the motivation for these analyses is to improve on
is correct: Is IFAD doing the right things? Are IFAD’s overall approaches by regularly considering
there areas that are over- or under-emphasized? the set of activities that IFAD is supporting. They
Are there gaps in the approach? In other words, help ensure that IFAD is doing the right things.

3. See the Campbell Collaboration definition for details: www.campbellcollaboration.org/library.html

17
 Monitoring and analysing the The previous sections of this chapter have
focused on building from the project level to
use of resources to ensure value
identify corporate indicators, including impact.
for money Similarly, corporate VfM requires that both
While the focus of activities should be on projects and the overall institution use resources
achieving key development results, particularly wisely. At IFAD, economic and financial analysis
the SDGs, this work should be accomplished is completed for projects as part of their
using resources as efficiently and effectively as preparation. This analysis considers the returns
possible. This section highlights IFAD’s efforts to to the project, and the economy and efficiency
improve internal processes to enhance value for of resource use. As described above, measures
money (VfM). of outputs and impact determine project
effectiveness.
In the IFAD context, VfM means that IFAD
maximizes the impact of each dollar invested At the corporate level, the RMF includes
to improve the lives of poor and food-insecure indicators linked to operational and institutional
rural men and women.4 It requires balancing the efficiency, and it seeks to monitor and
“4Es”: economy, efficiency, effectiveness and improve corporate-level resource use. Overall,
equity. Economy means reducing the cost of corporate‑level VfM is a product of project-level
individual resources used for an activity while VfM as well as of corporate operational and
maintaining high-quality output. Efficiency means institutional efficiency. As it is IFAD’s business
increasing output at the same quality but at a model that largely determines the approach
lower cost or, equivalently, minimizing costs for a to resource use, improving VfM depends on
given quality of output, generally by reallocating conceptualizing and implementing a business
resources. Economy and efficiency are closely model that is created with the 4Es in mind.
linked – they focus on obtaining more at the The VfM literature emphasizes the need to
same cost. Effectiveness means achieving an systematically and simultaneously consider the
activity’s intended impact. Equity means ensuring use of resources, at both project and corporate
that the impacts are felt by poor or marginalized level, to be sure that the best option is used
beneficiaries. The 4Es are not new, but VfM to obtain the highest output and impact.
seeks to integrate a number of concepts that This consideration requires regular analysis of
are already part of the results agenda so that resource use and its links to the project and
development practitioners focus on resource use corporate results chain. As one analysis states,
as well as impact (Schiere, 2016). “Evaluative reasoning needs to preside over
VfM can be considered at the project level or the measurement” (King and Guimaraes, 2016, 67).
corporate level (Jackson, 2016). At the project VfM is not simply about reducing operational
level, it entails ensuring that investments in costs or cutting budgets, but rather about using
project activities are the best use of resources evaluative reasoning to think carefully about
to achieve the goals of the project – that is, that maximizing impact for the lowest cost possible.
project resources could not be used in a better In that sense, IFAD recognizes that moving
way to achieve the same project impact, nor
towards a culture of VfM goes beyond operations.
could the project have a greater impact with an
It requires ensuring that policies and procedures
alternative approach. At the corporate level, VfM
be designed to achieve intended outcomes
focuses on whether the business model used by
cost‑effectively and implemented following the
IFAD is the best approach to transforming core
same principles. If the corporate mechanisms
resources into impact.
behind IFAD’s operations do not incorporate the

4. This definition follows standard definitions used elsewhere, including, for example, DFID (2011), ICAI (2011), Jackson
(2016), and the articles in IDEV (2016).

18
Box 1.2 Value-for-money scorecard

IFAD has developed a value-for-money (VfM) scorecard to drive the VfM agenda and to help
management use evaluative reasoning to identify and balance the trade-offs inherent in pursuing
that agenda. These trade-offs include: (1) maximizing the short-term versus long-term benefits
of any course of action; (2) maximizing the number of poor and food-insecure men and women

Chapter 1
who benefit versus supporting the poorest countries or those in the most fragile situations; and
(3) reducing overheads versus strengthening the quality of operations. By helping clarify such
trade-offs, the scorecard will support IFAD’s efforts to increase its effectiveness, efficiency and
sustainability.


The scorecard is composed of a subset of indicators from the Results Management Framework
of the Eleventh Replenishment of IFAD’s Resources that measure progress in improving economy,
efficiency, effectiveness and equity in IFAD’s operational and organizational performance.
The indicators are chosen based on their potential to enhance IFAD’s VfM, to achieve greater
economy and equity, and to overcome long-standing barriers to greater operational efficiency
and effectiveness.

Selected RMF indicators for the VfM scorecard are also linked to the different dimensions of
the business model (that is, resource mobilization, resource allocation, resource utilization, and
resource transformation).
Source: IFAD (2017).

principles of efficiency, effectiveness, economy the allocation of resources and the Fund’s latest
and equity, then the consequences could strategic objectives and outcomes as expressed
significantly compromise the VfM of IFAD’s in the Strategic Framework. Thus, IFAD aims
operations (National Audit Office, 2001). For this to ensure that programming of outputs drives
reason, IFAD has developed a VfM scorecard the budget process, rather than having budget
as described in box 1.2, and related to selected allocations drive the programme.
RMF indicators shown in table 1.1.
Chapters 3 and 4 reflect efforts to improve the
Improving the VfM of a development organization
use of resources by analysing internal systems.
such as IFAD is not easy. Among other elements,
Chapter 3 presents trends in IFAD’s disbursement
it requires being able to establish the pathway
performance and the drivers of that performance.
from budget allocations to results achieved,
Disbursements at IFAD, as at other IFIs, were
at both the project and the corporate level.
falling below targets, and the analysis provides
Towards this end, IFAD has recently moved
input into internal IFAD plans to improve
from a budgeting approach based on clusters of
activities to a pillars approach based on outputs. disbursement. Chapter 4 considers the use of
IFAD has aligned all instruments in its strategic evidence to make claims in project completion
planning and budgeting architecture around four reports (PCRs) – the final reports on project
main results pillars (box 1.3). The overall objective performance, which include lessons for future
is to improve the effectiveness and efficiency of projects. The chapter reveals a strong need for
corporate planning and budgeting processes, IFAD to improve the use of evidence in drawing
as well as to ensure better consistency between conclusions from projects.

19
 Table 1.1 IFAD’s value-for-money (VfM) scorecard

Dimensions of Key problems Actions taken to enhance VfM

business model

Leverage resources through

borrowing

Core resources not being

Cofinancing with domestic and
Resource mobilization leveraged to the greatest
international partners
possible degree

Mobilization of supplementary
funds linked to climate, youth,
fragility (refugees) and private
sector

Country selection and resource

allocation through performance-
based allocation system (PBAS)

Targeting of countries and within

Resource allocation
countries needs strengthening
Tailoring country-level
approaches

Enhanced targeting of youth

Decentralization and enhanced

country-based model

Enhanced synergies between

Resource use within countries lending and non-lending activities
Resource utilization
not reaching full potential
Increased loan size

Mainstreaming climate, gender,

nutrition and youth

Development Effectiveness
Framework and framework to
manage for results

Impact assessment initiative

Insufficient focus on measuring

Resource transformation
and managing for results Enhanced transparency through
systematic action plan

Service delivery platform

improvements

20
Link to VfM “4E” dimensions Measurement of success through Results
Management Framework indicators

Economy and efficiency. Allows each dollar of

official development assistance to have a multiplier

Chapter 1
effect on the total amount of loans, thereby increasing
the efficiency and economy of these resources.

Effectiveness. Enhances effectiveness by improving • Debt-to-equity ratio (3.1.2)

impact with funds and knowledge that complement • Cofinancing ratio (3.1.3 and 3.1.4)


IFAD’s approaches and reinforce domestic • Number of persons receiving services (millions)
ownership. (2.3.1)
Effectiveness and equity. Enhances equity
by facilitating targeting of funds, and enhances
effectiveness by addressing particular concerns of
disadvantaged groups.

Efficiency and equity. Enhances equity through • Share of core resources allocated to
a focus on countries with strong needs and low-income, lower-middle-income and
effectiveness through an emphasis on performance. upper‑middle-income countries (3.2.1)
It also improves efficiency by sequencing services to
• Percentage of PBAS resources reallocated
borrowers.
in the Eleventh Replenishment of IFAD’s
Effectiveness and equity. Enhances equity by Resources (3.2.2)
ensuring that targeting is appropriate for the context • Number of countries included in the PBAS at
and leads to effective projects. the beginning of the cycle (3.2.3)
Equity. Enhances equity by ensuring reach to key • Number of persons receiving services (millions)
populations. (2.3.1)

4Es. Enhances the 4Es through expanded country

presence, which allows for better information • Time from concept note to approval (3.5.1)
flow and engagement, and more effective use of
• Time from project approval to first disbursement
resources.
(3.5.2)
Economy and effectiveness. Enhances economy • Disbursement ratio (3.5.3)
and efficiency through better solutions and enhances
• Ratio of budgeted staff positions in IFAD country
effectiveness through improved impact.
offices / regional hubs (3.6.1)
Economy and efficiency. Enhances economy and • Average size of IFAD’s investments projects
efficiency through economies of scale in project (IFAD financing) (3.2.4)
design and implementation. • Percentage of operations rated 5 and above
Equity. Enhances equity through improved targeting at completion for overall project achievement
and effectiveness by focusing on key issues (e.g. (IFAD’s Independent Office of Evaluation) (2.2.3)
climate and nutrition).

4Es. Ensures adequate information to drive • Number of persons receiving services (millions)
increases in the 4Es through evidence-based (2.3.1)
decisions. • Number of people with: greater economic
mobility, greater production, greater market
Effectiveness. Ensures attributable impact to
access and increased resilience (2.1.1, 2.1.2,
determine effectiveness.
2.1.3, 2.1.4)
Effectiveness. Creates an openness to data in • Percentage of countries with disbursable
order to provide incentives for improving the 4Es, projects using the IFAD Client Portal (3.7.5)
and reinforces domestic accountability mechanisms • Percentage of IFAD operations using
to increase aid effectiveness. Operational Results Management System
(3.7.6)
Economy and efficiency. Enhances corporate-level • Percentage of IFAD-supported projects trained
economy and efficiency by shortening processing through Centers for Learning on Evaluation and
times and facilitating nimbler business processes. Results initiative (3.7.7)

21
 Box 1.3 IFAD’s results pillars

• Pillar 1 – Country programme delivery: Strengthened effectiveness of IFAD-supported

programmes and their capacity to systematically scale up successful interventions.
• Pillar 2 – Knowledge‑building, dissemination and policy engagement: Strengthened
capacity to learn, generate and disseminate evidence-based lessons on rural development
so that innovations and proven solutions are scaled up.
• Pillar 3 – Financial capacity and instruments: Diversified IFAD financing instruments relevant
to different country contexts, with funds mobilized from a broader range of partners to
expand public and private investments for inclusive and sustainable rural transformation.
• Pillar 4 – Institutional functions, services and systems: Strengthened effectiveness and
efficiency of IFAD’s institutional management, administration and service delivery platform,
including part of the decentralization process.

Staff and counterparts’ readiness for knowledge‑sharing tools. Its learning methods
results management are based on three key principles: participation,
Taking the actions noted in the previous experiential learning and brain-based learning.
sections will not move IFAD to a results-based The Academy is structured around three main
organization unless a corporate-wide culture of complementary and mutually reinforcing pillars
results is constructed. This requires ensuring (figure 1.5). The first pillar, the Core Learning
that staff, development partners and country Curriculum, is at the heart of the Academy.
counterparts are equipped to manage for results It includes a set of basic training modules aimed
and embrace the ideas underlying results-based at ensuring that staff have the skills necessary to
management. As part of IFAD’s efforts in this effectively design and implement IFAD programmes
area, special training programmes have been while at the same time taking into account the
developed to ensure readiness to manage Fund’s priorities, commitments and evolving delivery
for results. model. In this pillar, special emphasis is given
To provide a foundation for results-based to data collection, analysis and reporting at the
management, and more generally to become different stages of projects. The second pillar offers
a more competitive and relevant IFI, IFAD is a set of technical training modules that complement
updating the knowledge, skills and competencies the core curriculum, delving deeper into many of
of its staff. The IFAD Operations Academy has the technical and fiduciary areas covered in the
been designed to equip staff to deliver. It provides basic training. Finally, the third pillar offers a space
a learning environment that allows staff to for peer-to-peer learning in order to capitalize on the
develop their core operational competencies, wealth of experience accumulated by the Fund and
while offering them opportunities to improve by external experts and partner institutions.
their technical capacities and providing a peer- IFAD-financed projects, as those financed by
to-peer platform where staff can learn from one most IFIs, are implemented by national staff in
another or from external experts who can provide the borrowing countries, and occasionally by
cutting‑edge knowledge.
other development partners or non-governmental
Operating at a high pedagogic level, the organizations (NGOs). Thus, to effectively track
Academy offers a mix of training, most of which and achieve development outcomes, IFAD must
is face-to-face. These face-to-face sessions enhance not only its own culture of results but
are complemented by follow-up activities, also those of its partners at the country level.
e-learning, video tutorials and other distance Specifically, countries should invest in developing

22
Figure 1.5 Structure of the IFAD Operations Academy

1 2 3
core learning technical peer-to-peer
curriculum training learning

Chapter 1
Module 1 Technical training on disciplines relevant to • Learning events
Operationalizing IFAD’s IFAD’s rural transformation agenda • Exchanges
decentralization plan • Mentoring/


Examples of topics that will be covered in
coaching
Module 2 technical training sessions
• Participating in
Developing country
missions of other
strategies
• Agricultural risk • Land tenure divisions
Module 3 management • Livestock
Strengthening project • Anti-corruption development
design • Climate and • Nutrition
development • Partnership-
Module 4
• Climate finance building skills
Strengthening project
• Conservation • Private-sector
supervision
agriculture engagement
Module 5 • Economic and • Procurement
Commitments and financial analysis (for reviewers)
safeguards – advanced – advanced
• Engaging with • Reduction of food
indigenous peoples losses
• Environmental • Remittances and
sustainability migrant investment
• Financial • Rural finance
management • Rural infrastructure
• Fisheries • Seed management
• Gender • Small-scale
• Institutional analysis irrigation
• Knowledge • Support of farmers’
management organizations
in country • Value chains
programmes • Youth

Mandatory
On demand Initiative-based
(within 2 years)

adequate in-country M&E capacities and in efforts (or standardized tools) to measure
putting in place systems that facilitate data governmental capacities at the institutional
collection, analysis, and reporting of results. level for results‑based management in the rural

These types of efforts are not new. In fact, and agricultural development sector. There is
most efforts to date, both by IFAD and by usually little emphasis on the capacities and
other development partners, have focused on systems required to collect, understand and
increasing capacities in their own institutions use data; to systematically link evidence to
and the M&E activities of their projects. decision‑making; and even less to effectively
However, there have been no systematic incorporate evidence in public decisions.

23
 Box 1.4 AVANTI – Advancing Knowledge for Agricultural Impact

AVANTI adapts a tried and tested self-assessment methodology of governments’ capacities

in managing for development results (CAP-Scan) to the rural sector. This results into a new
product (Ag-Scan) following three steps:
1. Identify key stakeholders, engage senior government officials, motivate champions to
engage, and coordinate with CAP-Scan facilitators.
2. Conduct preparatory working sessions and a two-day workshop to discuss and prioritize
needs.
3. Implement action plan, and collect lessons learned.

AVANTI will also generate knowledge products and case studies that include developing methods
for measuring SDG indicators. In addition, information on the progress of these indicators in
concrete cases will feed into ongoing debates and foster the exchange of knowledge within the
interested community.

For further information, see: www.avantiagriculture.org

To identify and understand the capacities that are To support countries in improving overall
in place, where the gaps are, how to measure M&E systems in the rural and agricultural
progress, and how to build on and strengthen development sector, IFAD has brought together
what exists in order to achieve sustainable The Evaluators’ Institute with Centers for
development, IFAD has put forward the AVANTI Learning on Evaluation and Results to develop a
grant to adapt an existing tool (CAP-Scan) to the comprehensive rural development-focused M&E

rural sector (AG-Scan). This tool can be used to and impact assessment training and certification
framework, called the Program in Rural
assess in-country M&E systems and capacities
Monitoring and Evaluation (PRiME). This training,
and identify shortfalls. The application of this
described in box 1.5, aims to significantly expand
programme is in itself a capacity development
and add value to efforts to close data gaps by
exercise. Through facilitated self-assessment
strengthening country and regional capacities to
tools, governmental units and institutions will
collect and manage data on rural development.
deepen their understanding of the challenges,
success factors and open questions existing Taken together, these efforts at the individual and

around M&E. This assessment results in the institutional level seek to build a culture of results

development of an action plan for strengthening in order to facilitate an overall strategy to improve
development effectiveness. The expectation
rural-sector M&E capacities and systems.
is that the more widely results management is
To ensure synergies between these efforts, IFAD’s
understood and its value recognized, the more
upcoming country strategies will include the
likely it is that high-quality data will be collected
results of these action plans to guide M&E efforts
and used in decision-making. Ultimately,
and resources through future IFAD projects.
success in this area will be reflected in the active
Box 1.4 describes AVANTI.
engagement by IFAD staff and counterparts in
management for results.

24
Box 1.5 Program in Rural Monitoring and Evaluation

The Program in Rural Monitoring and Evaluation (PRiME) seeks to establish a high-quality,
global and sustainable training programme to certify IFAD-affiliated and other rural development
professionals. The programme aims to achieve the following outcomes:
1. an improved skill base in M&E and impact assessment in the rural sector of IFAD-financed

Chapter 1
client countries;
2. development of a knowledge management repository, where all curriculum materials
and learning from the global course delivery, the “train the trainer” component, and the
certification process will be codified, stored, and made available as public goods;


3. development of a new M&E and impact assessment certification framework that will be
quality assured, implemented and widely recognized by M&E and rural development
professionals globally.

PRiME works with government officials and ministries currently engaged with IFAD, improving
their knowledge of M&E as a tool for greater impact in the rural sector. Participants are nominated
by their respective governments, based on pre-established criteria. In addition, the training and
certification programme is open to other government officials, IFAD staff, consultants, partner
institutions, staff of other IFIs and multilateral development banks, and others.

The PRiME curriculum has two levels: a fundamentals course in M&E; and advanced training
in either M&E or impact assessment. The curriculum uses a modular approach, with exercises,
case studies, and other innovative tools for learning and sharing. PRiME is headquartered at
the Centers for Learning on Evaluation and Results (CLEAR)–Latin America, based at Centro
de Investigación y Docencia Económicas (CIDE), Mexico City. For further information, see:
www.primetraining.global

Republic
of Moldova Kazakhstan
CroatiaRomania Mongolia
Bosn ia an d H erzegovina
Montenegro Macedonia Georgia Uzbekistan Kyrgyzstan
Albania Azerbaijan Dem People's
Armenia Rep of Korea
Tu rkey Tajikistan
Syrian Arab Republic
Malta Cyprus Republic of Korea
Tu n isia Lebanon Islamic Repu blicAfghanistan
Global Hub Team Morocco Gaz a an d th e West Bank Iraq
Israel
of Iran China
Jordan Pakistan
World Bank Group (WBG) Algeria
Ku wait
Nepal
Bhutan
Libya Egypt
Mexico Washington, D.C., United States Qatar
Saudi ArabiaUnited Arab India
Bangladesh East Asia Center
Emirates
Cu ba Myanm ar
Lao People's Asia Pacific Finance and
HaitiDominican Mau ritania Oman
Belize Jamaica
Saint Kitts an d N evis
Republic
An tigua an d Barbu da
Mali
Niger
Dem Rep
Development Institute (AFDI)
Hon du ras Dominica Cabo Chad Sudan Eritrea Yemen Thailand Viet N am
Guatemala
Saint Lucia Verde Senegal Shanghai, China
Philippines
Latin America Center
El Salvador
Nicaragua Grenada
Saint Vincen t and th e Grenadin es
Barbados Gambia
Bu rkina Faso Djibouti
Cambodia
Costa Rica Trinidad and Tobago
Guinea-Bissau
Francophone Africa Center
Guinea BeninNigeria South Asia Center J-Pal
Centro de Investigación y
Panama Venezu ela Sierra Côte Togo Eth iopia
Somalia
Sri
Docencia Económicas Guyan a Centre Africain d'Etudes
Leone d'Ivoire
Liberia Ghana Republic
South Sudan
Cen tral African
South Asia Institute for Lanka Palau Federated States
of Micron esia
Marshall
(CIDE) Colom bia Su rin ame Supérieures en Gestion Cameroon
Equatorial Financial Management and Malaysia
Islands
(CESAG) Guinea UgandaKenya
Research (IFMR)
Mexico City,
EcuadorMexico
Sao TomeGabon
an d Prin cipe Democratic Rwanda
Maldiv es In donesia Nauru Kiribati
Dakar, Senegal Congo Republic
Chennai, India
Focal point for managing of th e Congo
Burundi
Un ited Repu blic
Seychelles
Papua New
the grant funds and of Tanz an ia
Timor-Leste
Guinea
Solomon Tuvalu
all activities Peru Brazil Comoros
Islands
Angola
Malawi
Zambia Van uatu
Bolivia Mozambique Madagascar
Fiji
Zimbabwe
Namibia Mauritius
Paraguay Botswana

Swazilan d
Brazil and Lusophone Lesotho
Uru guay Africa Center South Africa
Anglophone Africa Center
Argentina
São Paulo School of University of Witwatersrand
Ch ile Economics at Getulio Vargas
Foundation (EESP/FGV) Johannesburg, South Africa
São Paulo, Brazil

25
 Conclusions While IFAD has had a history of measuring
results, these efforts represent a surge in activity
The purpose of this report is to highlight IFAD’s to expand management for results. This is neither
ongoing efforts to generate evidence to inform a small nor an easy enterprise. New challenges
decision-making at the corporate and project will emerge, mistakes will be made, and new
levels, and to create a culture of results. The analysis will uncover areas where IFAD is not
chapters provide examples of corporate-level doing as well as it should. This is expected as
analysis (Part I: Chapters 2–4), project‑level part of the process of improving development
impact assessments (Part II: Chapters 5–8), effectiveness in rural areas. The insights should
and analysis on thematic areas (Part III: be used to draw lessons and continue to improve
Chapters 9–12). These chapters are based on so that IFAD can be more effective in promoting
quantitative and qualitative analysis, which have rural development.
been used to answer key questions. While the
individual chapters provide specific lessons,
the report overall highlights that creating a
culture of results‑based management requires a
combination of data collection and analyses that
can be used to inform decisions.

In addition to providing an overview of IFAD’s

results architecture, this chapter has drawn
attention to how the different pieces of IFAD’s
ongoing efforts fit together, and shown the
actions being undertaken more broadly to
ensure management for results. These efforts
are critical to ensuring that IFAD maintains its
focus on contributing to SDG 1 (no poverty) and
SDG 2 (zero hunger) in rural areas. This chapter
recognizes the importance of corporate buy-in
and the role of leadership, and it highlights that
building a culture of results requires working
throughout an organization and embedding this
culture in all processes and products, not just
in operations.

26
References OECD (Organisation for Economic Co-operation
and Development). 2002. Glossary of key terms in
DFID (UK Department for International evaluation and results-based management. Paris:
Development). 2011. DFID’s approach to value Development Assistance Committee (DAC).
for money (VfM). Manuscript, Quest reference:
OECD. 2012. The Busan Partnership for Effective
3116186. London.
Development Cooperation. Paris: Development
ICAI (Independent Commission for Aid Impact). Assistance Committee (DAC).

Chapter 1
2011. ICAI’s approach to effectiveness and value
Schiere, R. 2016. What is new in value for
for money. Report 1 November. London.
money? eVALUation Matters: A Quarterly
IDEV (Independent Development Evaluation).


Knowledge Publication on Development
2016. Value for money in development work. Evaluation (third quarter 2016). Abidjan: IDEV,
eVALUation Matters: A Quarterly Knowledge African Development Bank.
Publication on Development Evaluation (third
SDSN (Sustainable Development Solutions
quarter). Abidjan: IDEV, African Development Bank.
Network). 2015. Indicators and monitoring
IFAD (International Fund for Agricultural framework for the Sustainable Development
Development). 2016. IFAD development Goals: Launching a data revolution for the
effectiveness framework. EB 2016/119/R.12. SDGs. A report to the Secretary-General of the
Rome, https://webapps.ifad.org/members/ United Nations by the Leadership Council of the
eb/119/docs/EB-2016-119-R-12.pdf Sustainable Development Solutions Network.
IFAD. 2017. Report of the Consultation on the New York.
Eleventh Replenishment of IFAD’s Resources: Segone, M., ed. 2008. Bridging the gap:
Leaving no one behind: IFAD’s role in the 2030 The role of monitoring and evaluation in
agenda. IFAD11/4/R.2/Rev.1. December 8. evidence‑based policy making. Geneva:
Rome, https://webapps.ifad.org/members/ UNICEF Evaluation Office.
repl/11/04/docs/IFAD11-4-R-2-Rev-1.pdf

Jackson, P. 2016. Value for money and

international development: Deconstructing myths
to promote a more constructive discussion.
eVALUation Matters: A Quarterly Knowledge
Publication on Development Evaluation
(third quarter 2016). Abidjan: IDEV, African
Development Bank.

King, J. and Guimaraes L. 2016. Evaluating

value for money in international development:
The Ligada female economic empowerment
programme in Mozambique. eVALUation
Matters: A Quarterly Knowledge Publication on
Development Evaluation (third quarter 2016).
Abidjan: IDEV, African Development Bank.

National Audit Office. 2001. Modern

policy‑making: Ensuring policies deliver value
for money. HC 289 Session 2001–2002:
1 November 2001. London.

27
©IFAD/Fahad Abdullah Kaizer

28
PART I

Analysing corporate-level evidence

29
 Chapter 2

Approach to attributing IFAD’s

aggregate impact

by
Paul Winters
Alessandra Garbero

Attributing corporate impact there is an overall corporate impact. It depends on

whether the evaluated interventions can be seen
The Sustainable Development Goals (SDGs) have as representing overall corporate efforts.
generated a renewed focus on results‑based
Some might question the need to measure
management, inducing the development
impact at a corporate level at all; surely, what
community to place more emphasis on creating
matters is understanding whether individual
corporate monitoring and evaluation (M&E)
interventions have an impact and drawing
systems that generate evidence. This new
emphasis has been driven partly by the desire of lessons on successes and failures to improve
donor countries to ensure accountability for funds future projects and programmes. Worrying about
provided to development institutions, and partly corporate-level impact potentially distracts from
by the recognition that managing for results can an intervention-level focus and may come at the
improve the impact of interventions and generate expense of learning from innovative individual
lessons for subsequent activities in the same activities. Projects are often selected for impact
country or elsewhere. 5 assessments7 specifically for their potential to
produce key lessons.
Significant progress has been made in methods
to rigorously assess impact6 – the highest-level In fact, however, there are several reasons to
results that are linked to activities’ ultimate assess corporate-level impact. A corporate-level
objectives, such as increasing income, food estimate of impact can be crucial for justifying
security, nutrition or resilience. However, this funding to an institution as well as, more broadly,
progress has been made vis-à-vis specific funding for development. It can address taxpayers’
targeted interventions. With these approaches, questions about whether development assistance
it is possible to identify the impact of individual is effective. For donors, the lack of corporate-
projects and programmes supported by a given level impact assessments represents a missed
organization, but not corporate-level impact. opportunity, given their interest in assessing the
Showing that a number of individual interventions impact of providing an institution with funding as
have impacts does not necessarily mean that well as the value for money of their donations.

5. See, for example, the call by the Evaluation Gap Working Group (2006) to systematically build evidence about what
works in social development.
6. See Gertler et al. (2011) and Khandker, Koolwal and Samad (2010).
7. In general, an attempt to empirically attribute the causal impact of a project on an indicator is referred to as
“impact evaluation”, but IFAD and others use the term “impact assessment”. In this publication, the terms are
used synonymously.

30
In addition, corporate-level impact measurement an intervention. At the corporate level, as seen
induces systematic thinking about an institution’s in table 2.1, contribution to impact is what is
overall portfolio. While the selection of commonly measured.
individual projects for impact assessments is
Besides attribution, there is another issue in
often theoretically justified based on learning,
assessing corporate impact: aggregation.
practically speaking it often reflects what is

PART I / Chapter 2
Corporate impact requires having indicators
feasible and in the interest of researchers. This
that can be aggregated across a range of
leads to a collection of impact assessments that
interventions as well as a means to add up
are skewed towards certain sectors, particularly
the overall impacts across those interventions.
social sectors, or towards narrow research
If every intervention had the same objective,
questions. For example, the impact evaluation
such as higher income, and every intervention
database of the International Initiative for Impact
had an impact assessment, aggregation would
Evaluation includes more than 5,000 studies.
be straightforward – a matter of adding up the
Of those, about 70 per cent are in the social
individual estimates of impact in a systematic
sector and 20 per cent are in economic and
way. However, in reality, this is not the case.
productive sectors (Sabet and Brown, 2017).
Projects vary in their objectives based on local
This is in spite of the fact that the Organisation
development needs and country priorities.
for Economic Co-operation and Development
Impact assessments are costly and therefore
reports that about 40 per cent of funding goes
are rarely undertaken for every project as each
to the social sector and 35 per cent goes to
additional impact assessment provides less
economic and productive activities.8 A corporate-
level approach requires reflecting on the range of information. However, the absence of an impact

activities in a portfolio. It facilitates a consideration assessment means no ability to identify impact

of whether a corporation is “doing the right and learn from projects. Determining the optimal
things” rather than just “doing things right”. number of impact assessments and how to
aggregate these into a corporate measure is not
This chapter seeks to clarify the general
a trivial task.
challenges of measuring impact at a corporate
level and to describe IFAD’s overall approach. With these challenges in mind, IFAD has
A primary challenge to measuring impact is been seeking to attribute its corporate-level
assessing attribution. Attribution refers to the impact on key indicators linked to its Strategic
ability to claim that impact on a key high-order Framework. These efforts are a by-product of
indicator is the result of a particular investment the Consultation on the Ninth Replenishment of
– that is, that an intervention caused an IFAD’s Resources (IFAD9) (2013-2015), where its
observed effect. This is distinct from measuring Results Measurement Framework (RMF) included
contribution, which involves monitoring a key the aspirational goal of raising 80 million people
indicator to determine whether it is heading in out of poverty. The indicator “people out of
a direction consistent with an investment to poverty” can create perverse policy incentives.9
support the claim that an intervention helped It was found to be a narrow measure of IFAD’s
to create the observed effect. Attribution corporate impact because it did not reflect all
involves a claim of causality that contribution of IFAD’s corporate strategic objectives, which
does not include. As described more carefully include goals other than moving people over a
later in this chapter and discussed in box 2.1, poverty line. Moreover, the target itself was not
attribution requires the creation of a reasonable based on a careful analysis of IFAD’s portfolio – it
counterfactual that allows for comparison of is not clear how much of IFAD’s investment was
what would have happened in the absence of designed to move people over a poverty line.

8. See Table 1.3.3 in OECD (2017).

9. The issue of discrete indicators creating perverse incentives for policymakers is widely known and was recently
articulated in a blog by Lant Pritchett (2014) for the Center for Global Development.

31
 Table 2.1 Corporate measurement of impact

Institution Impact indicators Target Year Type of

assessment

Population living on less than US$1.25 per

27.4
day (%)

GDP per capita growth rate (%) N/A

Underweight children under 5 years old (%) 17.7

Under-5 child mortality (number per 1,000 live

Asian Development 17.7
births) 2015 Contribution
Bank (ADB)
Ratio of girls and boys in education 1.0

Gross lower secondary education graduation

N/A
rate (%)

Maternal mortality ratio (number per 100,000

100.7
births)

GDP growth (%)

GDP per capita (US$)

Proportion of population living below the

poverty line

Income inequality (Gini coefficient)

Life expectancy
African Enrolment in education (%)
Development N/A N/A Contribution
Bank (AfDB) Unemployment rate (%)

Economic Diversification Index

Global Competitiveness Index

Food insecurity (% of population)

Resilience to water shocks (index)

Institutional capacity for environmental

sustainability

Population below the poverty line (%) 34

Population below the indigence line (%) 17

Caribbean
Multidimensional Poverty Index (headcount %) N/A 2019 Contribution
Development Bank
GDP per capita growth rate 0.9

Unemployment rate Decrease

People assisted to exit poverty (of whom 50%

100 m
are women)
Consultative Group
People meeting minimum dietary energy
on International 150 m 2030 Attribution
requirements (of whom 50% are women)
Research (CGIAR)
Women of reproductive age who consume Reduce by
less than the adequate number of food groups 33%

Increased household income of direct

Global Agriculture beneficiaries
and Food Security N/A N/A Attribution
Program (GAFSP) Proportion of target population below the
minimum level of dietary energy consumption

32
 Institution Impact indicators Target Year Type of
assessment

Landscapes and seascapes under improved

300 million ha

PART I / Chapter 2
biodiversity management
Global
Tons of CO2 equivalent avoided, both direct and 750 million
Environment N/A Attribution
indirect, over the investment or impact period tons
Fund (GEF)
Globally overfished fisheries (volume) moved to
20%
more sustainable levels

Poverty headcount ratio (US$4 per day PPP) (%)

Inter-American
Development Gini coefficient N/A N/A Contribution
Bank (IDB)
Growth rate of GDP per person (%)

Number of people experiencing economic

International 40 million
mobility
Fund for
Agricultural Number of people with improved production 43 million 2018 Attribution
Development Number of people with improved market access 42 million
(IFAD)
Number of people with greater resilience 22 million

International
End extreme poverty and boost shared
Finance N/A 2030 Contribution
prosperity
Corporation (IFC)

Proportion of population living on < US$1.25

Reduce by ¾
a day

Poverty rate of member countries whose poverty

Halve
rate is above 40%

Islamic Poverty rate of member countries whose rate

Reduce by ¾
Development is below 40% 2020 Contribution
Bank
Under-5 mortality rate Reduce by ¾

Maternal mortality rate Reduce by ¾

GDP of member countries Double

Income inequality Halve

Proportion of population living on < US$1.90

3% 2030
a day

Median income growth rate of bottom 40%

N/A N/A
of population

Work Bank Agriculture value added per worker N/A N/A

Group and
GDP per capita N/A N/A
International
Contribution
Development Under-5 mortality rate N/A N/A
Association
Malnutrition N/A N/A
(IDA)
Population living in areas under water stress N/A N/A

Population exposed to harmful air pollution

N/A N/A
(PM2.5)

CO2 emissions (kg per 2011 PPP$ of GDP) N/A N/A

Note: N/A = not applicable; PPP = purchasing power parity.

33
 Box 2.1 Contribution and attribution in results measurement and management frameworks

The contribution-attribution tension can clearly be seen in results measurement and management
frameworks (RMFs) or similar corporate measurement systems. RMFs are performance
measurement and management frameworks mechanisms that contain a suite of indicators and
targets to monitor corporate performance. Progress against an RMF is tracked and reported for
the purpose of accountability and learning, although in practice the use of RMFs is often limited
to communication and reporting tools for corporate executive boards.

RMFs are reasonably homogeneous across international financial institutions (IFIs) in terms of
terminology and structure but vary for other institutes. In general, most organizations adopt a
multilevel structure composed of three tiers:
• Tier 1 focuses on the overall global picture and monitors progress towards the Sustainable
Development Goals (SDGs).
• Tier 2 determines the development results that are attributable to an institution – the causal
effect of activities.
• Tier 3 assesses operational and organizational performance, although some IFIs have
separate tiers for these two areas.

Whereas tier 1 monitors progress, and thus contribution, tier 2 measures results that can be
attributed to the institution’s efforts. Impact indicators that seek to measure the ultimate goals of
activities, such as those linked to the SDGs, tend to be in tier 1. In most cases, tier 2 results tend
to be outputs, such as the number of people reached or receiving some type of support.

Using the most recent information available at the time of writing, table 2.1 presents impact‑level
indicators from either tier 1 or tier 2 for the RMFs or equivalent corporate documents of IFAD,
selected regional development banks, the World Bank Group (including the International
Development Association and International Finance Corporation separately), and major
agricultural institutes (the CGIAR, the Global Agriculture and Food Security Program [GAFSP],
and the Global Environment Facility [GEF]). The impact-level indicators are noted, as well as
whether they measure contribution or attribution.

Generally, across all the major institutions included in the table, there is a lack of impact indicators
attributable to that institution. Only IFAD, CGIAR, GAFSP and GEF have set at least one impact
target generated by their own operations, whereas the others report only lower-level outputs and
outcomes.

Moreover, only in the case of IFAD, and to some extent CGIAR, can one see an attributed impact
target and an explanation of how it is to be produced and measured. In light of this, IFAD’s
approach of attempting to attribute an impact to the institution, rather than merely monitoring its
contribution, remains unusual among IFIs.

34
Even with these limitations, the IFAD9 Impact and 29 million and 23 million beneficiaries will
Assessment Initiative (IAI) recognized IFAD obtain significant gains in poultry and livestock
management’s responsibility to generate evidence asset ownership, respectively. More than 10 million
on project success in order to report on past beneficiaries will experience an increase in each of
projects and learn lessons for future projects. the following domains: overall assets, productive
The approach to the IFAD9 IAI attempted to be assets, gender empowerment, dietary diversity,

PART I / Chapter 2
scientific, systematic and comprehensive, and it and reduction in shock exposure. Finally, using
provided IFAD with significant lessons that helped an asset-based poverty measure, an estimated
it advance a results-based agenda.10 24 million beneficiaries are expected to move
out of poverty as a result of projects that were
Results of the IFAD9 IAI are reported in table 2.2
either closing or ongoing between 2010 and
to illustrate the types of corporate-level impact
2015. Overall, the analysis paints a portrait of
measures that can be calculated. The results
IFAD improving the well-being of rural people in
demonstrate that IFAD beneficiaries analysed
terms of asset accumulation and higher revenue
as part of the initiative are, on average, better
and income.
off in percentage terms when compared with a
comparison, or control, group (discussed in more While the IFAD9 IAI was able to provide an
detail later in this chapter). IFAD’s investments in assessment of impact, the process had some
rural people have generated returns in a number limitations. This is particularly true given that other
of critical areas, including assets, resilience, IFIs have not yet sought to attribute impact at a
livestock ownership, agricultural revenues, nutrition corporate level and lessons could not be drawn
and women’s empowerment. Aggregate impact from the experience of others. Reflecting on this
projections show that 43 million beneficiaries will process has led to lessons that have helped
see substantial increases in agricultural revenues, move forward IFAD’s agenda for measuring

Table 2.2 IFAD9 estimated impacts (for projects open during 2010–2015)

Impact domain Indicator Global impact Projected number

estimate (average of people impacted
treatment effect) (%) (million)

Overall asset index 6.6 15.8

Durables asset index 2.7 6.4

Economic mobility Productive asset index 5.6 13.4

Income 4.0 9.5

Asset poverty reduction 9.9 24.0

Ability to recover from shocks 1.5 3.5

Resilience Reduced shock severity 1.8 4.3

Reduced shock exposure 4.5 10.9

Nutrition Dietary diversity 4.6 11.0

Agricultural revenue 18.0 43.2

Agriculture
Yields 3.8 9.2

Livestock asset index 9.5 22.8

Livestock
Poultry count index 12.0 28.9

Gender Gender empowerment 4.8 11.6

10. For details of the IFAD9 IAI, see IFAD (2016).

35
 corporate-level impact. Although the IFAD9 IAI corporate and project objectives. If there is not,
had a number of fundamental flaws, starting from a problem arises. Without a link to the indicators,
the identification of key indicators and targets, it the projects would not be counted in an
provided a reasonable basis on which to build. institution’s aggregate success. While a mismatch
between project and corporate objectives may
The remainder of this chapter discusses and
seem inappropriate, in work with governments
justifies the approach IFAD is currently taking.
and communities it may be difficult to avoid.
The following section discusses how corporate
For example, IFAD has had projects that improve
indicators and targets can be and are identified.
drinking-water quality, which should provide clear
The chapter then examines how impact
health benefits. However, its strategic objectives,
assessments are conducted on individual projects,
which focus on productive investment, do not
and looks at how they are aggregated to the
include health outcomes.
corporate level. The final section discusses some
of the limitations of the approach and how to move Calculating the associated targets of success
forward to continue to improve measurement. for each indicator and for a given period is
more complicated than identifying the indicator.
For example, if the indicator of success is the
From projects to aggregate number of rural people with increased agricultural
production (linked to SDG 2.3 on doubling
impact: identifying indicators
agricultural productivity), how many people
and targets constitute success over what period? Answering
As described in Chapter 1, corporate-level this question requires a thorough understanding
indicators of success should reflect an of the institution’s portfolio of activities – what is
organization’s strategic goals and objectives, the magnitude of activities undertaken to achieve
and should be relatively straightforward to an objective? The target for a given indicator
identify provided that: (1) it is easy to quantify and period of corporate reporting then depends
the stated goals and objectives; and (2) it is on the objectives in the individual projects (e.g.
possible to identify indicators that can be easily recipients expected to benefit from the project
aggregated across projects. As an example, one for a given objective) and whether the project is
can consider the difference between the goal successful (the anticipated effectiveness rate of
of “improved livelihoods” and that of “greater projects). The key building blocks of measuring
income”. The former is a more encompassing aggregate performance are: the design and
goal but is not easily quantified as there is no implementation of effective projects; and
agreed-upon livelihood measure. The latter understanding the distribution and the potential
is a narrower goal but is easy to quantify in of such projects across the institution’s portfolio.
monetary terms and easy to aggregate across As the starting point for corporate-level
projects because it is expressed in the same
measurement of impact is individual projects, a
unit. Measuring project or corporate impact
sound project theory of change that addresses
requires measurable and aggregable indicators;
a development problem, combined with
hence, even if corporate goals have not been
effective implementation plans, constitutes
written with measurability in mind, a solution
the first step towards achieving aggregate
must be found.
performance. Assuming that corporate goals and
Even if an adequate indicator is found to project‑specific impact goals align, and that clear
represent corporate priorities, one cannot implementation plans identify the investments
assume that an institution’s individual projects are and the number of beneficiaries affected by
designed and implemented to reflect corporate different types of investments, it should be
strategic objectives and goals – that is, that possible to determine the anticipated impact of
there is a one-to-one correspondence between the project and the number of people obtaining

36
benefits associated with each corporate indicator. • They need to be aggregable – that is,
If this can be done for every project for the period one must be able to add up corporate
in question, determining the corporate-level indicators across distinct projects with
impact target for each indicator involves simply aligned but diverse objectives. For example,
adding the anticipated impacts on that indicator one project may increase rice production
for each project. through irrigation, and another may increase

PART I / Chapter 2
livestock production through finance and
The corporate-level impact targets for the
technical assistance. To add these together,
period relating to the Tenth Replenishment of
a broad category, such as increased value of
IFAD’s Resources (IFAD10) (2016-2018) can be
production, is necessary.
used to illustrate how this is done. Figure 2.1
• A clear rule of impact needs to be established.
summarizes the steps taken by IFAD to develop
If on average project beneficiaries improved
targets. IFAD’s Strategic Framework 2016-2025
resilience by 1 per cent, do they have greater
states that IFAD’s strategic goal is: poor rural
resilience? Although any rule is arbitrary, rules
people overcome poverty and achieve food
should reflect what project success would
security through remunerative, sustainable and
look like – that is, what percentage increase
resilient livelihoods. To achieve this goal, IFAD
would be considered successful.
identifies three strategic objectives (SOs): SO1
– increase rural people’s productive capacities; To determine the targets for each of these
SO2 – increase rural people’s benefits from impact indicators, IFAD performed a broad
market participation; and SO3 – strengthen portfolio analysis based on the distribution of
the environmental sustainability and climate IFAD interventions across the portfolio. Through
resilience of rural people’s economic activities. a lengthy process, IFAD analysed project design
In accordance with this strategy, the following documents and available data from all projects
impact indicators were proposed: approved between 2010 and 2015, for a total of
• Number of people experiencing economic 189 projects. A systematic analysis was carried
mobility (goal). out to identify each project’s theory of change
• Number of people with improved production and record these into a database, along with
(SO1). financial data and outreach estimates. The link
• Number of people with improved market between each project and the key indicator
access (SO2). categories was determined in order to identify the
• Number of people with greater resilience potential impact of that project.11
(SO3).
Using the anticipated outreach target of 110-130
The selection of these particular indicators follows million beneficiaries for IFAD10 and assuming a
a set of basic rules: 50 per cent success rate, the estimated RMF
• They need to be clearly defined, measurable targets for IFAD10 were calculated for the four
indicators that are aligned with IFAD’s indicators. According to these targets, as a
Strategic Framework. For the three SOs, this result of IFAD-financed investment projects,
was straightforward because the objectives 40 million individuals are expected to experience
are relatively precise – although even in this significant economic mobility, 43 million people to
case clear rules across diverse projects are significantly increase production (SO1), 42 million
required. For the goal, which is broad and people to significantly increase market access
includes multiple dimensions, choosing an (SO2), and 22 million people to experience
indicator was more complicated, and in greater resilience (SO3).
order to reflect the broad nature of the goal,
The indicators reflect IFAD’s corporate strategic
economic mobility was used.
goal and objectives, and the targets reflect what

11. This review of IFAD’s portfolio did find that one in five projects included an objective that was not aligned with
current corporate SOs.

37
 Figure 2.1 Identifying corporate impact targets

Strategic Framework Systematic analysis of projects IFAD10 targets

Theory of Change

Goal 40 million people

Poor rural people overcome poverty and 189 greater economic mobility
achieve food security through remunerative, projects
sustainable and resilient livelihoods 2010-2015
Strategic Objective 1 43 million people
Increase poor rural people's increase production
productive capacities

Strategic Objective 2 42 million people

Increase poor rural people's benefits increase market access
from market participation

Strategic Objective 3 22 million people

Strengthen the environmental sustainability greater resilience
and climate resilience of poor rural
people's economic activities

IFAD actually does in the field. Because this was the project and how they would have fared in
a new undertaking for IFAD, it was necessary to the absence of the project. It implies identifying
review the portfolio to understand where IFAD something that is inherently unobservable:
has historically invested. In moving forward, the beneficiaries’ status had they not received
ideally, IFAD will keep track of its anticipated the project; in other words, a parallel status
impacts per project beginning at the design stage referred to as the counterfactual. One can
and have a consistent flow of information on observe individuals in only one of the two states
the expected impact for a given indicator. Given – with the project or without it – and not the
these indicators and targets, the key challenge other. To substitute for this lack of information,
is then to estimate whether these targets are met. attribution requires using one set of individuals
or households as a counterfactual for another
set. The task then is to identify a comparison,
Impact assessment of individual or control, group that is as similar as possible
to the beneficiary, or treatment, group had it not
projects
received the project.
At the core of the proposed strategy to measure
The fundamental problem of attribution is
corporate-level impact for IFAD10 is the
preventing selection bias in estimates of impact
aggregation of the results of rigorous project-level
– that is, the tendency for those who receive a
impact assessments. The key to project-level
project (treatment group) to be different from
impact assessments is being able to attribute
those who do not (control group). Selection bias
the impact observed on the ground to the
can occur when individuals self-select into the
intervention being evaluated.
project (self-selection bias) or when implementers
Defining impact means estimating the difference allocate the intervention to particular types of
between how project beneficiaries fared with individuals (project placement bias). Those

38
who self-select into a project may be more Experimental designs, such as randomized
entrepreneurial, more willing to work with others, controlled trials, randomly assign the eligible
or have more time available. Those selected into population into treatment and control groups. This
a project by project leaders may be poorer or random assignment of those eligible for the project
from marginalized populations. If evaluators fail ensures that there are no systematic differences
to account for the fundamental characteristics between project recipients and those who serve

PART I / Chapter 2
of beneficiaries in conducting an impact as a comparison group. The only difference
assessment, selection bias can be introduced between treatment and control groups should be
in the impact estimate. The differences between that the treatment group receives the project.12
a treatment and control group could be due Random assignment can only be done before
to pre‑existing differences in those who self- project implementation and in the case of ex ante
selected into or are selected for the project rather impact assessments. Randomized controlled trials
than due to the activities of a project. are generally considered the gold standard for
There are several methods for constructing a assessing impact, because they clearly provide
comparison group to attempt to ensure unbiased attribution that avoids selection bias.
estimates of impact. These different evaluation Non-experimental approaches seek to mirror
designs all share the ultimate goal of trying experimental designs by using the best data
to validly construct a counterfactual, but they possible, and then applying statistical techniques
differ in their data requirements, methodological to create a counterfactual and address selection
strategy and assumptions. Methods that require bias. Non-experimental designs often require
fewer assumptions and are therefore valid in assumptions that cannot always be tested.
more settings are preferred. When they are designed ex ante, their underlying
Impact assessment can be designed ex ante or assumptions are easier to defend where baseline
ex post. Those designed ex ante are developed data are available on both treatment and control
during the project design phase and implemented groups. When they are designed ex post, their
in conjunction with the project. Those designed underlying assumptions are harder to defend.
ex post are initiated after implementation has Yet, under the right conditions, these approaches
begun, and require constructing treatment and can plausibly replicate experimental treatment
comparison groups after the fact. Ex ante impact effect estimates. These approaches include
assessments are preferred because they embed matching methods, difference-in-difference
data collection into the project, generating estimators, instrumental variable models, and
high-quality baseline data before the project is regression discontinuity designs.
implemented and high-quality follow-up data after Careful data collection and appropriate statistical
it is implemented. They tend to be challenging methods lie at the core of rigorous impact
to implement because they require that projects assessments at the project level. All impact
be well planned in advance so that beneficiaries assessments seek to estimate average treatment
for the entire project cycle are known. Ex post effects – a measure that estimates the difference
impact assessments face their own challenges,
in mean (or average) outcomes between
including the difficulty of identifying a reasonable
households or individuals in the treatment
control group after implementation has begun
group and households or individuals in the
and the lack of baseline data from which to build
control group. For example, if one objective of
an evidence base.
an agricultural project is to increase yields, the
There are two broad classes of impact average treatment effect would be the increase
assessment designs: experimental and in the value of production per hectare due to the
non‑experimental. They differ in the way project, which is the difference in the value of
they identify the counterfactual population. production per hectare between the treatment

12. This assumes that randomization and sampling are done following appropriate statistical procedures.

39
 group and the control group. From this, the However, it will take time to identify projects
attributable impact of the project is determined. and to implement ex ante impact assessments
It identifies whether, on average, the treatment in the field. Such impact assessments require
group is better off as a result of the project. a baseline before investment in the field, and
then follow-up of from two to four years after
Although the impact indicators selected in a
given project should be relevant to the project’s investment has actually occurred. Impact
theory of change, they need to be aggregable in estimates and the lessons learned from the
order to measure corporate-level impact. Thus, analysis will then be produced within three to five
data collection across individual projects must years after the initial investment. Given the need
include a reasonably comparable set of indicators to assess impact during IFAD10 (2016–2018),
linked to corporate objectives, and the average IFAD is focusing more heavily on ex post impact
treatment effect must be identified for each of assessments in the short run because they are
these indicators at the project level. If a project essential to ensuring sufficient reporting of results
is not linked to a given corporate indicator, for this replenishment period, and therefore to
this may be acceptable from a project-specific computing corporate impact estimates.
standpoint, but its impact will then be zero at the Using these project-level impact assessments
corporate level, suggesting that the project made for corporate reporting in the manner
no contribution to that objective. Therefore, each described below also requires a certain level of
project that is assessed should be linked to at comparability of impact assessments across
least one corporate objective.
projects and countries. To this end, IFAD has
Furthermore, in some cases, including in the case developed a management system to ensure
of IFAD, there may be a desire to determine the that a solid and consistent approach is put
number of people who receive the benefits of a forward and used across the various impact
project. For example, as figure 2.1 shows, one of assessments. This allows consistency in
IFAD’s impact indicators is the number of people methodology, approach and tools for any given
with increased market access. Because impact study, taking into account project and country
assessments calculate the average treatment specificities.
effect, it can be challenging to determine the
number of people who benefited. If, for example,
a project increases yields by 10 per cent on Aggregating to the corporate
average, some people will receive gains lower
than 10 per cent, and others, greater. An average
level
treatment effect does not reveal the distribution The key objective of measuring corporate-level
of impacts, and in fact this is not straightforward impact is to determine whether investments
to determine. However, with assumptions about allowed the institution to reach its targets. For
what level of change constitutes a substantial IFAD, these targets have been described above.
impact (e.g. a 10 per cent increase in yield) and In practice, estimating impact at a corporate level
about the distribution of benefits across the requires a number of ingredients. First, it requires
target population, it is possible to estimate the a critical mass of impact assessments that meet
number of people benefiting. This is discussed the attribution principle through carefully designed
further in the next section. impact identification strategies, and that provide
In its current strategy for IFAD10 and beyond, measurable, aggregable indicators. Second, it
IFAD has tried to conduct impact assessments requires a methodology that allows one to go
designed both ex ante and ex post. Given the from a set of individual estimates of impact to an
benefits of planning ahead to ensure high-quality aggregate estimate of corporate impact. Inferring
data and a sound counterfactual, the long‑term the potential impact of the entire portfolio requires
objective is to focus on ex ante studies. conducting a projection or extrapolation.

40
This can be termed “intelligent aggregation” – a accountability as the selected projects should be
result synthesis that is informed by a carefully representative – to the degree possible – of the
thought‑out methodology. portfolio. However, it may come at the expense
of learning, because the selected projects
The first step is to select a critical mass of projects
may not be those that can generate insightful
to be evaluated through an impact assessment.
recommendations and lessons learned. Random

PART I / Chapter 2
Ideally, all projects would include a rigorous impact
project selection is also difficult to implement in
assessment, but as already noted this is generally
practice, particularly when the critical mass of
not possible owing to cost considerations. Data
impact assessments includes studies designed ex
collection requires administering questionnaires,
ante. Most of the time, it is not possible to know
and data analysis requires technical expertise;
the entire corporate project portfolio distribution
each has its own costs. Knowledge generation
in advance with certainty, so randomly selecting
is also subject to diminishing returns as
projects is not feasible. Moreover, it is not always
each additional impact assessment on a
possible to assess the impact of a project.
given development approach provides less
For example, partner governments need to agree,
new information.
monitoring data must be sufficiently strong to
Projects selected for impact assessment should create the basis for an impact assessment, and
reflect the corporate portfolio both geographically it must be possible to create a counterfactual.
and in terms of the focus. In selecting projects These last three elements make random
for impact assessments, there are normally two selection of projects for impact assessments
options: (1) purposively select projects based on almost impossible to implement. In practice,
criteria that would reflect a preference towards some compromise between rigour and feasibility
strategic, innovative projects, or simply a potential is needed to ensure a realistic approach that
for learning; and (2) randomly select projects to contains a critical mass of impact assessments
be evaluated, ensuring some form of statistical able to reflect the corporate portfolio.
representativeness. There is a trade-off between
Once impact estimates for key measurable
purposive selection and random selection.
indicators from individual impact assessments
Choosing projects purposively to maximize have been obtained, the next step is to aggregate
learning from each individual impact assessment this measure to the corporate level through
is justified by the reality that not all projects have projection or extrapolation of some sort. Making
the same potential to generate knowledge. Some such an extrapolation requires understanding the
may implement business-as-usual or small‑scale portfolio and the relationship between the projects
interventions, whereas others may have selected for the impact assessment and those
large‑scale or innovative interventions that warrant that are not selected. In other words, it requires
an impact assessment. However, an approach an understanding of to what extent the selected
based on a purposive selection designed to impact assessments relate to the overall portfolio.
maximize learning does not necessarily lead
The extrapolation can be conducted in a variety
to a set of impact assessments that reflect the
of ways. The focus here is on the basic approach
corporate portfolio. This approach can thus bias
used by IFAD. This approach begins by developing
corporate reporting.
a global, or aggregate, estimate of impact for
The alternative is to randomly select projects for the indicator of interest and then projecting it
an impact assessment from the list of projects. onto the population of interest. This means using
If the randomization is done properly, it should
13
the estimated average treatment effects for an
lead to a representative sample of projects that indicator from each project to calculate one global
can be used to make inferences about the overall estimate of impact showing the average treatment
portfolio. This approach maximizes corporate effect for an indicator based on IFAD’s full portfolio

13. Random selection would necessarily require some form of stratification to ensure that all regions and types of
projects are covered.

41
 of work. For example, if the indicator is an increase than average. Precisely calculating who benefits
in the value of production per hectare, the average is challenging. One conservative approach is to
treatment effects for each project can be used to assume a doubling of benefits for some and zero
make a global estimate of increase in the value benefits for the remaining population, and then
of production per hectare based on an intelligent to extrapolate this to the projected population of
aggregation of the average treatment effect of the beneficiaries. In effect, this is multiplying the global
projects evaluated. This intelligent aggregation can estimated effect by the total number of projected
potentially be done through meta-analysis, which beneficiaries. Alternative, more reasonable
is a systematic approach designed to provide assumptions could be made.
a quantitative summary of statistical indicators
Figure 2.2 summarizes how this was done
reported in similar empirical studies.14
under IFAD9 and how the results shown in table
Once this global estimate has been calculated, 2.2 were obtained. Under IFAD9, there was an
the measure of corporate impact must be attempt to randomly select projects that could be
computed. As noted in figure 2.1, this is the evaluated to compile a critical mass of projects
number of people achieving certain benefits, representative of IFAD projects across IFAD’s five
such as increased value of production. This is regions. Initially, about 200 projects were identified
calculated by applying the global estimate to as being near completion during the 2010–2015
the population of interest, which consists of the period and thus could be assessed for impact.
beneficiaries of the projects in the portfolio (within Of those, only 122 had sufficient data. Of the
a given time frame). As the global estimate is a projects with data, 41 were randomly selected for
global average treatment effect, assumptions must potential inclusion – 26 projects was the target
be made about the distribution of benefits – that number of impact assessments, so 15 were
is, the different percentage gains achieved by selected as reserves. For a number of reasons
different beneficiaries. Actual impacts on individual (such as data availability, strategic relevance, and
beneficiaries will vary, with some receiving greater timing), only 14 of the 26 were retained and 8
benefits than the average and others receiving less additional projects purposively selected, for a total

14. For an example of how this is done, see Garbero (2016).

Figure 2.2 Steps of the IFAD9 Impact Assessment Initiative

1 2 3 4

Review of portfolio Selection of projects Estimates of impact Meta-analysis and

across selected projects projection to the portfolio
Number and types 15 per cent of portfolio
of projects

Representative of portfolio
Projects closing
during replenishment
period

X million
people
impacted

42
of 22 impact assessments. To augment those, an post impact assessments and will include the
additional 14 projects were chosen for analysis estimation of average treatment effects for the
using secondary data. A total of 36 projects impact indicators of interest noted in figure 2.1
underwent some form of impact assessment (among other indicators). The third step will be to
(with primary or secondary data). The process carry out a meta-analysis across all the ex post
highlighted the challenges of random selection of impact assessments to derive global estimates of

PART I / Chapter 2
projects for impact assessments, and provided a corporate impact. The latter can be stratified by
strong argument that purposive selection from the the key portfolio features. The last step would be
beginning was more likely to be successful. to extrapolate to the portfolio universe.

Next, the average treatment effects on key The end result of this approach would be
variables were calculated for these projects based the generation of corporate-level impact
on available data – not all indicators were available estimates across the main impact indicators
in every study. A meta-analysis was conducted – namely, agricultural production, market
to obtain one global estimate per indicator. access, resilience and economic mobility.
This procedure provided the global estimates These can then be compared with the targets
of impact reported in table 2.2, column heading noted in figure 2.1. The main strength of this
“Global impact estimate (average treatment approach lies in its simplicity, transparency
effect)”. Such global estimates of impact were and, ultimately, coherence. Projects evaluated
then projected to a universe of beneficiaries not only support learning, they also respond
(both direct and indirect) reached by projects to the need for accountability because they
that were active or due to be completed in the are able to systematically reflect IFAD’s project
2010‑2015 period. Outreach estimates on the portfolio distribution.
number of beneficiaries reached were about
However, these approaches rely on a host of
240 million. It was therefore possible to project
assumptions, starting from those linked to the
impact with a simple multiplicative function, given
individual impact assessments and including the
that the global average treatment effects were
assumptions underlying the meta-analysis and
representative of the portfolio at the regional
extrapolation. In interpreting results coming out
level. For example, the poverty reduction global
of this exercise, these caveats must be kept in
estimate of 9.9 per cent translates into 24 million
mind. The primary benefit of conducting such an
people projected to move out of asset poverty
analysis is to increase and maintain the focus on
from the investments of projects open and closed
achieving the results articulated in a corporate
during the 2010‑2015 period.
strategic framework.
As part of IFAD10, a similar logic will be applied
to estimate corporate impact. The first step
has entailed identifying projects suitable for an Moving forward: opportunities
impact assessment in the existing portfolio of
and challenges
projects ongoing and closing between 2016 and
2018 according to a number of key features, Discussions of impact assessment often focus
including region and project type. The projects on reporting for the purpose of accountability,
have been selected purposively according to which is important for stakeholders. However,
project type and geographical distribution in this focus significantly underestimates the value
order to reflect the overall portfolio. The second of impact assessment as a means of learning and
step entails conducting impact assessments of as a way to focus development efforts. Attempts
the selected projects by 2018. As generating to attribute impact put a spotlight on a project
evidence through ex ante impact assessment by raising questions about how investments
takes time, this approach relies only on ex should ultimately lead to impact. Such scrutiny

43
 can lead to a reassessment of the project’s First, buy-in from both IFAD and government
theory of change and reconsideration of the stakeholders is crucial to ensure that
adequacy of a project’s monitoring and evaluation policy‑relevant questions are asked in the impact
(M&E) systems. assessment and to allow impact assessments
to be properly designed and implemented.
Highlighting the role of impact assessments,
An impact assessment provides the greatest
a paper by Legovini, Di Maro and Piza (2015)
benefits when there is a strong partnership
analysed the differences between projects with
between the project team and the technical team
and without an impact assessment. The analysis
conducting the assessment.
showed that the delivery of projects with an
impact assessment was significantly timelier and Second, project design can create challenges.
avoided common delays. These effects were A project that lacks a clear focus on delivering a
attributed to a clearer implementation road map, key result and that has multiple components that
evidence-based midcourse corrections, and are not clearly connected is difficult to assess
strengthened capacity on the ground. Thus, the because there is no critical mass of interventions
benefits of impact assessments include not just linked to a clear and measurable result.
improving learning and accountability but also In contrast, a project whose components all
focusing project managers on results and thereby drive to a single consistent set of results is easier
enhancing project design and general M&E. to assess.

The scrutiny that accompanies an individual Third, weak M&E systems within a project
impact assessment also accompanies hinder the overall impact assessment because
corporate‑level impact measurement. IFAD’s it is difficult to identify impact without adequate
attempt to assess impact both in IFAD9 and in information. This is particularly challenging for
IFAD10 has been a key factor in the elaboration impact assessments designed ex post because
of its Development Effectiveness Framework. the lack of adequate monitoring data, including
As described in Chapter 1, this framework is baselines and lists of beneficiaries, means
an institutional set-up that seeks to create a researchers have to try to create information from
structure to facilitate the use of evidence in past activities.
decisions on the design and implementation of Fourth, the best timing for the selection of projects
projects. A focus on development effectiveness that will undergo an impact assessment is also
requires the adequate measuring of results an issue. Data collection often needs to fit not
– from outputs to outcomes and impact – and only the agricultural cycle but also the project
the design and implementation of effective timeline. Ex post impact assessment ideally
projects. As with individual projects, the benefit of occur prior to the closure of the project so project
corporate-level impact assessment is to enhance completion reports can benefit from the impact
the focus on results. assessment findings. The timing of ex ante impact
assessments depends on project start-up.
While a corporate impact assessment agenda
presents an opportunity for IFAD in the long run, Another set of challenges relates to determining
there are significant challenges to implementing corporate-level impact. IFAD has been striving
the plans noted in this chapter and deriving the for evaluable projects that are suitable for impact
benefits of corporate measurement. One set assessments, that can maximize learning, and
of challenges revolves around the project-level that reflect IFAD’s overall portfolio distribution.
impact assessments that are the basis of the However, there is often a mismatch between a
corporate approach. project’s initial concept and its implementation.

44
This design-implementation mismatch means References
that projects do not reflect the expected project
type and therefore the portfolio distribution. Evaluation Gap Working Group. 2006. When will
Corrections or adaptations in implementation we ever learn? Improving lives through impact
are not always carefully documented, and this evaluation. Washington, D.C.: Center for Global
is particularly problematic because projects with Development.

PART I / Chapter 2
impact assessments form the empirical basis of Garbero, A. 2016. Measuring IFAD’s impact:
the extrapolation methodology. Background paper to the IFAD9 Impact
The resources needed to conduct an impact Assessment Initiative. IFAD Research Series 07.
assessment also present a challenge for two Rome: IFAD.
reasons. First, if a primary objective of impact Gertler, P., Martinez, S., Premand, P., Rawlings,
assessment is corporate reporting, it is difficult L. B. and Vermeersch, C. M. J. 2011. Impact
to embed data collection costs within a project. evaluation in practice. Washington, D.C.:
This is potentially resolved by ensuring that the World Bank.
impact assessment is relevant for governments.
IFAD (International Fund for Agricultural
Second, knowledge generation is a public good,
Development). 2016. Synthesis of lessons
and its value goes beyond the individual project or
learned from the IFAD9 Impact Assessment
country – impact assessments provide knowledge
Initiative. Rome, https://webapps.ifad.org/
with potentially broad implications. Consequently,
members/eb/117/docs/EB-2016-117-R-8-
it tends to suffer from underinvestment. This
Rev-1.pdf
means that new resources are likely to be required
for impact assessment, putting pressure on Khandker, S. R., Koolwal, G. B. and Samad,
already tight development assistance budgets. H. A. 2010. Handbook on impact evaluation:
Quantitative methods and practices. Washington,
Last, there are challenges linked to the overall
D.C.: World Bank.
aggregation methodology. The methods
necessary for individual impact assessments Legovini, A., Di Maro, V. and Piza, C. 2015.
and corporate aggregation are technically Impact evaluation helps deliver development
sophisticated and require specialist knowledge. projects. Policy Research Working Paper No.
This creates challenges in conveying the WPS 7157. Washington, D.C.: World Bank,
arguments for appropriate methods and in http://documents.worldbank.org/curated/
providing the caveats in any reporting. It also en/676351468320935363/Impact-evaluation-
creates challenges in communicating results. helps-deliver-development-projects

Nevertheless, these challenges represent OECD (Organisation for Economic Co-operation

tremendous opportunities for improvement. and Development). 2017. Development aid at a
Ultimately, the lessons learned, both positive and glance: Statistics by region, 2017 edition. Paris.
negative, will improve M&E systems overall. They
Pritchett, L. 2014. Saturday Night Live satirizes
will simplify and therefore focus IFAD-supported
kinky development. Blog post, October
projects, increasing IFAD’s capacity to deliver
21. Washington, D.C.: Center for Global
evidence-based, policy-relevant results; and they
Development, www.cgdev.org/blog/saturday-
will allow IFAD to achieve higher (and measurable)
night-live-satirizes-kinky-development
standards in aggregate terms.
Sabet, S. M. and Brown, A. N. 2017. Is impact
evaluation still on the rise? Recent trends in the
evidence base for development. 3ie Draft paper.

45
 Chapter 3

IFAD’s disbursement
performance: trends and drivers
by
Tim Balint
Daniel Higgins
Paola Mallia
Silvana Scalzo
Paul Winters

In the wake of the Addis Ababa Action Agenda’s This reflects broader efforts by IFAD to move
call for unprecedented investments in achieving towards results‑based management and to
the Sustainable Development Goals, international inform decisions through the analysis of data.
financial institutions (IFIs) – such as the World
As a specialized United Nations agency and
Bank and the regional development banks –
IFI that provides grants and concessional and
have been playing an increasingly important role
non-concessional loans to governments of
in providing financial assistance to developing developing countries, IFAD currently manages
countries. Unlike budget support, which is non- a portfolio of about US$6.8 billion of its own
earmarked and provided as a one-off payment, financing to 230 rural and smallholder agricultural
operations financed by IFIs are typically tied to development projects in more than 90 countries.
the implementation of a particular programme or Projects are collaboratively designed by IFAD
project for which funding is granted in tranches and the recipient government, and implemented
of disbursements. Accordingly, the success of by a project management unit composed of
IFI-financed operations, and their ability to have recipient government staff, with IFAD adopting
the intended development impact, depends on a supervisory role throughout implementation.
a rapid project start-up, and regular and timely Thus, when testing the factors that influence
disbursements of funding to recipient countries disbursement performance, this study considers
(Gohou and Soumaré, 2009; Sunjka and Jacob, factors related both to the country and to IFAD.
2013; Mishra, 2016). Although the observed
The number of published studies of IFI
disbursement performance is to a large extent
disbursement performance is relatively small
a reflection of the recipient’s ability to implement
– table 3.1 provides an overview. By adding
the project, IFIs have a shared responsibility by
to that body of work, the study presented in
supervising project implementation and releasing
this chapter can provide important lessons not
funds to the recipient when necessary and
only for IFAD but also for other IFIs. Internally, it
deemed appropriate.
represents the first systematic investigation of the
In an effort to promote more timely and drivers of IFAD’s disbursement performance and
accelerated disbursements to IFAD’s projects, seeks to offer evidence-based insights for future
the study presented in this chapter rigorously improvement. Externally, the study represents
analyses trends in the Fund’s disbursement the most comprehensive, and possibly the
performance and the drivers of those trends. most methodologically robust, analysis of

46
disbursement performance of an IFI to date. • What are the trends in IFAD’s disbursement
It can serve as a potential best-practice approach performance over the past 20 years?
for other IFIs wanting to assess and improve their • What are the most significant factors that
disbursement performance. influence IFAD’s disbursement performance?
• How can IFAD (and other IFIs, if relevant)
Formally, the investigation aims to answer the
improve disbursement performance in
following research questions:

PART I / Chapter 3
the future?

Table 3.1 Existing quantitative studies of the drivers of IFI disbursement performance

Organization and study Focus and sample Results

African Development Bank • Focus: Factors that influence • Positive effect: Higher GDP per
(AfDB) time between approval and first capita; higher project budget; project
disbursement. being financed by a grant rather
Gohou and Soumaré (2009)
• Sample: 2,195 AfDB-financed than a loan; project being related to
projects approved between 1967 communication sector.
and 2008. • Negative effect: Project in the south
of the continent.
• No effect: Project sector; project in
north, west or east of the continent.

African Development Bank • Focus: Causes of delay between • Positive effect: Higher approved
(AfDB) approval and first disbursement. project budget; more recent approval
• Sample: 525 AfDB-financed date; higher number of components.
Nkamleu, Tourino and Edwin
(2011) projects approved between 1990 • Negative effect: Longer planned
and 2007 in the agriculture and duration.
agro-industry sector. • No significant effect: Goal linked to
agricultural and productivity growth;
share of budget covered by AfDB;
located in an AfDB-only eligible
country.

Inter-American • Focus: Factors that drive • Positive effect: Higher approved

Development Bank (IDB) disbursement performance from budget; team leader located in
approval until end of project. country (after policy change in 2009).
Álvarez, Bueso-Merriam and
Stucchi (2012) • Sample: 1,049 IDB projects • Negative effect: Team leader
approved between 1996 and located in country (before policy
2011. change in 2009); longer preparation
time; change in team leader during
project life; number of projects
managed by team leader.
• No effect: Age and experience of
team leader.

World Bank • Focus: Effects of country • Positive effect on accelerating

having an upcoming election disbursements): When a country
Kersting and Kilby (2015)
and being aligned with the that is aligned with the United States
United States of America on the of America on the United Nations
United Nations Security Council, Security Council has an upcoming
against the number of months election.
taken to disburse 25 per cent of • Limitation: Effect does not hold for
project budget. non-competitive elections where
• Sample: 5,115 projects there is little risk to the incumbent.
implemented between 1984
and 2012.

47
 Data and methodology and (2) those that come thereafter. Through this
consultation process, indicators were formulated
Data for these two areas of disbursement performance.

This study analyses the disbursement Accordingly, IFAD’s disbursement performance is

performance of the 577 IFAD-supported projects considered from two perspectives: disbursement
approved between 1995 and 2014. It excludes readiness, and disbursement effectiveness.
investment projects approved before 1995 Disbursement readiness covers the period
because data on potential explanatory factors leading up to the first two disbursements. This
for IFAD’s disbursement performance are not period represents the ability to actually begin
available for the 1980s and early 1990s owing implementation of a project after formal approval.
to the lack of corporate databases at that time. Disbursement effectiveness covers the remaining
To ensure that the most up-to-date data were period, looking at the pace and amount of
included, disbursements until 31 December 2015 disbursements throughout the project lifetime until
were considered for the selected project sample. closure. These two dimensions are of the utmost
Data on the disbursements were drawn from an importance for development effectiveness. They
internal data source and carefully adjusted, taking also reflect the most common indicators used in
into account the specifics of start-up advances the results measurement frameworks of other IFIs,
and consistent retroactive financing in Botswana, including the African Development Bank, Asian
Mexico and Morocco, as well as returns and Development Bank, Inter-American Development
refunds. Country-level data from a variety of Bank and World Bank, as well as those used in
sources were merged with these internal data. previous studies conducted on their disbursement
performance (see table 3.1).
Defining disbursement performance To assess disbursement readiness, the analysis
indicators uses five indicators covering the three key
Disbursement-delaying issues can occur at many stages of this period, which are: being declared
points along a project’s life cycle. Figure 3.1 effective/entering into force; receiving the
gives an overview of the life of an IFAD-financed first disbursement; and receiving the second
project from approval to closure, highlighting disbursement. The second disbursement was
these points. Based on internal documents and identified as an important readiness milestone.
consultations with in-house disbursement experts, This was because, while the first disbursement
it was determined that the most meaningful way is sometimes an advance to cover planned
of categorizing these points for the purpose of project expenditures, the second disbursement
analysis was to group them into: (1) those that occurs only once proof has been provided that
come before the first two project disbursements; expenditures are under way and therefore that

Figure 3.1 The life of an IFAD project

Executive Board Financing agreement Entry into First Second Project Financial
approval is signed force disbursement disbursement closure closure

Possibility to
Ratification request Series of disbursements Further disbursements
(if needed) start-up funds per year (settlement of activities)

If ratification is not
needed, signing =
entry into force
Withdrawal conditions
to be met by borrower

48
project activities have actually begun. The five for each of these categories and subcategories
indicators of disbursement readiness are: were obtained from internal and external sources
• number of days between approval and for inclusion in the analysis.
effectiveness/entry into force;
One key part of this process was separating out
• number of days between effectiveness/entry
variables in these sets that were highly correlated
into force and first disbursement; with one another. Two highly correlated variables

PART I / Chapter 3
• number of days between approval and first cannot be included in the same analytical model
disbursement; because it is impossible to measure the individual
• number of days between first and second effects of each. Consequently, for each of the
disbursement; correlated pairs, the variable of the highest interest
• number of days between approval and to IFAD was selected for inclusion and the other
second disbursement. variable excluded.
To assess disbursement effectiveness, the
Generating a benchmark disbursement
analysis uses two indicators:
schedule
• disbursement rate at any point in time
across project stages; As part of the trend analysis for this study, it was
• disbursement rate at financial closure. important to obtain an idea of how a typical IFAD
project disburses funds across its life. This was
These disbursement rates are calculated as the
done using a method similar to that of Álvarez,
cumulative amount disbursed as a proportion
Bueso-Merriam and Stucchi (2012). Using data on
of the approved budget. For disbursements
disbursement rates from all of the projects in the
across project life, rates were calculated for given
sample, an econometric regression technique was
points along the project’s length, measured by
used to identify a line of best fit to serve as the
the time elapsed since approval as a proportion
benchmark disbursement curve for disbursements
of the planned length. Proportions, rather than
across project life. However, whereas Álvarez
absolute amounts of time, were used so that
Bueso-Merriam and Stucchi (2012) assessed
disbursement rates could be compared at
rates according to the amount of absolute time
similar project stages across projects of different
passed since project approval, this study used
planned lengths.
disbursement rates at different project stages,
measured by the amount of time since project
Identifying potential
approval as a percentage of the project’s planned
disbursement‑influencing factors
length. In this way, disbursement rates for projects
Input from in-house experts and insights from of different lengths could be compared at similar
the previous IFI studies were also sought in stages of implementation. This was deemed a
order to identify the factors whose influence on more meaningful comparison. Trends in IFAD
disbursement performance would be tested. disbursement performance could be observed by
This process highlighted the importance of both looking at deviations from the benchmark curve of
country- and IFAD-side factors. On the country project disbursement rates at a given stage.
side, important political and socio-economic
characteristics were identified, as well as the Analytical models
frequency of important incidents such as elections Three different analytical models were employed
and natural disasters. On the IFAD side, important to assess the effects of the various influencing
subcategories of factors were related to: the factors on the disbursement performance
financing of the project; the management of the indicators. All three models use econometric
project, specifically various characteristics of the techniques that provide an average effect size
country programme manager (CPM) in charge; of each influencing factor on the performance
and the conditions included in the project’s indicators, and a measure of the “statistical
financing agreement. Accordingly, sets of variables significance” of the effect.

49
 For the disbursement readiness analysis, a survival Trends in IFAD’s disbursement
model analysis was employed. This structures the
performance
data into time periods and assesses the influence
of a given factor on the likelihood of a project’s
Disbursement readiness
first disbursement occurring in that period (see
Cameron and Trivedi, 2005). The benefit of this IFAD’s overall disbursement readiness has
approach over other econometric models is that improved slightly over its past five three-year
it allows for the inclusion of projects that have replenishment periods (table 3.2). The average
not yet disbursed, as these are classed as “right time from approval to first disbursement fell from
censored” in the survival model, and the periods 21.0 months to 17.6 months, and the average
that have already occurred before the end of the time from approval to second disbursement
dataset can still be analysed. dropped from 28.3 months to 24.5 months.
Strikingly, the time it takes to complete different
The analytical model used for the disbursement
phases has changed dramatically over the years.
effectiveness indicator of disbursements across
Projects that disbursed for the first time in the
project life was a dynamic random-effects
IFAD9 replenishment period (2013-2015) needed
generalized least squares model (see Wooldridge,
only 7.4 months to be declared effective – half the
2002). This was deemed the most appropriate
time needed during IFAD5 (2001-2003). At the
owing to its usefulness for analysing data
same time, projects in IFAD9 needed twice as
taken across multiple points over time. For the
much time to obtain a first disbursement after
assessment of factors influencing the amount
effectiveness as those in IFAD6. This change is
disbursed by project closure, which would have
probably an effect of the 2009 revision of the
only one data point for each project, a standard
General Conditions for Agricultural Development
ordinary least squares regression analysis was
Financing, which abolished effectiveness
deemed the most appropriate.
conditions but resulted in the addition of many
more first disbursement conditions to project
financing agreements.

Table 3.2 Disbursement readiness in IFAD’s past five replenishment periods

IFAD5 IFAD6 IFAD7 IFAD8 IFAD9

(2001-2003) (2004-2006) (2007-2009) (2010-2012) (2013-2015)

Board approval to
15.8 15.3 13.7 9.9 7.4
effectiveness

Effectiveness to first
5.3 5.1 5.3 8.2 10.2
disbursement

Subtotal (Board approval to

21.0 20.4 19.0 18.1 17.6
first disbursement)

First to second
7.3 9.4 7.4 8.3 6.9
disbursement

Total (Board approval to

28.3 29.8 26.4 26.4 24.5
second disbursement)

50
Table 3.3 Average time from approval to first disbursement by IFAD region and replenishment period

IFAD5 IFAD6 IFAD7 IFAD8 IFAD9

Region
(2001-2003) (2004-2006) (2007-2009) (2010-2012) (2013-2015)

Asia and the Pacific 19.6 16.1 17.7 15.6 14.8

PART I / Chapter 3
East and Southern Africa 19.3 17.6 15.5 16.6 14.2

Latin America and the

26.0 31.3 28.3 26.5 23.3
Caribbean

Near East, North Africa and

17.1 20.1 19.7 15.4 21.1
Europe

West and Central Africa 22.7 24.2 17.4 18.3 15.9

Regionally, the average time from approval to Table 3.4 Disbursement rates along the line
first disbursement improved across the board of best fit
over time, with the recent exception of the
Near East, North Africa and Europe region % planned time Disbursement rate (%)
(table 3.3). The West and Central Africa region’s
performance greatly improved from a peak of 10 2.8
more than 24 months in IFAD6 to 16 months in
20 8.6
IFAD9. Projects in the Asia and the Pacific and
in the East and Southern Africa regions also 30 16.7
managed to reduce the time lag by more than
30 per cent, from about 19 months in IFAD5 to 40 25.1

14-15 months in IFAD9. The Latin America and

50 33.4
the Caribbean region still has the highest lag
between approval and first disbursement, but 60 41.6
here as well disbursement readiness has steadily
70 49.4
improved since IFAD6.
80 57.0
Disbursement effectiveness
90 64.4
IFAD projects often run beyond their planned
length. The data on disbursement effectiveness 100 71.4
show that by the 100 per cent planned
completion point, the benchmark curve rate of 110 78.2
disbursement of funds reaches only 71.4 per cent
120 83.5
(figure 3.2 and table 3.4). In addition, figure 3.2
highlights the high amount of variation in the 130 91.3
disbursement performance of IFAD projects.
While values for the disbursement curve are not a 140 97.5

reflection of average values but a smoothed-out

150 100.0
stylized depiction, the figure shows that many
projects fall both well above and well below the
benchmark curve. Moreover, it is clear that IFAD

51
 Figure 3.2 Disbursement rates of all projects with line of best fit

100
80
Gross disbursement rate (%)
60
40
20
0

0 50 100 150
Percentage of planned length complete

Note: These data show the benchmark disbursement curve plotted across all of the disbursement rates of the
577 IFAD projects included in the sample, taken at intervals of 2.5 per cent of planned project length.

Figure 3.3 Disbursement rates at project closure by region

100
84.4 88.4 81.5 85 90.8 79.7
90
80
Gross disbursement rate (%)

70
60
50
40
30
20
10
0
All APR ESA LAC NEN WCA

Note: APR = Asia and the Pacific; ESA = East and Southern Africa; LAC = Latin America and the Caribbean;
NEN = Near East, North Africa and Europe; WCA = West and Central Africa.

52
Table 3.5 Results for country-level factors on disbursement readiness and effectiveness

Variables Disbursement readiness Disbursement effectiveness

Time to first Time to second Disbursement Disbursement

disbursement disbursement rate across rate at project

PART I / Chapter 3
project life closure

Country classification

Slower***
State fragility index (The higher, Lower*
ns (+14 days per ns
the ...) (-0.5%)
point)

Gross national income (GNI)

Slower** Slower*** - -
per capita (The higher, the ...)

Financing terms (The more grant Higher*** Higher***

- -
versus non‑grant financing, the ...) (+15%) (+12%)

Higher**
Small state (If yes, then ...) - - ns
(+13%)

Socio-economic and political variables

Official development assistance

(ODA) to agriculture (The higher, Quicker** (#) Quicker*** (#) ns ns
the …)

Higher*
Elections in current year Slower*** Slower*** (#)
ns (+0.9% per
(If yes, then …) (+84 days) (+141 days)
election)

General government revenue

Quicker** Quicker** - -
(The higher as % of GDP, the …)

Current account balance

Slower*** ns ns ns
(The higher as % of GDP, the …)

SDR/local currency exchange rate

Lower**
(The stronger SDR versus local - - -
(-3.2%)
currency, the …)

Events in country during implementation

Slower*** Slower*** Higher*

Number of extreme weather
(+8 days per (+11 days per ns (+0.2% per
events (The more events, the …)
event) event) event)

Slower*** Slower*
Number of epidemics
(+7 days per (+3 days per - -
(The more epidemics, the …)
event) event)

Higher**
Number of incidents of drought
- - ns (+0.7% per
(The more events, the …)
event)

Number of incidents of agricultural

disasters (The more events, - - ns ns
the …)

Notes: ns = not significant; # = level only significant in one of the two statistical models. – = variable not tested for
this indicator. SDR = special drawing rights. Coefficients in brackets are shown only if easy to interpret. Estimated
p-values of statistics: *** = p < 0.01 (meaning a 99% probability that the result is different from 0), ** = p < 0.05 (95%),
* = p < 0.1 (90%).

53
 Table 3.6 Results for project-level factors on disbursement readiness and effectiveness

Variables Disbursement readiness Disbursement effectiveness

Time to first Time to second Disbursement Disbursement

disbursement disbursement rate across rate at project
project life closure

Financing

Quicker***
Approved IFAD financing
(optimal: US$32 Quicker* - -
(The higher the amount, the …)
million)

First disbursement (as % of approved)

(The higher the first disbursement, - Quicker*** - -
the …)

Use of start-up funds

Slower*** (#) ns - -
(If used, then …)

Start-up advance (as % of approved)

- ns - -
(The higher the start-up, the …)

Cofinancing ratio Lower**

- - ns
(The higher, the …) (-0.4%)

Infrastructure financing
- - ns ns
(The higher as % planned, the …)

Credit financing
- - ns ns
(The higher as % planned, the …)

Soft activities financing

- - ns ns
(The higher as % planned, the …)

Management

Operational IFAD country office (ICO) Higher** Lower*

Slower* Slower*** (#)
(If yes, then …) (+4.6%) (-9%)

ICO is headed by country programme

Quicker** (#) Quicker*** (#) Higher***
manager (CPM) ns
(-62 days) (-9 days) (+24%)
(If yes, then …)

Project was directly supervised Higher***

- - ns
(If yes, then …) (+16%)

Country Programme Manager

CPM turnover (If CPM who designed

Quicker*** Quicker* (#)
the project is still in charge during ns ns
(-160 days) (-146 days)
start‑up, then …)

CPM experience (The more years of

Quicker*** (#) Quicker*** ns ns
IFAD experience the CPM has, the …)

CPM workload (The larger the portfolio

Slower** (#) Slower* (#) ns ns
managed by the CPM, the …)

CPM gender ns ns
(If the CPM is a woman, then …) (Quicker for time (Quicker from
ns ns
to entry into first to second
force**) disbursement**)

54
 Variables Disbursement readiness Disbursement effectiveness

Time to first Time to second Disbursement Disbursement

disbursement disbursement rate across rate at project

PART I / Chapter 3
project life closure

Financing agreement conditions

ns
(Slower for time to
Ratification of financing agreement entry into force***)
ns - -
(If required, then …) (Quicker for entry
into force to first
disbursement***)

ns
Financing agreement was signed by (Slower for
ns - -
correspondence (If yes, then …) approval to entry
into force***)

ns
Project implementation manual (Quicker for
ns - -
required (If yes, then …) first to second
disbursement*)

Number of accounts required

(The higher the number of accounts, ns ns - -
the …)

Project management unit required Quicker***

ns - -
(If yes, then …) (-61 days)

Project director required

ns ns - -
(If yes, then …)

Number of conditions (The higher the

ns ns - -
number of conditions, the …)

Note: ns = not significant; # = level only significant in one of the two statistical models. – = variable not tested for this
indicator. Coefficients in brackets are shown only if easy to interpret. Estimated p-values of statistics: *** = p < 0.01
(meaning a 99% probability that the result is different from 0), ** = p < 0.05 (95%), * = p < 0.1 (90%).

55
 projects are typically slow starters, with the curve likelihood of fully disbursing the approved project
reaching around just 33 per cent by the halfway budget, and the need to do so. Elections and
mark of planned project length. natural disasters have a negative effect during
early project stages but a positive effect in the
For various reasons, IFAD projects do not have
long term. This result may occur because a
to have disbursed their total approved budget
high incidence of such events generates more
by the time of financial closure, and undisbursed
demand for spending the available resources, but
balances are returned to IFAD. In fact, the
such demand is offset by the disruption caused if
average amount actually disbursed across all
they occur during project start-up.
projects is 84.4 per cent (figure 3.3). Across
regions, the Near East, North Africa and Europe On the IFAD side, the results show that the
stands out with the highest rate, at 90.8 per cent, size, type and sequencing of project financing
followed by projects in Asia and the Pacific all play a role. Projects with larger budgets
with 88.4 per cent, and Latin America and disburse faster. However, during implementation,
the Caribbean with 85.0 per cent. The closed IFAD’s disbursement rates suffer from excessive
portfolio for projects in sub-Saharan Africa lags dependence on cofinancing. Although the
behind, with West and Central Africa reaching a study also finds that start-up advances do not
disbursement rate below 80 per cent. accelerate first disbursements, it shows that
a relatively high first disbursement is a good
predictor of a more rapid second disbursement.
Explaining IFAD’s disbursement This finding suggests that IFAD’s efforts to support
the submission of larger and better‑quality
performance
withdrawal applications are bearing fruit.
Tables 3.5 and 3.6 present the findings of the
Furthermore, the variables over which IFAD has
econometric analysis for the country- and
the most control clearly make a difference. The
the IFAD‑side factors affecting disbursement
results assert that having offices in the field (IFAD
readiness and effectiveness indicators. R-squared
country offices [ICOs]) with nationally recruited
statistics are a measure of the strength of an
IFAD staff (country programme officers) benefits
analytical model. In this case, they show that the
IFAD’s disbursement performance during
included variables explain more than 60 per cent
implementation. ICOs headed by outposted
of the variation for the disbursement readiness
country programme managers (CPMs) are
models and 45 per cent of the variation for the
also shown to have a positive impact on both
effectiveness models.
disbursement readiness and effectiveness.
On the country side, the analysis finds that the Moreover, signing financing agreements
likelihood of disbursements being delayed during face‑to‑face (rather than by correspondence)
start-up significantly increases if a country is a is shown to accelerate the process and almost
fragile middle-income country, has constrained offsets the delays resulting from ratification
fiscal space (measured by government revenue requirements. Finally, the study shows that the
and current account balance), or is in an turnover, experience and workload of CPMs
election year. In addition, non-fragile low-income are important elements of IFAD’s disbursement
countries, countries with a stable local currency, readiness. If the CPM remains the same at
and those that have had a higher frequency of approval and during start-up of a project, the
elections and natural disasters (e.g. droughts) first and second disbursements are roughly
during the project life tend to have higher five months quicker than in cases of turnover.
disbursement rates. Thus, the study confirms Performance is further enhanced if the CPM is
the hypothesis by Ahsan and Gunawan (2010) experienced, is not overloaded with work and/or
that a devalued national currency decreases the is a woman.

56
Concluding remarks showed that the gender, experience, workload,
location and amount of turnover of IFAD’s CPMs
The study presented in this chapter is the first affected disbursement performance. Investments
in-depth investigation of the trends and drivers of capitalizing on this knowledge are thus likely to
IFAD’s disbursement performance. It is also the bear fruit. In addition, the results for disbursement
most extensive study of this area conducted for readiness show that financing agreement

PART I / Chapter 3
any IFI. It reveals areas where IFAD, and perhaps conditions constitute a powerful tool for shaping
other IFIs, can improve their performance in disbursement performance; these conditions
disbursing funds for development projects, and should be carefully considered to ensure more
thereby achieve better end results. streamlined project preparation.
Indeed, the analysis of disbursement trends Although country-level factors are less under the
highlights the need to do better. From 2003 control of funding agencies, findings in this area
to 2015, the time between approval and first also have implications for future decisions and
disbursement improved only slightly, from policy. Factors such as state fragility, constrained
21.0 months to 17.6 months. IFAD-supported fiscal space and a devalued currency were
projects disbursed, on average, only 84.4 per cent shown to hinder disbursement performance.
of their approved budget, albeit with significant IFIs should adjust their disbursement performance
differences across countries and regions. On a expectations accordingly, so that when these
year-by-year basis, projects were generally hindrances occur they are not a surprise and
slow in disbursing funds, reaching a rate of only thus have a less debilitating effect on a project’s
33 per cent at the halfway point of their originally implementation and impact.
planned implementation schedule, and not
IFAD has already taken on board several of the
more than 71 per cent at the originally planned
lessons of this study. IFAD management has
completion date.
approved a corporate action plan aimed at better
These trends, according to the study, result projecting, measuring, monitoring and managing
from several factors under the control of the Fund’s disbursements. All actions have a
recipient countries and IFAD itself. These factors specific timeline, as well as clear responsibilities
– especially the internal factors over which a and measures of success. A corporate workshop
funding organization such as IFAD has control held in early 2017 further raised awareness
– have implications for measuring and improving among all responsible staff about the importance
the disbursement performance of IFAD and of disbursement performance to demonstrate
other IFIs. Perhaps the strongest message is efficient use of Member States’ contributions and
that larger projects perform better, especially in to ensure that IFAD clients – and especially target
terms of disbursement readiness. This message beneficiaries – receive adequate support.
is supported by previous studies on disbursement
Looking ahead, measures to enhance IFAD’s
performance in the African Development Bank and
disbursement performance will include revisiting
the Inter-American Development Bank (see Gohou
policies (e.g. cancellation policy and restructuring
and Soumaré, 2009; Nkamleu, Tourino and Edwin,
policy) and reviewing current products, processes
2011; Álvarez, Bueso-Merriam and Stucchi, 2012).
and procedures. IFAD will need to give critical
In addition, relationships in which one cofinancier
attention to ex ante disbursement forecasts and
shoulders a much higher burden than others
the disbursement dimension of projects, aiming
can lead to disbursement problems for all parties
for a balance of project types within each country
involved if the most important cofinancier performs
portfolio. Finally, to help strengthen borrowers’
poorly. Hence, careful consideration should be
readiness to implement projects, IFAD will make
given to how financiers share costs.
more targeted use of start-up advances and focus
Other controllable factors include the role of more on procurement issues in project design.
project managers, as also shown by Álvarez, The results of the analysis are facilitating the
Bueso-Merriam and Stucchi (2012). The analysis management of disbursement performance.

57
 References Nkamleu, G. B., Tourino, I. and Edwin, J. 2011.
Always late: Measures and determinants of
Ahsan, K. and Gunawan, I. 2010. Analysis of disbursement delays at the African Development
cost and schedule performance of international Bank. Working Paper No. 141. Tunis: African
development projects. International Journal of Development Bank.
Project Management, 28: 68-78.
Sunjka, B. P. and Jacob, U. 2013. Significant
Álvarez, C., Bueso-Merriam, J. and Stucchi, causes and effects of project delays in the Niger
R. 2012. So you think you know what drives Delta region, Nigeria. Paper prepared for the
disbursements at the IDB? Think, think again... 25th annual conference of the Southern African
Technical Note No. IDB-TN-479. Washington, Institute of Industrial Engineering, July 9-11,
D.C.: Inter-American Development Bank. Stellenbosch, South Africa.
Cameron, A. C. and Trivedi, P. K. 2005. Wooldridge, J. 2002. Econometric analysis of
Microeconomics: Methods and applications. cross section and panel data. Cambridge, USA:
Cambridge: Cambridge University Press. MIT Press.
Gohou, G. and Soumaré, I. 2009. The impact of
project cost on aid disbursement delay: The case
of the African Development Bank,
www.afdb.org/fileadmin/uploads/afdb/
Documents/Knowledge/2009%20AEC-%20
The%20Impact%20of%20Project%20Cost%20
on%20Aid%20Disbursement%20Delay%20
The%20Case%20of%20the%20African%20
Development%20Bank.pdf

Kersting, E. K. and Kilby, C. 2015. With a little

help from my friends: Global electioneering and
World Bank lending. Journal of Development
Economics, 121: 153-165.

Mishra, P. K. 2016. Managing international

development projects: Case studies of
implementation of large-scale projects in India.
International Journal of Rural Management,
12(1): 4-26.

58
Chapter 4

Assessment of IFAD’s

PART I / Chapter 4
end-of-project documentation

by
Alessandra Garbero

In the past couple of decades, as international This study sought to examine how well claims
development discourse and practice have about project results were supported by evidence
focused more heavily on improving development in a sample of PCRs from IFAD. Using an
effectiveness, the role of evidence-based reporting IFAD‑specific conceptual framework to capture
on results has also gained prominence. The the diversity of activities and potential impacts of
development community has recognized the IFAD investments, it classified claims about project
need to focus on results rather than inputs and results according to their themes, the results level,
to ensure that a project’s processes contribute and the presence and types of evidence sources.
to the achievement of its objectives (Flint, 2003).
Multiagency commitments such as the Paris
Declaration on Aid Effectiveness (OECD, 2016) Study design
have underlined issues around the quality, impact
and accountability of development interventions. Under the Results Management Framework (RMF)
covering the period between 2013 and 2015,
At a practical level, several international
IFAD committed to providing a broader and more
development agencies have established
in‑depth assessment of the Fund’s impact on
development effectiveness frameworks that
poverty reduction. This study was originally part of
seek to integrate project data into a learning
a larger impact evaluation initiative that employed
process (ACFID, 2015). Within these frameworks,
a mixed-methods approach to assess IFAD’s
project completion reports (PCRs) are the main
contribution to movements out of poverty across
instrument many development institutions use
the project portfolio. Among other things, it aimed
to report on results. They tell the story of a
to verify the hypothesis that because PCRs – as
project, make sense of data generated during the
standardized documents that take stock of project
project cycle, and discuss the project’s success
performance – play an important role within the
at achieving its development objectives, the
project cycle and in broader institutional planning,
challenges or failures it encountered, and the
they are based on robust evidence.
sustainability of interventions. They are crucial
tools for internal and external accountability as To accomplish this goal, categories of analysis
well as for learning from experience to inform the were divided into two distinct types: case coding,
design of future operations (Crespo et al., 2013). and thematic coding. Case coding was used
As such, these documents should provide solid to characterize the claims, and thematic coding
evidence of a project’s effectiveness in bringing described the subject of the claims; the two were
about development. combined for a comprehensive analysis. In order

59
 to ensure consistency and reliability during the capital”, which captures outputs related to
coding process, a detailed codebook that defined training, demonstrations and model farming.
the parameters that coders should consider for Yet the medium-term outcomes of such
each of the indicators within the categories of capacity‑building initiatives, which were grouped
analysis was developed for both case coding and under the theme “capacity to innovate”, received
thematic coding. significantly fewer claims, revealing a lack of
reporting on how well people applied the skills
The characteristics of claims were categorized into
they learned through the extensive training.
four types: claim direction, claim support, claim
quantification, and evidence source. Thematic At the bottom of the list, representing just
coding described the subject or the content of a 1.5 per cent of claims, was “resilience capacity”,
claim, representing the most important category which encompassed several impact-level
of analysis by organizing the various results indicators related to the ability of individuals,
associated with IFAD’s projects by themes. households and communities to cope with
The ten themes were: (1) agricultural production, shocks. “Women’s empowerment”, another
(2) capacity to innovate, (3) commerce and value theme that pertained to longer-term results, was
chain, (4) economic mobility, (5) environmental mentioned in just 6 per cent of all claims.
sustainability, (6) food security and health, Across all themes, 71 per cent of claims in the
(7) human capital, (8) policies and institutions, sample were not explicitly supported by a source
(9) resilience capacity, and (10) women’s of evidence. The “commerce and value chain”
empowerment. This categorization system theme had the fewest explicitly supported claims:
allowed for the analysis of results according to the 77 per cent were either partially supported
recurring themes found in the theories of change or not supported at all. Indeed, in both of the
of IFAD-funded projects, as well as aligned with predominant themes – “commerce and value
the Fund’s broader rural development mission. chain” and “economic mobility” – most claims
When combined with the case coding, findings were either not supported or only partially
across themes could be further contextualized and supported by evidence. Conversely, “food
described across all case coding categories. security and health” had the largest share of
supported claims, 41 per cent.
IFAD PCRs were the principal source for analysis.
After defining and applying eligibility criteria, the The analysis also looked at the level of
research team coded 72 project documents. results reported. The vast majority of claims
– 96 per cent – report on relatively immediate
outputs and outcomes rather than long-term
Results impacts. Still, although impact claims have such
a negligible presence across PCRs, 49 per cent
The research team coded almost 4,000 unique of such claims are supported by a source of
claims across the sample of 72 projects. They evidence, compared with just 36 per cent
found considerable variation in the rates at which of outcomes and 22 per cent of outputs.
claims were made for different themes, as well as in Medium‑term changes account for the largest
the degree of evidence used to support the claims. level of claims – more than 930 outcome‑level
The “commerce and value chain” theme claims are backed by direct evidence, and
tallied more claims than any other category a similar number are backed by “partially

– a result that reflects IFAD’s market-oriented supported” results. Across the sample, outputs
were rarely supported by direct evidence; more
strategy. “Economic mobility” was a close
than 1,400 claims, or 66 per cent, at this result
second; although this theme is aligned with
level were coded as “partially supported”.
IFAD’s goal of reducing poverty, reporting in
this area included few mentions of longer-term The research team also categorized the sources
impacts such as moving people out of poverty. of evidence. Many of the data came from external
The third-most frequent theme was “human sources rather than IFAD’s own efforts to collect

60
impact data. The share of claims supported This inefficient use of evidence in PCRs
by IFAD’s Results and Impact Management makes it difficult to gauge the successes and
System (RIMS) rose from 2 per cent in 2010 to achievements of projects. Although these
32 per cent in 2014. However, this rate is still low, documents often compile the full extent of
especially considering that external, non-RIMS project reporting, their findings can be opaque,
surveys served as sources of evidence almost and the large amounts of unclear information

PART I / Chapter 4
as often as RIMS data until 2013. Reliance on sometimes obscure significant results or learning.
PCR missions for data collection decreased Moreover, the positive reporting bias confounds
10 percentage points, from 29 per cent in 2010 the transparency of project results. This bias is
to 19 per cent in 2014. This shift could point to structural to internal project reporting processes,
improved attempts to collect data throughout as PCR authors are more likely to exaggerate
the life of the projects rather than relying on positive results while overlooking neutral and
end-of-programme support. Nevertheless, negative ones for fear of losing subsequent
PCRs overwhelmingly rely on project monitoring funding. The result is that learning from project
and evaluation (M&E) data for reporting. This is success is not balanced with learning from
problematic given that many of IFAD’s project project failures. Such a bias could be minimized
M&E systems were not designed to support through the use of objective indicators.
reporting on longer-term results. Sixty-six per
Because the quality of reporting largely depends
cent of project M&E data were related to outputs,
on the quality of data available – which in
24 per cent to outcomes, and only 9 per cent
turn depends on the capacity of the project
to impacts.
management team – agencies would benefit from
Finally, claims made were overwhelmingly encouraging the incorporation of robust impact
– 84 per cent – positive. The lack of neutral evaluation strategies into project design.
claims (at just 5 per cent of the total) shows
insufficient reporting on indicators that have
not been affected by IFAD initiatives. About
References
11 per cent of claims were negative. This
overall pattern indicates a reporting bias that ACFID (Australian Council for International
reduces the transparency of project results and Development). 2015. Developing an effectiveness
hinders the effective learning that could arise framework: A toolkit for small and medium sized
out of PCRs. NGOs. Canberra.

Crespo, A., Hallberg, K., Ruiz, N., Martin, L.,

Alonso, D., Zambrano, P., Dal-Toe, T., et al. 2013.
Conclusion IDB-9: The development effectiveness framework

This study highlights a significant need for IFAD and overview. Washington, D.C.: Inter-American
to incorporate evidence-based results into its Development Bank.
end-of-project reporting processes. Current Flint, M. 2003. Easier said than done: A review
reporting suffers from a lack of clear evidence and of RBM in multilateral development institutions.
from an unbalanced focus on immediate results. Herefordshire, UK: UK Department for
The sources of evidence used to support project International Development (DFID).
claims rely too heavily on project M&E systems
OECD (Organisation for Economic Co‑Operation
that are ill-equipped to provide data on outcomes
and Development). 2016. Paris Declaration
and impacts. Despite the availability of RIMS – the
and Accra Agenda for Action. Paris, www.
precursor to the Operational Results Management
oecd.org/development/effectiveness/
System (ORMS) – as a tool for impact reporting,
parisdeclarationandaccraagendaforaction.htm
compliance with its procedures remains low, which
in turn affects the quality of reporting.

61
©IFAD/Susan Beccio

62
PART II

Assessing project-level impact

63
 Chapter 5

Ethiopia’s Participatory
Small‑Scale Irrigation Development
Programme (PASIDP)
by
Alessandra Garbero
Tisorn Songsermsawas

Ethiopian agriculture faces a number of challenges to improve the food security and incomes of
that prevent smallholder farmers from attaining households in project communities.
higher levels of productivity. Although average
The total cost of the project was US$57.8 million,
rainfall in Ethiopia is higher than in the rest of
of which about 70 per cent was supported
sub-Saharan Africa, farmers’ production is
by IFAD as a grant and a highly concessional
often affected by frequent drought, severe soil
loan. The remaining portion of the project cost
degradation, unpredictable weather patterns,
was funded by the Government of Ethiopia as
and a lack of irrigation, transportation and market
well as by direct payments of water user fees
infrastructure (Escobal, 2005; Kassahun, 2007;
to the water user associations (WUAs). This
Matouš, Todo and Mojo, 2013). Because these
study examines the evidence on the impacts of
smallholder farmers’ main source of income is their
this project and considers lessons that can be
agricultural production, these challenges have direct applied to similar future projects.
implications for their welfare and poverty outcomes.

The Government of Ethiopia has addressed some

of these challenges by promoting investments The project’s theory of change
in small-scale irrigation systems, as put forward
in the country’s Poverty Reduction Strategy PASIDP institutionalized a participatory
Paper 2005/06. One such investment was the approach to small-scale irrigation development

Participatory Small-Scale Irrigation Development in rural Ethiopia by helping members of project

Programme (PASIDP), a project funded by IFAD, communities form WUAs. These WUAs consisted

which took place between 2008 and 2015. PASIDP of farmers whose agricultural plots were located

was designed to develop small-scale irrigation within range of the project’s irrigation systems
and were suitable for crop cultivation. WUA
infrastructure systems in four regions: Amhara,
members pooled their financial, technical,
Oromia, Tigray and Southern Nations, Nationalities
material and labour resources to plan, operate,
and Peoples Region. PASIDP had three main
maintain and repair the irrigation systems. WUAs
components: (1) institutional development;
elected their leaders and committee members,
(2) small-scale irrigation development; and
collected water fees, maintained the irrigation
(3) agricultural development. Through these
systems, and settled water-related conflicts and
components, PASIDP sought to develop
disputes within the WUAs.
sustainable irrigation systems and empower project
communities by engaging community members in The project also built new irrigation schemes
designing, constructing, operating, managing and or rehabilitated existing schemes, including
repairing such systems. The project was expected river diversion, spate irrigation, spring sourcing

64
and shallow wells. Finally, as part of the project farmers’ productivity and income. Irrigation can:
interventions, capacity-building and training lead to more diverse income or livelihood strategies
activities related to best practices in agricultural (Smith, 2004); buffer the effects of drought (Prasad,
production were delivered to farmers in the Van Koppen and Strzepek, 2006); increase access
project communities. to output markets (Gidwani, 2002; Dorward et al.,
2004); raise demand for wage labour on irrigated
PASIDP interventions were expected to improve
farms (Narayanamoorthy and Deshpande, 2003);
the outcomes of households that received the
reduce fluctuations in output prices (Lipton,
project through a number of possible channels.
Litchfield and Faurès, 2003); and increase resilience
Most importantly, an irrigation system can help
to exogenous shocks and adverse weather events
improve farm productivity by increasing crop
(Barrett, Reardon and Webb, 2001). Figure 5.1
yields as farmers receive a greater overall volume
illustrates the theory of change of the PASIDP
of water for cultivation, a more consistent water
project, presenting the project’s causal chain from
supply, and timely access to water when they
inputs to outputs, outcome and impacts.
most need it for crop production (Hussain and
Hanjra, 2003). In addition, irrigation allows farmers Based on this theory of change, the research
to cultivate a larger area and cultivate crops over team designed an ex post impact assessment
multiple rotations in a given year (Huang et al., to investigate whether households with access
2006). Access to irrigation may also complement to irrigation systems developed under PASIDP
farmers’ use of inputs including improved seeds, exhibited higher farm productivity and welfare
fertilizer and pesticides (Evenson and Gollin, outcomes than those in a comparison group – that
2003). Previous studies have noted other possible is, a group of households facing similar agroclimatic

PART II / Chapter 5
channels through which irrigation can improve conditions and with similar socio‑economic

Figure 5.1 The theory of change of PASIDP

• Formation of water user associations (WUAs)

• Training of WUA leaders and members about water management and water
Inputs/activities distribution/allocation
• Provision of agricultural capacity‑building and training services
• Construction of irrigation infrastructure systems

• Greater access to water available for crops (through construction of irrigation

infrastructure system)
• More efficient water management (through formation and training of WUA
leaders and members)
• More efficient water distribution/allocation (through formation and training of
Outputs
WUA leaders and members)
• Greater ability to adopt risk management and risk coping strategies (through
construction of irrigation infrastructure system)
• Improved access to information about agricultural knowledge and technology
(through training activities)

• Adoption of new and more suitable agricultural technology

• Expansion of cultivation areas (extensive margin) and crop rotations (intensive
margins)
Outcomes • Increased crop production / higher yields of crop
• More higher-valued crops cultivated
• Increased agricultural diversification
• Greater resilience against negative shocks

• More stable and increased household income

Impacts • Higher ability to adopt risk coping and risk management strategies against
negative shocks

65
 characteristics that did not access PASIDP could be compared to measure the project impact.
irrigation systems. Specific focus was devoted to To establish valid comparison groups, researchers
assessing the project’s impact on crop revenues used a number of characteristics that should have
and on household consumption levels (in terms been uninfluenced by the project. These included:
of both food and non-food expenditures). household size; sex, age and education level of
The impact assessment complements existing the household head; an asset‑based wealth index
studies that explore the extent to which changes based on recalled measures prior to the project
in household-level productivity and welfare start (with durable, livestock and productive assets);
outcomes are attributable to the development landownership; elevation; average precipitation
of small-scale irrigation infrastructure systems during the past five years; and time employed by
(Hamdy, Abu-Zeid and Lacirignola, 1998; Del farmers to reach the market.
Carpio, Loayza and Datar, 2011; Dillon, 2011).
Then, based on the information collected in the
enumeration survey, the household survey was
carried out with one treatment group (households
Project outreach and outputs with access to PASIDP-built irrigation schemes)
and two comparison groups (those using water
Before proceeding to explore the project impact,
from traditional irrigation sources, and those
an analysis was conducted of the project’s
relying solely on rainfall for agricultural production).
coverage, the types of people reached, and
the outputs generated as a result of the project To further ensure that the households in the

interventions. Between 2008 and 2015, PASIDP two comparison groups were comparable to
interventions facilitated the formation of 175 the ones in the treatment group, extensive
WUAs and developed 116 small-scale irrigation discussions with the staff members from the
schemes that covered at least 12,020 hectares project management unit and the development
of arable land. The project interventions were agents office were held to validate the selection
estimated to have reached more than 311,000 of the households to be part of the comparison
people in 62,200 households in the four regions groups. Of the 1,531 households in the full sample,
covered by the project. 766 (50 per cent) were households with access
to PASIDP‑supported irrigation systems, 438 (29
per cent) were traditional irrigation users, and 327
(21 per cent) relied mainly on rainfall for agriculture.
Project impact
Qualitative data were used to support and
Data and methods triangulate the findings from the quantitative
household surveys. These qualitative surveys
The impact assessment drew on both quantitative
consisted of key informant interviews with kebele
and qualitative data. The quantitative data
leaders and development agents and focus
consisted of a household survey collected from
group discussions with households in project and
1,531 households in 20 kebeles (or subdistricts) in
comparison kebeles. In total, 24 key informant
the four Ethiopian regions. To identify households
interviews (six in each project, four in treatment
similar to the ones receiving the project as potential
kebeles and two in comparison kebeles), and
candidates for the comparison group, the research
24 focus group discussions (six in each region,
team first conducted an enumeration survey to
four in treatment kebeles and two in comparison
collect information about agricultural production,
kebeles) were administered.
poverty status, agroclimatic conditions, and source
of irrigation in all 20 kebeles. The purpose of this The impact of the PASIDP project was estimated
enumeration survey was to help ensure that the using a non-experimental approach. Researchers
characteristics of the households in the comparison looked at the treatment group and the two
group (those not receiving the project) were similar comparison groups in pairs to compare the
to those in the treatment group (those receiving observed outcomes of households in each
the project) so that the outcomes of such groups group. This approach allowed them to observe

66
Figure 5.2 Average crop yield (factor)

Rainfed agriculture

Traditional irrigation

PASIDP

0.0 0.5 1.0 1.5 2.0

Figure 5.3 Per capita food expenditure (factor)

Rainfed agriculture

Traditional irrigation

PASIDP

0.0 0.5 1.0 1.5 2.0

PART II / Chapter 5
the gradient of effects ranging from farmers with (improved seeds, fertilizers and pesticides), size
access to PASIDP-supported irrigation, to those of crop cultivation area, or number of crops grown,
with traditional irrigation only, to those relying only the results provide evidence that the increase in
on rainfall. revenues observed among PASIDP households
was driven largely by the increase in crop yields.

For outcomes related to household consumption,

Key quantitative findings and particularly food consumption expenditures,

The key impact estimates centred on indicators results indicated similar impact magnitudes
of agricultural production and household (figure 5.3). Relative to households relying mainly
consumption, and the impacts in these areas on rainfall, the per capita food expenditures of
were sizeable. In terms of agricultural production, PASIDP households were twice as high, and
the largest impact was on average crop yields those of farmers that relied on traditional irrigation
(figure 5.2). Compared with households that were 50 per cent higher. However, the project had
relied mainly on rainfed agriculture (the reference no significant impact on non-food expenditures.
group in the figure), PASIDP households had This impact assessment also explored project
average crop yields that were twice as high, and impacts on other outcome indicators. These
traditional irrigation users 20 per cent higher. indicators included: poverty measures (whether
PASIDP had similar impacts on crop revenues. or not the household was above asset-based
Households using PASIDP-supported irrigation and expenditure-based poverty lines); asset
doubled their revenue compared with rainfed accumulation (durable assets and productive
farmers, and those using traditional irrigation had assets); and measures of female empowerment
crop revenues 30 per cent higher than those (level of participation in agricultural-related
relying only on rainfall. Given that the project decisions and income-related decisions).
had no significant effects on other production However, no significant effects of the project were
outcomes, such as use of physical inputs observed on such outcome indicators.

67
 Key qualitative findings this project, PASIDP – Phase II, has included these
lessons in its design and implementation stages.
Key informant interviews with kebele leaders and
The main lessons learned can be summarized
development agents highlighted the importance
as follows:
of forming WUAs before constructing the irrigation
• Returns on investment in irrigation
schemes. This approach engaged potential
infrastructure systems can be transformative.
water users in developing the irrigation schemes
By raising agricultural revenues, mainly through
in an organized manner according to their local
higher crop yields, the project increased
needs. Community members contributed labour,
household income and consumption levels of
construction materials and cash to construct the
treated households. Relative to farmers who
irrigation schemes. In addition, capacity-building
relied only on rainfall for agricultural production,
and training activities on water management,
the crop yields of those with access to
operation and maintenance of irrigation schemes,
PASIDP-supported irrigation doubled.
WUA rules and regulations, and bookkeeping skills
• Better-off farmers, who were usually more
were delivered to WUA leaders and committee
productive farmers, benefited the most
members to ensure efficient management of
from the investments in irrigation in terms
WUAs. After the training, project staff members
of increased crop yields and crop revenues.
transferred responsibility for managing and
Interventions that benefit mostly more
maintaining the irrigation schemes to WUAs.
productive farmers may raise questions
Interviews with kebele leaders and development about equity, so this finding implies the
agents revealed a number of capacity gaps within need for other complementary interventions
the WUAs. WUAs commonly faced difficulties for farmers with less productive capacity.
collecting membership and water-use fees, For example, climate-smart solutions
mobilizing financial resources, and controlling compounded with irrigation could be a
members’ water use. Several incidents of water potential intervention for worse-off farmers
theft and conflicts over water use took place. Key who live in the watersheds and depend
informant interviews also suggested that training on rainfed agriculture.The main channel
in agricultural marketing would have been useful through which access to irrigation raises
for those who received the project. However, crop revenues is through higher crop yields.
such training was not contemplated in the training The project had no significant effects on
package provided by PASIDP. other intermediate channels such as input
Focus group discussions with those who received expenditures (seed, fertilizer and pesticide),
the project confirmed that project benefits took the cultivation area or number of crops cultivated
form of increased agricultural productivity through in a season. Follow-up projects should
improved yields, and promotion of new agricultural devote further attention to these aspects.
technologies and crop cultivation practices through • PASIDP fostered a strong participatory
strengthened extension services and formation of approach that engaged community members
farmer research groups. However, the discussions in forming WUAs and involved them in
also revealed several development challenges, planning, designing, constructing, managing
including lack of access to input and output and maintaining the irrigation systems.
markets, high input prices, and low crop prices. This approach helped create a sense of
ownership within the community and was
crucial to the project’s sustainability.
• Although improved access to irrigation can
Lessons learned
help produce higher levels of agricultural
This ex post impact assessment of PASIDP offers outputs, farmers may not be able to take full
a number of lessons for future project design and advantage of the project interventions if they
implementation, as well as for future policy and cannot sell their crops at the market at the right
country strategies. Notably, the scale-up phase of time. To help farmers manage post-harvest

68
 losses, interventions that strengthen market Escobal, J. 2005. The role of public infrastructure in
access and agricultural marketing should be market development in rural Peru. PhD dissertation,
included along with irrigation development Wageningen University.
interventions. A key lesson learned is that
Evenson, R. E. and Gollin, D. 2003. Assessing the
projects need to take into account markets
impact of the Green Revolution, 1960 to 2000.
and viable crop models to meet these
Science, 300(5620): 758-762.
markets. The participatory processes during
project design and implementation must allow Gidwani, V. 2002. The unbearable modernity of
farmers to understand their options and the development? Canal irrigation and development
financial implications in terms of production planning in Western India. Progress in Planning,
cost. At the same time, the project should 58(1): 1-80.
facilitate linkages with aggregators, service Hamdy, A., Abu-Zeid, M. and Lacirignola, C. 1998.
providers, financial institutions and input Institutional capacity building for water sector
dealers, among others, to build transparent development. Water International, 23(3): 126-133.
and trusting relationships between all of these
value chain operators. These relationships Huang, Q., Rozelle, S., Lohmar, B., Huang, J. and

can form the basis on which marketing Wang, J. 2006. Irrigation, agricultural performance

associations (or cooperatives) can elaborate and poverty reduction in China. Food Policy, 31(1):
concrete production plans (agricultural 30-52.
development plans). Hussain, I. and Hanjra, M. A. 2003. Does irrigation
• Project implementation activities experienced water matter for rural poverty alleviation? Evidence
a number of delays. One approach to from South and South-East Asia. Water Policy,

PART II / Chapter 5
address this issue is to recruit and train 5(5-6): 429-442.
project staff early on, and to involve relevant
Kassahun, D. 2007. Rainwater harvesting in
stakeholders in the early stages of planning.
Ethiopia: Capturing the realities and exploring
The roles and responsibilities of project staff
opportunities. Working paper. Addis Ababa: Forum
and other stakeholders should be well defined
for Social Studies.
within a strict implementation timeline.
Lipton, M., Litchfield, J. and Faurès, J.-M. 2003.
The effects of irrigation on poverty: A framework for
References analysis. Water Policy, 5(5-6): 413-427.

Barrett, C. B., Reardon, T. and Webb, P. 2001. Matouš, P., Todo, Y. and Mojo, D. 2013. Roles
Non-farm income diversification and household of extension and ethno-religious networks in
livelihood strategies in rural Africa: Concepts, acceptance of resource-conserving agriculture
dynamics, and policy implications. Food Policy, among Ethiopian farmers. International Journal of
26(4): 315-331. Agricultural Sustainability, 11(4): 301-316.

Del Carpio, X. V., Loayza, N. and Datar, G. 2011. Is Narayanamoorthy, A. and Deshpande, R. 2003.
irrigation rehabilitation good for poor farmers? An Irrigation development and agricultural wages:
impact evaluation of a non‐experimental irrigation An analysis across states. Economic and Political
project in Peru. Journal of Agricultural Economics, Weekly, 38(35): 3716-3722.
62(2): 449-473.
Prasad, K. C., Van Koppen, B. and Strzepek,
Dillon, A. 2011. The effect of irrigation on poverty K. 2006. Equity and productivity assessments
reduction, asset accumulation, and informal in the Olifants River basin, South Africa. Natural
insurance: Evidence from Northern Mali. World Resources Forum, 30(1): 63-75.
Development, 39(12): 2165-2175.
Smith, L. E. D. 2004. Assessment of the
Dorward, A., Kydd, J., Morrison, J. and Urey, I. contribution of irrigation to poverty reduction and
2004. A policy agenda for pro-poor agricultural sustainable livelihoods. International Journal of
growth. World Development, 32(1): 73-89. Water Resources Development, 20(2): 243-257.

69
 Chapter 6

Kenya’s Smallholder Dairy

Commercialization Programme
by
Romina Cavatassi
Peter Brueckmann
Nancy McCarthy
Juan Bonilla

Agriculture is the largest contributor to Kenya’s was designed to increase the income of
GDP, and it employs 61 per cent of the country’s resource‑poor rural smallholders who depend
workforce, mainly represented by smallholder substantially on production and trade of dairy and
farmers living in rural settings (Muriuki, 2011; dairy-related products for their livelihoods in nine
FAO, 2014). Within agriculture, the country’s dairy commercialization counties in central and
livestock dairy sector has special importance. central-western Kenya. To this end, the project
Kenya is one of the largest milk-producing aimed to address constraints to smallholders’
countries in Africa – it currently has more than milk production, productivity and participation
5.6 million dairy cattle, accounting for about in milk markets by training dairy groups, offering
15 per cent of East Africa’s dairy livestock (FAO, technical support for household dairy production,
2017). Investing in smallholder dairy farming is and developing milk marketing chains. Special
an effective way to improve farmers’ production emphasis was placed on ensuring women’s
and commercialization, which in turn can help participation in all project activities, given that
alleviate poverty and increase food security women play a key role in dairy production in
(Burke et al., 2007; Muriuki, 2011; Olwande et al., Kenya and have been traditionally disadvantaged
2015; Randolph et al., 2007; Smith et al., 2013). (e.g. in terms of education and access to
The need to invest in and improve Kenya’s dairy credit). The first phase of this programme was
industry was especially strong after the collapse implemented from 2005 to 2015 at a total cost
of the monopolistic dairy cooperative structure of US$19.75 million, jointly funded by IFAD,
in the 1990s (Muriuki, 2011), when farmers the Government of Kenya and beneficiary
faced high transaction costs for production communities. In December 2015, IFAD decided
and marketing (Staal, Delgado and Nicholson, to extend this project, with completion expected
1997), and dairy groups and cooperatives were in September 2019.
unable to address these issues (Holloway et al.,
A number of studies and impact assessments
2000). Smallholder dairy farmers faced additional
of interventions in the dairy sector in Kenya have
barriers to improving their dairy production,
shown that technical support to dairy groups
including poor and unreliable quality of feed, lack
and farmers, as well as stronger market linkages,
of access to animal health and breeding services,
can have important and positive impacts on the
loss of milk production, and inadequate access
livelihoods of smallholder dairy farmers. Given
to milk markets (Atieno and Kanyinga, 2008).
the great relevance of dairy cattle as a source of
The Smallholder Dairy Commercialization food consumption and income for smallholder
Programme (SDCP), which started in 2005, dairy farms in Kenya and East Africa as a whole

70
(Bingi and Tondel, 2015; Muriuki, 2011), the and better management of fodder and feed
SDCP has been selected as the subject of a (Nafula, 2013; Richards et al., 2016); and better
rigorous impact assessment, the results of which enterprise management practices. These results
are presented in this chapter. This study aims to would lead to more milk per cow, higher and
provide further evidence on the effectiveness of more stable output, and lower production costs.
investments in dairy agriculture, and to identify
Furthermore, the programme sought to improve
challenges and barriers that could be addressed
the linkages of small-scale milk producers,
to improve future project and policy designs and
traders and processors with local milk markets,
development impact.
and to increase producer access to the
processing sector. To this end, the project
invested in improving market infrastructure,
The project’s theory of change such as milk-cooling facilities, improved road
Before the SDCP, dairy groups in the nine infrastructure, conducted capacity-building
selected milk-producing counties suffered from: activities for milk marketing groups, developed
weak links with input suppliers and output low-cost market information systems and trained
purchasers; a limited ability to disseminate beneficiaries on hygienic milk handling, among
production advice to farmers; weak business and other things. These activities should lead to
commercialization skills; and financial services better-quality milk (which is likely to lead to higher
constraints. To address these barriers, the SDCP prices), reduced transaction costs, an overall
trained beneficiary dairy groups on organizational increase in the size of the country’s dairy market,
and enterprise skills, and helped them establish and more reliable trade relations and sales

PART II / Chapter 6
and maintain links with input providers, output (Burke, Myers and Jayne, 2015).
purchasers and extension systems. In addition, Through these highly interrelated and mutually
the project sought to improve dairy groups’ supportive activities, the SDCP was expected
access to financial services by promoting to achieve the overall objective of the project:
competitive investment grants designed to increasing incomes through dairy farming. This
improve dairy business activities and techniques. in turn should have led to an increase in the food
Through these activities, the SDCP aimed to:
security of participating dairy farmers (Smith et
establish sustainable dairy enterprises; enable
al., 2013). Moreover, given the overall importance
members to obtain financial services; reduce
of resource-poor farmers and the special
transaction and input costs; raise output prices;
emphasis on women in the Kenyan dairy sector,
and increase beneficiaries’ production and
it was expected to have a positive impact on the
market knowledge (Atieno and Kanyinga, 2008).
participation of these groups in milk markets and
To further boost productivity and reduce as leaders in dairy groups (Walton et al., 2012).
production costs, the project used training
sessions, field days and demonstrations to
provide smallholder farm households with Project outreach and outputs
technical support on artificial insemination, fodder
production and management, dairy enterprise The first phase of the SDCP was implemented
management practices, and animal diseases and in 27 divisions within nine counties across
disease management. With the help of newly three regions of central and central-western
established community-level artificial insemination Kenya – namely, Rift Valley Region, Western
schemes, community-based animal health funds, Region and Nyanza Region, home to about
and information-dissemination activities, the 350,000 dairy farmers. The beneficiaries of the
project was expected to result in: better-bred and programme and the members of the targeted
healthier dairy cows (Gelan and Muriithi, 2015; dairy groups comprised mainly: resource-
VanLeeuwen et al., 2012); greater production poor dairy farmers; part-time and small-scale

71
 intensive dairy farmers; crop-oriented farmers of 184 dairy groups in the nine project counties.
with dairy cows; and small-scale milk bars, To conduct a valid assessment of the project’s
shop operators and mobile milk traders. As impact on the livelihoods of smallholder dairy
mentioned above, particular attention was paid farmers, it was also necessary to identify
to women, given that many female‑headed and establish a valid comparison group
households in the project’s districts had dairy that shares the same characteristics as the
cows. Even in male-headed households, females SDCP beneficiaries.
controlled more than 60 per cent of the income
The impact assessment focused on the
from dairy activities. Women in these counties
first phase of the SDCP project, which was
face a number of constraints. They own smaller
completed in September 2015; data were
farms, which affects their access to credit using
collected throughout November and December
land as collateral. They are less educated and
2016. Given the ex post nature of the evaluation,
receive less extension advice, which limits their
a quasi-experimental matching approach was
access to technical information for enhancing
used to establish a meaningful comparison
production. They often do not play a vital role
group. As a first step, this assessment aimed to
in dairy groups. The project sought to address
replicate the initial targeting process conducted
all of these constraints. The first phase of the by the SDCP in 2005, in order to identify
SDCP intervention supported 13,132 small non‑programme areas that had characteristics
dairy farmers, of whom 60 per cent were similar to the project areas before the programme
women, through 556 dairy groups and apex started. Drawing on the original database used
organizations, reaching 120,000 beneficiaries to identify the 27 SDCP divisions, and taking
overall. Upon completion of the extended SDCP into account the opinions of local experts, the
in September 2015, various outputs had been research team identified eight study sites as valid
achieved, including: comparison areas to serve as controls.
• 8 milk-cooling or -processing facilities had
been set up. Within these SDCP divisions and the eligible

• 6,123 farmers had participated in control area divisions, 95 treatment and

educational and exchange tours. 89 control dairy groups were identified. From
these, 1,297 beneficiary and 1,265 comparison
• 500 fodder-bulking sites had been
dairy farm households were randomly selected
established by dairy groups.
and interviewed on household characteristics
• 27 community-level artificial insemination
and dairy farming activities, including inputs,
schemes had been established.
production activities, costs and sales. In addition,
• 81 dairy groups and 13 apex organizations
each existing dairy group was surveyed to
had benefited from dairy enterprise grants,
obtain information on its structure, activities
with an average amount of Ksh 638,000
and performance. Based on the available and
(about US$6,250).
collected information, and using dairy group
• 1,821 goats had been procured and
performance indicators, household characteristics,
distributed to the 801 most vulnerable
dairy farming information, and pre‑project
resource-poor women.
divisional and community characteristics, SDCP
beneficiaries were matched with comparison
smallholder farming households, and the outcome
Project impact indicators of interest were compared using a
doubly robust estimator (see Van der Laan and
Data and methods
Robins, 2003). The impact of the SDCP – the
The estimation of the project’s impact was average of the differences between the matched
based on a comprehensive quantitative survey treatment and comparison households – was
covering 2,562 dairy farmers who are members estimated for numerous indicators, including:

72
 • husbandry practices; positive effect, and the improvement in animal
• extension services; management was the most reported benefit of
• input and transaction costs; the SDCP. A female participant stated: “Initially
• gross margins and milk prices. I would send the children to go and graze, [but]
I now practice zero grazing. I realized I used to
To better understand the experiences, opinions
lose a lot on milk yields.”
and perspectives of the SDCP beneficiaries,
12 focus group discussions with six to eight Furthermore, the SDCP increased the likelihood
farmers each from 12 dairy groups were that farmers kept their cattle in a paddock with a
conducted. Moreover, key informants were stall floor made of concrete. Such paddocks are
interviewed on the uptake of the programme, easier to clean and reduce the chances of milk
coordination of government departments and being contaminated.
the functioning of the dairy sector in the regions The SDCP also led to an improvement in
of interest. These qualitative results augment the breeding services. Programme participants were
quantitative findings. 7 per cent more likely to monitor their cattle
Given the SDCP’s main objective of increasing on a regular basis, and 12 per cent more likely
the incomes of resource-poor rural smallholder to have used artificial insemination services
households engaged in dairy farming activities, in the preceding 12 months. No differences
the impact evaluation addressed the following between participants and comparison farmers
main questions of interest: (1) Does the project were observed in the availability of livestock
improve the well-being of dairy farmers through technology, showing that the SDCP was
improved animal management, increased successful in promoting the benefits of new

PART II / Chapter 6
production and productivity, improved efficiency artificial insemination breeding practices, which
and increased incomes for farmers? and (2) Does resulted in a higher uptake of these services.
increased income from dairy farming ultimately Finally, the SDCP project had positive and
improve food security? significant effects on the availability and
accessibility of specific animal health services.
Relative to non-participants, SDCP farming
Key impact estimates households were 25 percentage points more
likely to have access to and use vaccination
In line with the main research questions, the services, 10 percentage points more likely to
impact evaluation investigated the effect have access to curative treatment services, and
of the SDCP project on various intended 7 percentage points more likely to make use of
intermediate and final outcome indicators of them. However, the programme did not seem to
dairy farming households. have changed access to other health services,
including deworming or tick control services.
Animal management practices

Regarding the impact of the intervention on Access to extension and information

animal management activities, the project was One of the project’s key activities was to
estimated to have had statistically significant disseminate information on farming practices
positive effects on beneficiaries’ feeding through extension services and training of dairy
practices. SDCP participants were 8 per cent group members. The impact evaluation shows
more likely to practise zero grazing with their that beneficiary farmers were 12-18 percentage
cattle – a practice that gave farmers better points more likely to have cattle extension
control over how much and what cows ate, services available in their village. The categories
and reduced losses in potential milk production of information for which the effects were
that resulted from searching for grazing land estimated to be greatest were: general livestock
across large areas. Farmers recognized the best practices (15 percentage points); milk

73
 processing and quality control (10 percentage and shared milk price data, and facilitated links
points); and fodder establishment and fresh between members and input suppliers and milk
milk marketing (7 percentage points). Most purchasers. The only exception was that SDCP
importantly, farmers tended to adopt these new groups were more likely to contract with milk
practices, although adoption rates remained, as purchasers on behalf of members, even though
did the probability of receiving information, mainly the rate was relatively low: 11 per cent, as
below 15 per cent. opposed to 2 per cent for the comparison groups.
This result shows that significant room remains to
improve the performance of SDCP dairy groups,
Dairy group performance particularly in terms of providing services that aim
to reduce smallholders’ transaction costs.
The SDCP aimed to establish sustainable dairy
groups with formalized structures and better
performance indicators to link these groups
Number of livestock and milk
to the milk value chain. In terms of formalized
structures, the evidence was mixed. There were
production
no differences between SDCP groups and control Overall, the project had positive and statistically
dairy groups with respect to legal registration, the significant effects on the composition and
election of chairs and representation of women. number of cattle per household as well as on milk
SDCP groups were more likely to have an elected production and the value of milk sold. Relative
secretary and treasurer as well as to have more to the comparison group, SDCP farmers owned
monthly meetings to share information and engage 0.49 more head of cattle, 0.35 more cows that
in decision-making. had calved at least once, and 0.44 more animals
Treatment dairy groups were far more likely to they were currently milking (figure 6.1). These
obtain financing from microfinance institutions, results were explained mostly by a higher number
local credit and savings groups, and commercial of cross-bred cattle and not of the exotic, more
banks, and this financing can help farmers gain productive breeds.
access to input and output markets. However, The intervention’s positive impact on the number
both SDCP and comparison groups still relied of livestock was reflected in the effect on milk
heavily on members’ financial contributions, production. Beneficiary farmers had 37 per cent
with 93 per cent of all groups stating that they higher total milk production and 58 per cent
depended on membership dues. more total milk production at calving compared
For the most part, SDCP and comparison groups with non-beneficiaries. SDCP farmers were
faced similar levels of conflict among members, 8 percentage points more likely to have sold milk
with the exception of conflict surrounding unpaid at any time during the day before the interview,
dues. Only 21 per cent of beneficiary dairy groups indicating positive impacts on milk marketing.
reported conflicts over unpaid dues compared The evidence suggests that there was no statistical
with 37 per cent of comparison groups. significant difference between participants and
comparison farmers in terms of quantity sold.
The most striking difference in the performance
Nevertheless, those SDCP farmers who sold to
between the two groups concerned the number
the market obtained a 31 per cent higher selling
of training sessions provided to the dairy group
price than that received by non-beneficiaries. This
members. SDCP group members were more likely
finding could mean that beneficiaries have better
to be trained on fodder management, proposal
linkages with the milk market or higher-quality dairy
writing or dairy group management.
products. The total value of milk sold – that is,
Few differences between SDCP and comparison the quantity of milk sold times the price obtained
dairy groups existed in terms of services provided – obtained by SDCP farming households was
by the respective dairy groups. Both groups 43 per cent higher than the value obtained by the
organized educational exchange tours, collected control group (figure 6.2).

74
These quantitative positive results on farmers’ the growth of crops and more yields – this too
incomes were supported by the beneficiaries’ has ensured a consistent supply of food in
comments during the qualitative fieldwork. the household.”
One SDCP farmer said: “My milk production
These qualitative findings were supported by the
has increased, and that’s more income. With it, estimated quantitative data on food consumption
I have taken my children to school.” Moreover, within households (using seven-day recall). Those
many farmers perceived a significant increase results provided some evidence that programme
in their productivity. Another SDCP farmer beneficiaries exhibited higher levels of food
reported: “Initially I would only get 3 cups of milk, diversification, a good proxy for food security.
but currently my cow production is 7 bottles.” As highlighted in figure 6.3, SDCP households
Based on these results, it can be assumed that were more likely to have a more diverse food
overall the SDCP was successful in increasing basket, with higher levels of animal and vegetable
smallholder dairy farmers’ milk production, milk proteins (such as red meat, milk products and
marketing and milk prices received. legumes) and lower levels of tubers and fruits.
All of these findings suggest that SDCP activities
Food security contributed to a positive overall impact on Kenyan
Did this positive impact on dairy farmers’ income dairy farmers, particularly for dairy production.
also lead to increased food security, as envisaged
by the project?

According to multiple SDCP participants, the

Lessons learned
increase in income as a result of the SDCP These quantitative and qualitative results point

PART II / Chapter 6
intervention enabled them to keep a variety of to a number of important lessons for informing
foods available in their households, confirming the design and implementation of similar dairy
the objective of the project. One farmer stated policies and projects elsewhere in Kenya and in
that his family’s general health had improved, other developing countries:
while others said that they now consistently had • Interventions that aim to support dairy groups,
tea with milk in their house. Farmers’ perception enhance farmers’ productivity through
of increased food security may have resulted training and strengthen market linkages for
partially from their increased understanding of small‑scale milk producers can translate
better practices for growing crops, including into higher incomes for smallholder farmers.
the use of cow dung as fertilizer to obtain The results presented here show that this
higher yields. One beneficiary farmer said: “The is determined mainly by the higher per‑litre
animal manure from both the goats and cattle is selling price that participants were able
channelled on the farm, which in turn accelerates to obtain.

Figure 6.1 Average number of animals

Milking cows

Cows, calved at least once

Cattle owned

0 1 2 3

Comparison dairy farmers

SDCP beneficiaries

75
 Figure 6.2 Average milk production and total value

Total value of milk sold

Selling price of milk/litre

Total litres of milk sold

Total milk production at calving

Total milk production

0.0 0.5 1.0 1.5 2.0

Percentage-point increase

Comparison dairy farmers

SDCP beneficiaries

• Disseminating information on different References

aspects of production through training,
Atieno, R. and Kanyinga, K. 2008. The
field days and demonstrations to dairy
revitalisation of Kenya Cooperative Creameries:
group members increases the availability
The politics of policy reforms in the dairy sector in
of extension services, and beneficiaries
Kenya. Research Paper 9. Brighton, UK: Future
tend to adopt these practices more than
Agricultures Consortium.
do comparison farmers. However, adoption
Bingi, S. and Tondel, F. 2015. Recent
rates for all promoted activities remain low,
developments in the dairy sector in Eastern
suggesting that there is still significant room
Africa. Briefing Note 78. Maastricht, Netherlands:
to improve activities and training in future
European Centre for Development Policy
project designs. Management.
• Special emphasis should be placed on
Burke, W., Myers, J. and Jayne, T. 2015. A
disseminating market-related information
triple-hurdle model of production and market
and promoting dairy group marketing.
participation in Kenya’s dairy market. American
Quantitative results show that the services Journal of Agricultural Economics, 97(4): 1227-
provided to dairy group members have 1246.
limited impacts on marketing. Only a
Burke, W. J., Jayne, T., Freeman, H. and
minority of dairy groups facilitate links
Kristjansen, P. 2007. Factors associated with
between members and input suppliers as
rural farm households’ movements into and out
well as milk purchasers. Although SDCP of poverty: The rising importance of livestock.
dairy groups contracted more with milk International Development Working Paper 90.
purchasers than comparison groups did, East Lansing, USA: Department of Agricultural
the number remained low. Despite these Economics, Michigan State University.
low numbers, SDCP farmers obtained FAO (Food and Agriculture Organization of the
higher prices in the market, indicating the United Nations). 2014. FAOSTAT Statistics
considerable potential to strengthen market Database. Statistical Yearbook-Kenya, http://
linkages for small-scale milk producers. faostat.fao.org/static/syb/syb_114.pdf

76
Figure 6.3 Food consumption (seven-day recall)

Comparison dairy farmers

Milk products
SDCP beneficiaries
Red meats, other organ meats

Beans, peas, lentils, nuts

Fruits

Patatoes, yams, cassava

0.0 0.2 0.4 0.6 0.8 1.0 1.2

Percentage-point increase

FAO. 2017. FAOSTAT Statistics Database. Cattle, Richards, S., VanLeeuwen, J. A., Shepelo, G.,
dairy stocks Kenya and Eastern Africa, www.fao. Gitau, G. K., Wichtel, J., Kamunde, C. and
org/faostat/en/#data/GE Uehlinger, F. 2016. Randomized controlled trial
on impacts of dairy meal feeding interventions
Gelan, A. and Muriithi, B. W. 2015. Examining
on early lactation milk production in smallholder
returns to scale in smallholder dairy farms in
dairy farms of Central Kenya. Preventive Veterinary
East Africa. Quarterly Journal of International

PART II / Chapter 6
Medicine, 125: 46-53.
Agriculture, 54(3): 239-261.
Smith, J., Sones, K., Grace, D., MacMillan, S.,
Holloway, G., Nicholson, C., Delgado, C., Staal,
Tarawali, S. and Herrero, M. 2013. Beyond milk,
S. and Ehui, S. 2000. Agroindustrialization
meat, and eggs: Role of livestock in food and
through institutional innovation: Transaction
nutrition security. Animal Frontiers, 3(1): 6-13.
costs, cooperatives and milk‐market
development in the east‐African highlands. Staal, S., Delgado, C. and Nicholson, C. 1997.
Agricultural Economics, 23(3): 279-288. Smallholder dairying under transactions costs in
East Africa. World Development, 25(5): 779-794.
Muriuki, H. G. 2011. Dairy development in Kenya.
Rome: FAO. Van der Laan, M. and Robins, J. 2003. Unified
methods for censored longitudinal data and
Nafula, S. S. 2013. The impact of enhanced causality. New York, USA: Springer-Verlag.
nutritional and feeding practices on growth and
VanLeeuwen, J. A., Mellish, T., Walton, C.,
health of dairy calves in Mukurweini district of
Kaniaru, A., Gitau, R., Mellish, K., et al. 2012.
Kenya. Doctoral dissertation, University of Nairobi.
Management, productivity and livelihood effects on
Olwande, J., Smale, M., Mathenge, M. K., Place, Kenyan smallholder dairy farms from interventions
F. and Mithöfer, D. 2015. Agricultural marketing addressing animal health and nutrition and milk
by smallholders in Kenya: A comparison of maize, quality. Tropical Animal Health and Production,
kale and dairy. Food Policy, 52: 22-32. 44(2): 231-238.
Randolph, T. F., Schelling, E., Grace, D., Walton, C., VanLeeuwen, J. A., Yeudall, F. and
Nicholson, C. F., Leroy, J. L., Cole, D. C. and Taylor, J. 2012. Association between duration
Ruel, M. 2007. Role of livestock in human nutrition of community-based group membership and
and health for poverty reduction in developing sustainable livelihoods for Kenyan women dairy
countries. Journal of Animal Science, 85(11): farmers. Journal of Agriculture, Food Systems,
2788-2800. and Community Development, 3(1): 43-60.

77
 Chapter 7

Madagascar’s AD2M-I Project

by
Romina Cavatassi
Peter Brueckmann
Nancy McCarthy
Mitchell Morey

Madagascar, Africa’s largest island state, remains In the mid-2000s, the Government of
one of the least-developed countries in the Madagascar began adopting laws and
world after decades of slow economic growth, implementing policies aimed at empowering the
unstable governance and political turmoil. poor and decentralizing administration and public
About 65 per cent of its population lives in rural services. Beginning in 2005, this process focused
settings, mainly relying on subsistence‑oriented strongly on land reforms and decentralization of
farming practices, exploiting small areas of land offices. The 2009 coup delayed activities
land (INSTAT, 2011; World Bank, 2017a), and significantly; nevertheless, the new government
suffering from persistent high levels of poverty, managed to continue the land reform and other
food insecurity and malnutrition (Dostie, decentralization plans. At the same time, despite
Haggblade and Randriamamonjy, 2002; Minten relatively generous rainfall and water resources
and Barrett, 2008). Rice is the main staple crop within the country, many farm households
and the main source of income for farming remained mired in poverty and vulnerable to the
households in Madagascar, with 87.5 per cent vagaries of weather (Harvey et al., 2014; Tadross
of all farming households growing rice and et al., 2008). Climate change increases the need
48 per cent of agricultural income generated to protect farmers from weather shocks that
through rice cultivation (INSTAT, 2011). As such, reduce agricultural yields. This is particularly true
rice is of critical importance to the welfare of
for farmers in the regions of Menabe and Melaky
the Malagasy population and farmers, as well
in central western Madagascar, who currently
to the development of the country’s agriculture
face very high levels of poverty and malnutrition.
sector. Nonetheless, Malagasy rice farmers face
numerous barriers to improving their agricultural Most of the empirical evidence suggests
production and well-being, including: tenure that when irrigation infrastructure is well built
insecurity, land degradation, lack of irrigation and governance of the irrigated areas is well
infrastructure, high marketing transaction costs, functioning, investments in irrigation infrastructure
inadequate agricultural production techniques, can both increase agriculture’s climate resilience
and low adoption rates of new rice cultivation and also significantly raise farmers’ yields,
techniques and of improved rice varieties (Barrett leading to higher and more stable productivity
et al., 2004; Harper et al., 2007; Harvey et al., and increased net revenue (Hussain and Hanjra,
2014; Minten and Barrett 2008; Moser, Barrett 2004; Knox, Daccache and Hess, 2013; Nakano
and Minten, 2009). et al., 2013). In the case of Madagascar, Jacoby

78
and Minten (2007) documented higher yields Evaluation, the American Institutes for Research,
and net revenues per hectare (of more than and Lead Analytics. This impact assessment
30 per cent in each case) on irrigated lands can provide evidence on the effectiveness
compared with non-irrigated lands. At the same of irrigation to promote more stable crop
time, experience has shown that many projects production and farmer livelihoods, and it can
aiming to increase the area under irrigation have be used to better design and implement future
not led to the hoped-for gains. The literature cites irrigation interventions and policies, not only in
a number of reasons, including: the quality of Madagascar but also in other countries that share
initial feasibility studies; the quality and location the features and challenges of Madagascar’s rural
of irrigation infrastructure; and the efficacy of farmers. The descriptive analysis of infrastructure
governance structures put in place to regulate construction, management and maintenance also
water uses and users, and ensure maintenance sheds light on the relative importance of these
of irrigation infrastructure. components in generating gains at the farm level.
Madagascar is one of the six most-irrigated
countries in Africa (You et al., 2011), but
only 2.18 per cent of its agricultural land is The project’s theory of change
under irrigation (World Bank, 2017b). Thus,
increasing and rehabilitating irrigation facilities in Given the complexity of the project and its
Madagascar was considered of key importance various components, the impact assessment
by policymaker and development organizations. focused on household-level outcomes and
As such, between 2006 and 2015, IFAD impacts from access to certificated irrigation
implemented the Appui au Développement land. However, this chapter describes the theory

PART II / Chapter 7
du Menabe et du Melaky (AD2M-I) project in of change for the entire project, including both
the two regions. The project consisted of two the certification and irrigation activities that were
components designed to strengthen farmers’ the focus of the impact assessment, as well as all
tenure security and result in higher, more stable other project inputs on which data were collected
and sustainable agricultural incomes. It sought and thus controlled for in the analysis in order
to meet these goals by increasing access to provide a more comprehensive picture of the
to irrigation, introducing new seed varieties, assessment conducted.
promoting additional cash crops, increasing
access to agricultural equipment, and increasing Strengthening local governance and tenure
access to markets through road construction security
and rehabilitation, waterway infrastructure and Customary tenure institutions alone were
microfinance kiosks. The project had a total insufficient to guarantee tenure security to many
cost of US$27.2 million, cofinanced by IFAD, the farmers, including women and other vulnerable
Millennium Challenge Account, the European groups, or to resolve conflicts adequately.
Union and the Government of Madagascar. A The resulting tenure insecurity dampened
second phase of the project is currently ongoing farmers’ incentives to invest in sustainable land
and expected to be completed by the end of management practices. The purpose of the
2022 (AD2M-II). AD2M-I intervention was to complement and
To analyse the effects of the AD2M-I project take part in the Governmental National Land
and provide insights on the impact of irrigation Programme process by establishing and training
infrastructure on the agricultural production local land administration offices. These would
and well-being of smallholder farmers, IFAD be part of a well-functioning decentralized land
conducted a rigorous ex post impact assessment administration system that establishes local land
through the International Initiative for Impact management plans and issues land certificates

79
 efficiently and effectively. This would give farmers crop and resource management systems,
more tenure security and therefore greater including the system of rice intensification15
incentives to engage in sustainable soil and land and the improved rice system. The project
management practices. undertook farmers field schools, demonstrations
and workshops to encourage farmers to adopt
Support for sustainable development of the cultivation techniques, more resistant seed
productive base varieties, livestock management techniques,
Despite relatively generous rainfall and water environmental protection practices, and crop
resources in the regions of Menabe and Melaky, diversification. These were complemented with
farmers’ rice yields had been low and stagnating the provision of agricultural inputs, including
in the years before project inception. Farming seeds and farming tools, to project beneficiaries.
households in the project regions either had The expectation was that adoption would lead
access to irrigation systems that were in poor to more diversified agricultural production and
condition or they had no access at all, suggesting income sources, enhanced land productivity
that one key intervention should focus on and lower costs of cultivation (Barrett et al.,
irrigation. As such, the AD2M-I project focused 2004; Tsujimoto et al., 2009) through more
on building new, or rehabilitating existing, productive inputs and techniques, and through
irrigation perimeters in rice-growing communities. reduced erosion and increased adaptation to
This work was supported by capacity-building climatic events.
activities aimed at better coordination and As an important complementary subcomponent,
management of irrigation facilities through new rural roads and waterway transport infrastructure
or informally existing water user associations were constructed or rehabilitated. This improved
(WUAs). WUAs – with formalized leadership transportation and infrastructure was expected
structures, rules and chains of command – are to reduce marketing transaction costs and raise
an important element in operating, maintaining marketable surpluses.
and repairing irrigation schemes, and they are
All activities of this multifaceted and complex
a key determinant in whether farmers will gain
AD2M-I project strongly complemented one
benefits from irrigated land (Burney and Naylor,
another. Overall, it was expected that the outputs
2012; Hamdy, Abu-Zeid and Lacirignola, 2009;
of the AD2M-I activities would lead to positive
Meinzen-Dick, 2007). Improved access to
impacts on the community environment, land use
irrigation and increased capacity of associated
and management and, most importantly, the level
WUAs should lead to higher-efficiency water use
and stability of beneficiaries’ incomes. This in
and, most importantly, to stable irrigation of farm
turn should lead to present and future household
plots. The consistent availability of irrigation water
welfare gains and reduced poverty.
should allow rice and other crop production
throughout the year by enabling a second or
even a third crop season, and this in turn should
lead to higher, more diversified and more stable Project outreach and outputs
productivity and yields for farm households
Given the intervention’s complex approach
(Knox, Daccache and Hess, 2013; Minten and
in 19 communes throughout five districts
Barrett, 2008; Nakano et al., 2013).
across the Menabe and Melaky Regions, the
To achieve an even more ambitious objective, the AD2M-I project targeted numerous groups of
AD2M-I project conducted training on integrated beneficiaries. Among these groups were: landless

15. The system of rice intensification is an agroecological methodology intended to increase the productivity of irrigated
rice by changing the management of plants, soil, water and nutrients. Originating in Madagascar in the 1980s, it
is based on the cropping principles of significantly reducing the plant population, improving soil conditions and
irrigation methods for root and plant development, and improving plant establishment methods (Uphoff, 2002;
Glover, 2011).

80
agricultural wage earners; farm households with implementation of the project itself, which was
non-irrigated landholdings smaller than two mostly based on agroecological observable
hectares; poor, often indigenous, households characteristics. During the inception process of
focusing on fishing activities; farmers with small AD2M-I, the project conducted detailed irrigation
irrigated landholdings and small numbers of feasibility studies to identify irrigated areas in
cattle; and pastoral farmers engaged mainly need of rehabilitation, as well as sites for new
in cattle breeding. Project activities reached irrigation infrastructure. The feasibility studies
40,000 households – about 40 per cent of the included detailed information on hydrological
rural households in both project regions – and and topographical features, rainfall, the size of
16,000 of these households constituted direct the irrigable area and cost estimates for irrigation
beneficiaries. infrastructure as well as basic information on
the socio-economic characteristics of potential
Over the project’s nine years, many activities
beneficiaries. Of 62 sites considered, 36 were
were undertaken and outputs obtained. With
deemed feasible, of which 18 obtained new
respect to land administration and legal land irrigation infrastructure or had existing irrigation
rights, project activities included land-use infrastructure rehabilitated.
diagnostics, the development of local land-use
plans and the establishment of 15 land offices, as For the impact evaluation, the overall sampling
strategy replicated the targeting process
well as numerous training sessions for land office
used in 2007 by using the same or similar
staff, local leaders and community members. In
available historic data sources. As a first step,
addition, 8,840 land certificates, covering 7,257
observation-based targeting was used, with
hectares, were issued.
AD2M-I project sites being plotted on a map to

PART II / Chapter 7
With respect to irrigation, 18 sites received new identify the common characteristics of the sites.
or rehabilitated infrastructure covering 5,588 This information was then used to determine a
hectares of irrigated land (3,393 hectares covered group of potential non-project areas that had
by new infrastructure and 2,195 hectares characteristics similar to AD2M-I areas before
covered by rehabilitated infrastructure) reaching the project started. Once the first group of
5,600 agricultural producers (4,000 in Menabe potential areas had been selected, criteria-based
and 1,600 in Melaky). In addition, 32 WUAs and targeting was used to refine the selection of the
2 WUA unions were created or strengthened, comparison sites.16
accounting for a total of 3,724 members.
From the final list of AD2M-I intervention
and comparison areas, 1,986 households in
62 villages were interviewed in October and
Project impact November 2016 (682 treatment households
and 1,304 comparison households), and these
Data and methods interviews produced detailed information on
In every impact evaluation, great importance is household and plot management characteristics.
attached to carefully selecting an appropriate In addition, one survey in each treatment and

control group in order to be able to assess and comparison commune was conducted to obtain
additional community-level variables.
attribute project impacts. For this evaluation, the
strategy for constructing a reliable comparison Given the complexity of the project, the
group was to replicate the targeting process impact assessment focuses on households
used to select AD2M-I beneficiaries during the that benefited from access to irrigated plots

16. Criteria-based targeting is based on expert opinions to determine to what extent non-targeted areas could have
been chosen for the project. These expert opinions relied on historical variables likely to be associated with the
uptake of the intervention in 2007 such as climate, population density and other agroecological conditions.

81
 in Menabe.17 To complement characteristics Answers to both questions can be used to draw
obtained from household interviews, additional conclusions about the extent to which farmers
control variables covering biophysical obtained higher and more stable incomes. In
characteristics and a set of spatial dummies were addition, the following questions were addressed:
used to account for differences that would have
• Did project activities lead to an increase in
explained differences in the outcome indicators
sustainable land management practices at
of interest between treatment and comparison
both the community and household levels?
households. These indicators included a
• Given the importance of well-functioning
normalized difference vegetation index – a useful
WUAs in maintaining irrigation infrastructure
measure of soil fertility computed by using
and regulating water use, did measures of
available satellite images of the treatment and
WUA performance affect the extent to which
comparison areas. Based on this comprehensive
farmers were able to realize benefits from
dataset, treatment households were matched
their irrigated plots?
with comparison households, and the impact of
the AD2M-I irrigation activities was quantified.18

The quantitative data collection and subsequent Key results

estimation of results were complemented with
focus group discussions and key informant Impacts on annual crop production
interviews with programme beneficiaries and
Overall, the impact assessment estimated that
relevant officials. These qualitative data were
the intervention made meaningful improvements
collected to: facilitate a better understanding
to the agricultural production and productivity of
of projects’ mechanisms and impacts; support
beneficiaries.19 As shown in figure 7.1, the annual
and validate or amend key impact findings; and
per hectare value of beneficiaries’ crop production
identify challenges and potential moderating
was 24 per cent higher than that of control
factors. Moreover, they allowed the research
households, and the annual per capita value of
team to draw conclusions not only about the
their crop production was 23 per cent higher than
impact of the irrigation component itself but also
that of control households.
about the role of the WUAs and of other project
components. Results show similarly positive effects for project
participants’ rice yields. Annual rice yields were
Based on the theory of change and the focus
estimated to be 27 per cent higher for beneficiary
of the impact evaluation of AD2M-I investments
households compared with control farmers,
in new and rehabilitated irrigation infrastructure,
whereas the quantity of rice harvested per capita
the following key research questions were
was 22 per cent higher for beneficiary farmers
addressed:
than for control farmers.
• Did access to irrigated land increase
farmers’ rice yields, the total value of their These positive impacts on crop production
irrigated crop production, their total net crop were supported by the qualitative analysis that
revenues and the value of crop output they complemented the impact assessment. A farmer
marketed? from Belo, one of the beneficiary villages, reported:
• Did access to irrigated land enable farmers “The harvests improved, mostly thanks to the
to increase the number of cropping increased water supply.” A farmer from Mahabo
seasons within the year and to have a more said: “Yes, we are able to produce in excess and
diversified crop portfolio? have to sell [the rice] outside our region.”

17. Melaky was excluded for a number of reasons, including logistics, resources and the fact that some areas were not
covered by irrigation, the main component being assessed.
18. Specifically, the inverse-probability-weighted regression adjustment estimator was used. For more information, see
Wooldridge (2010).
19. All results reported are statistically significant at a 95 per cent confidence interval level or higher (in most cases at
99 per cent) unless otherwise specified.
82
These quantified and perceived benefits also training sessions and workshops, and they are
had positive effects on the community, according also 8 per cent more likely to attend training than
to many key informants and group discussion were control households.
participants. These effects included positive
Beneficiaries are also 8 per cent more likely to use
impacts on the labour market, households’
pesticides than control farmers – although the
purchasing power and village economies as a
average number of beneficiary farmers applying
whole. “The community is also more peaceful,
herbicides or pesticides remains low, with an
we eat well, our harvests are successful and
overall average of 9 per cent and 23 per cent,
even the unemployed get work,” stated a farmer
respectively. It is estimated that the project had no
from Mahabo. A key informant from Mahabo
impact on the use of inorganic fertilizers.
reported: “In the past, a farmer, one family did
not own a zebu working the fields, but now after Further evidence that farmers’ addition of a
they harvested the rice, they bought two zebus, second cropping season, thanks to irrigation,
and this resulting in that, this transformed the was behind the increase in agricultural production
economy in general in the … village.” also comes from the results on adoption of
new farming practices, which were expected
One important reason behind the rise in
to produce higher yields per se but which
beneficiaries’ annual crop production is that
the farmers largely failed to adopt or soon
irrigation allows farmers to increase the number
abandoned after the completion of the AD2M-I
of cropping seasons per year. Beneficiary
project. During the qualitative analysis, a farmer
households are 19 per cent more likely to engage
from Belo said: “Crop production has decreased
in a second cropping season compared with
since AD2M-I left the town. Some farmers went
control farmers. Moreover, researchers found

PART II / Chapter 7
back to traditional agricultural practices.”
no statistically significant impacts for crop and
rice production when looking only at the primary Similarly, quantitative data show limited evidence
season. This combination of results implies that of farmers in beneficiary areas expanding into
most of the gains come from farmers’ ability to new crops. However, the qualitative analysis does
crop for a second season, which counteracts not validate this finding. During interviews and
the reduced yield and higher risk of agricultural focus group discussions with beneficiaries, many
production linked to the vagaries of weather respondents reported that AD2M-I encouraged
patterns. Indeed, among the factors reported farmers to cultivate new crops (besides only rice),
during the qualitative fieldwork and documented such as onions, beans, peanuts and tomatoes,
in the literature mentioned above that can and that farmers in Mahabo and Belo continue
potentially reduce or impede improvements in to harvest these crops today. According to a
crop production per season, it is worth reporting community leader from Ankilizato Nord (Mahabo):
recurring weather shocks (mainly cyclones and “Onions and beans are the main alternative crops
floods), in addition to local market constraints that AD2M has introduced here. Now they have
and safety concerns. a warehouse where the onions are stored here.”
A community leader from Antrobiky corroborated
Impacts on farmers’ access to extension this statement: “We continued growing peanuts
services and crop inputs and onions. We have truly continued.”

In contrast to control households, of which Several respondents also commented that

30 per cent reported having received extension multicropping improved the quality of their soil
services from any source, 38 per cent of and ultimately the quality of their rice. However,
beneficiaries reported having participated in experiences cultivating new crops in Belo were

83
 Figure 7.1 Crop production and management indicators

1.4 Comparison farmers

1.2 AD2M-I beneficiaries
Percentage-point increase

1.0

0.8

0.6

0.4

0.2

0.0
Value crop Value crop Rice yields Quantity
production production of rice
per hectare per capita per capita

slightly less positive, and both farmers and WUA Quality of irrigation and performance of
members seemed more reluctant to focus on water user associations
cultivating crops other than rice.
According to quantitative and qualitative results,
the project had positive effects on the quality and
Impacts on household welfare gains
efficiency of irrigation compared with the control.
Given that annual rice yields and total annual AD2M-I beneficiaries with plots located in a
value of crop production increased significantly, gravity-based irrigation system were 15 per cent
clear higher welfare impacts were expected. more likely to receive water on time, and
Indeed, farmers and project implementers from 26 per cent more likely to receive good-quality,
both districts reported higher incomes thanks non-brackish water. However, only 40 per cent
to higher rice yields. The quantitative impact of the beneficiaries rated the quality of the water
evaluation confirms these findings to a certain as good, and 27 per cent of project participants
extent. In contrast to control households, of did not receive water on time. Moreover, there is
which 57 per cent reported worrying about no difference between beneficiaries and controls
securing enough food in the preceding seven with regard to whether they received their full
days, 46 per cent of project beneficiaries allocation of water. About 37 per cent of both
reported such worries. Nonetheless, no groups claimed that they did not receive all of the
significant differences were found in annual food promised water allocation. These results show
security, indicating high levels of intra-annual food that significant room for improvement remains.
insecurity across the entire sample. Beneficiary It is crucial to meet the irrigation needs of farmers
households also had 1.1 more household who follow different harvesting schedules and
durable assets than control households, and they have varying water needs, and to maintain the
spent more on education. Perceived broader irrigation infrastructure. The performance of
community benefits include lessened disputes WUAs is particularly important to maintaining this
over water supply, enhanced communication infrastructure and using it effectively. According
between neighbourhoods, and a unifying effect to the qualitative results, one main challenge was
on the local community. a lack of communication by AD2M-I between

84
the WUAs’ leadership and farmers. In particular, • Well-constructed and well-functioning
some AD2M-I farming households reported WUAs are key to successfully implementing
having been left uninformed about water supply projects and to achieving positive impacts.
schedules, resulting in unforeseen water cuts and Trusted and well-respected WUA leaders
crop losses. In addition, some farmers were not who communicate transparently with
informed about the objectives and importance of community members on the benefits
WUAs, and thus were not encouraged to spend of maintaining and improving irrigation
their time actively participating in WUA activities. schemes can do much to improve
community cohesion and engagement, and
The importance of WUA leadership that has the
generate higher benefits from irrigated plots.
trust of the community was often stressed in the
• Given the importance of land tenure
qualitative analysis. WUAs led by trustworthy
insecurity in Madagascar, further evidence
people elected by the community resulted in
on the effectiveness of tenure interventions,
more active community participation, fewer
especially in combination with irrigation
concerns about corruption and fewer disputes
investments, will help determine the need for
about members’ fees. After project completion,
additional investments in land administration
some respondents stated that groups fell apart
systems (Jacoby and Minten, 2007).
or became ineffective, preventing farmers
• The long-term success and
from benefiting from irrigation schemes in a
cost‑effectiveness of irrigation investments
sustainable manner. This was a less serious
depend largely on sustainable irrigation
problem among groups for which community
structures that will last and can be
leaders stepped up and took more responsibility;
maintained by local community members
these groups continued to function and maintain

PART II / Chapter 7
and WUAs after project completion.
the irrigation schemes successfully.
As such, capacity-building activities to
improve WUAs’ management of irrigation
structures are of great importance.
Lessons learned • Quantitative results suggest a good success
The AD2M-I programme experience offers a of the project following its logic that a
number of important lessons: well-managed irrigation system translates
• Improved access to irrigation infrastructure into higher yields and harvests, mainly
and a more efficient water supply increase thanks to a second agricultural season.
the rice yields and annual value of crop However, a deeper analysis of the social
production of smallholder farmers in capital and networking system of the
Madagascar, mainly by allowing them WUA, as well as strengthening of training
to cultivate for a second season. Future and capacity to maintain the adoption of
project designs could promote agricultural improved techniques, would be advisable in
strategies – including adoption of order to better understand social aspects,
techniques, crops or varieties – specifically particularly given the fact that some farmers
geared to cultivating land for a second seem to suggest scope for improvement.
cropping season.
• To address farmers’ reluctance to adopt
new farming techniques, future project
designs must use activities and information
dissemination that will raise awareness
of interventions’ long-term benefits
for sustainability.

85
 References INSTAT (Institut National de la Statistique de
Madagascar). 2010, 2011. Enquêtes périodique
Barrett, C. B., Moser, C. M., McHugh, O. V. auprès des ménages 2010, 2011. Antananarivo,
and Barison, J. 2004. Better technology, better catalog.ihsn.org/index.php/catalog/5127/
plots, or better farmers? Identifying changes download/62289
in productivity and risk among Malagasy rice Jacoby, H. G. and Minten, B. 2007. Is land titling
farmers. American Journal of Agricultural in Sub-Saharan Africa cost-effective? Evidence
Economics, 86(4): 869-888. from Madagascar. The World Bank Economic
Review, 21(3): 461-485.
Burney, J. A. and Naylor, R. L. 2012. Smallholder
irrigation as a poverty alleviation tool in Knox, J., Daccache, A. and Hess, T. 2013. What
sub‑Saharan Africa. World Development, 40(1): is the impact of infrastructure investments in
110-123. roads, electricity and irrigation on agricultural
productivity? Systematic review. CEE Review,
Dostie, B., Haggblade, S. and Randriamamonjy, 11-007. Bedford, UK: Collaboration for
J. 2002. Seasonal poverty in Madagascar: Environmental Evidence.
Magnitude and solutions. Food Policy, 27(5):
Meinzen-Dick, R. 2007. Beyond panaceas in
493-518.
water institutions. Proceedings of the National
Glover, D. 2011. The System of Rice Academy of Sciences, 104(39): 15200-15205.
Intensification: Time for an empirical turn. NJAS Minten, B. and Barrett, C. B. 2008. Agricultural
– Wageningen Journal of Life Sciences, 57: technology, productivity, and poverty in
217-224. Madagascar. World Development, 36(5):
797-822.
Hamdy, A., Abu-Zeid, M. and Lacirignola, C.
2009. Institutional capacity building for water Moser, C., Barrett, C. and Minten, B. 2009.
sector development. Water International, 23(3): Spatial integration at multiple scales: Rice

126-133. markets in Madagascar. Agricultural Economics,

40(3): 281-294.
Harper, G. J., Steininger, M. K., Tucker, C. J.,
Nakano, Y., Bamba, I., Diagne, A., Otsuka,
Juhn, D. and Hawkins, F. 2007. Fifty years
K. and Kajisa, K. 2013. The possibility of a
of deforestation and forest fragmentation in
rice Green Revolution in large-scale irrigation
Madagascar. Environmental Conservation, 34(4):
schemes in Sub-Saharan Africa. In K. Otsuka and
325-333. D. Larson, eds., An African Green Revolution.
Harvey, C. A., Rakotobe, Z. L., Rao, N. S., Dave, Dordrecht, Netherlands: Springer.

R., Razafimahatratra, H., Rabarijohn, R. H., et Tadross, M., Randriamarolaza, L., Rabefitia,
al. 2014. Extreme vulnerability of smallholder Z. and Zheng, K. Y. 2008. Climate change
farmers to agricultural risks and climate change in Madagascar: Recent past and future.
in Madagascar. Philosophical Transactions of the Washington, D.C.: World Bank.
Royal Society B, 369(1639): 20130089. Tsujimoto, Y., Horie, T., Randriamihary, H.,
Shiraiwa, T. and Homma, K. 2009. Soil
Hussain, I. and Hanjra, M. A. 2004. Irrigation
management: The key factors for higher
and poverty alleviation: Review of the empirical
productivity in the fields utilizing the system of
evidence. Irrigation and Drainage, 53(1): 1-15.
rice intensification (SRI) in the central highland of
Madagascar. Agricultural Systems, 100(1): 61-71.

86
Uphoff, N. 2002. Opportunities for raising yields
by changing management practices: the system
of rice intensification in Madagascar. In N. Uphoff,
ed. Agroecological innovations: increasing food
production with participatory development,
145-161. London: Earthscan.

Wooldridge, J. M. 2010. Econometric analysis

of cross section and panel data. Second edition.
Cambridge, USA: MIT Press.

World Bank. 2017a. World development

indicators. Rural population (% of total
population), http://data.worldbank.org/indicator/
SP.RUR.TOTL.ZS?locations=MG

World Bank. 2017b. World development

indicators. Agricultural irrigated land (% of total
agricultural land), http://data.worldbank.org/
indicator/AG.LND.IRIG.AG.ZS?locations=MG

You, L., Ringler, C., Wood-Sichra, U., Robertson,

R., Wood, S., Zhu, T., Nelson, G., Guo, Z. and
Sun, Y. 2011. What is the irrigation potential
for Africa? A combined biophysical and

PART II / Chapter 7
socioeconomic approach. Food Policy, 36(6):
770-782.

87
 Chapter 8

The Philippines’s Irrigated Rice

Production Enhancement Project

by
Aslihan Arslan
Daniel Higgins
Paul Winters

Although agriculture generates about 11 per cent the extent to which irrigation management should
of the GDP of the Philippines, it provides be decentralized to communal irrigation systems
livelihoods for 30 per cent of the population and irrigators associations (IAs), and how best
(OECD, 2017). This share is higher in rural the IAs should be organized (Bandyopadhyay,
areas, where 70 per cent of the population is Shyamsundar and Xie, 2007; Kakuta, 2014;
poor and depends on agriculture – primarily rice Nguyen et al., 2015). The impact assessment
production. Rice is one of the most important of the Irrigated Rice Production Enhancement
commodities for the Philippines. In Filipino Project (IRPEP) contributes timely evidence to
diets, rice is the principal source of energy and the ongoing discourse on the policy agenda and
protein, accounting for 46 per cent of calorie programme design.
availability and 34 per cent of protein (WFP,
IRPEP was implemented in three regions of
2017). Therefore, improving rice production
the Philippines between 2010 and 2015 with
and productivity remain major policy priorities.
funding from IFAD, the European Union and
The country has one of the largest yield gaps in
the Government of the Philippines. IRPEP was
South-East Asia (FAO, 2010), and a large focus
a subproject of the Rapid Food Production
of agricultural policy is on how best to improve
Enhancement Project, along with the two-year
smallholder family farm rice productivity in order
Rapid Seed Supply Financing Project that
to keep prices down in the face of production
directly preceded it. While the latter provided
efficiency issues (Bordey et al., 2016), rising
emergency seeds to smallholder rice farmers in
demand from a fast-growing population (PSA,
response to the 2008 food price spike, IRPEP
2016), and climate change effects (Rosegrant
targeted longer-term rice productivity and
et al., 2016). However, trade restrictions in the
income increases by expanding and improving
sector20 undercut rice accessibility by leading to
irrigation infrastructure, building capacity of IAs,
artificially high consumer prices and negatively
and providing farmer field schools (FFSs), rice
affecting food security (OECD, 2017). At the
marketing support, and seed buffer stocks. The
same time, the government allocates large
project was implemented by six agencies of the
amounts of money to irrigation support, but the
Government of the Philippines, with technical
effectiveness of the methods and management
support from IFAD.
of irrigation projects has been questioned
(Inocencio et al., 2016). There is debate about This assessment adds to a small number of
in-depth impact assessments conducted on

20. The Government of the Philippines recently decided to extend quantitative restrictions on rice imports for another
three years (President of the Philippines, 2017).

88
similar projects in the region. An assessment agricultural productivity and income generation,
of an irrigation project in Bohol Province found primarily by improving their supply of irrigation
positive effects on yields and income but noted water, as well as teaching new skills and
that downstream households may have benefited practices, and providing input and marketing
less. It also suggested that downsizing IAs led support. By expanding and rehabilitating irrigation
to increased benefits through reduced water canals to improve water supply, strengthening
conflict (JICA, 2012). Similarly, a study of the IAs to improve management and maintenance,
Communal Groundwater Irrigation Sector Project and teaching improved practices through FFSs,
in Nepal found an increase in yields, especially for the project is expected to help farmers increase
smaller water user groups (ADB, 2012). Finally, their productivity and the quality of their produce,
highlighting the difficulty of conducting such contributing to food security and nutrition. The
assessments, a study of the Mao Lao Irrigation increased yields, plus improved post-harvest
Project in Thailand produced inconclusive processing facilities and other marketing support,
findings, attributed to difficulties in identifying a are expected to contribute to increased incomes.
comparison group of non-beneficiary households Because IRPEP also promoted women’s
that had not received similar support from
involvement in IAs as both members and officers,
elsewhere (Palmer-Jones et al., 2012).
improved gender empowerment is also expected.

This impact assessment focuses only on the

impact of the irrigation, IA capacity-building
The project’s theory of change
and market support activities of IRPEP. This
and research questions is because in the three project regions similar

PART II / Chapter 8
IRPEP was implemented by first identifying FFS and buffer stock support was offered to all
suitable communal irrigation systems (CISs) in non-IRPEP smallholder households. Thus, no
Regions VI, VIII and X of the country, and by suitable comparison group could be found to
offering support to the smallholder rice farmers assess the impact of these components. This
that use them and the IAs that manage them. assessment effectively compares households that
CISs have smaller land coverage than national have received only FFS and buffer stock support
or private irrigation systems and have a more with those that have also received irrigation, IA
decentralized management system run by the IAs capacity-building and marketing support. It also
(Decena, 2016). IRPEP used the following criteria focuses mainly on the production of rice – the
to select beneficiary CISs: project’s target crop and the only crop grown by
• baseline annual paddy productivity less than most of the households. Project implementation
3.78 tons per hectare; and operations differed significantly across
• average landholding size less than 0.76 regions. Activities started two years later in
hectares; Region VI, and a severe typhoon damaged
• low or inadequate supply of water through canals and storage facilities, particularly in
CIS; Region VIII. The impact assessment takes these
• high incidence of poverty; differences into account. With this in mind, and
• sufficient irrigation potential of CIS; based on the expected impact chains of IRPEP
• feasibility of support from implementing and the findings of previous studies, the following
agency; research questions were used to guide this
• willingness and capacity of local government impact assessment:
units to provide timely counterpart funding. • Did IRPEP have the intended impacts on
Figure 8.1 presents the theory of change for irrigation supply, rice productivity, market
IRPEP, which maps the pathways linking its participation and other outcomes? Are there
activities to its expected impacts. The project constraints that limit farmers from achieving
aimed to address constraints to farmers’ these objectives that should be addressed?

89
 Figure 8.1 Theory of change for IRPEP activities

INPUTS AND ACTIVITIES OUTPUTS OUTCOMES IMPACTS

Rehabilitation of • CIS area expanded, Household level Household level

communal irrigation and timely water
• Increased input use • Increased income
systems (CISs) delivery to farmers
• Two-season planting • Increased food
improved
Investment in canal and harvesting security and nutrition
infrastructure leading to • IAs established and
• Increased rice • Increased resilience of
improved water delivery functioning with more
productivity production
and expansion of area members, including
women • Increased rice market • Empowerment of
participation women
• Farmers trained in
water management • Increased rice
and new rice profitability
production
technologies and
techniques
Strengthening of • Improved availability of
irrigators associations seeds during natural
(IAs) disasters
• Strengthening of IA • Farmers provided
rules with information and
• Training of IA skills on post-harvest
leadership management
IA level IA level
• Inclusion of women • Solar dryers, storage
warehouses, and • Increased • Ability to mobilize
other post-harvest membership and IA-owned implements
facilities established participation • Ability to mobilize
• Sustained additional resources
management structure • Ability to expand
• Collection activities
Farmer capacity
and adequate
building
management of water
• Training on water and user fees
crop management • Increased involvement
• Provision of seed of women in IAs
buffer stocks
• Improvement
of post‑harvest
management,
including access to
facilities and marketing
support

90
 • Did the impact of IRPEP vary across The sample was further refined during the
regions, parcel locations, and small and analysis stage to ensure that the impacts were
large IAs? estimated using a valid comparison group.
• Did IRPEP strengthen IA participation
For the household and IA samples, the process
and the capacity of IAs to support their
of identifying the control samples sought to
members and to sustain improvements in
mirror IRPEP’s process for selecting beneficiary
smallholders’ livelihoods?
households and IAs. As the project first identified
suitable CISs, and provided support to the
households that use them and the IAs that
Project outreach and outputs manage them, the first stage of sample selection
The project reached a total of 14,082 smallholder was conducted at the CIS level. First, using a
rice farmer beneficiaries – of whom 4,225 were small set of variables on CIS characteristics
women – in 112 IAs. Irrigation facilities of 109 before the project had started, researchers
CISs were rehabilitated and restored, covering estimated propensity scores for each CIS. These
9,347 hectares of irrigated rice land, exceeding propensity scores were used to identify treatment
the original target of 4,664 hectares (mainly and control CISs that would have had similar
thanks to the expansion of the project to include chances of being selected for the project (Austin,
Region VI). As part of the support provided 2011). Second, regional project staff selected
to the 112 IAs, 5,048 IA officers (30 per cent well-matched treatment and control CISs for
women) and members were trained on irrigation the final sample based on their contextual
operation and maintenance, and given access knowledge. The IA sample was thus comprised
of those managing the 58 treatment and

PART II / Chapter 8
to post-harvest facilities and marketing. The FFS
component (not evaluated here for the reasons 55 control CISs selected through this process,
mentioned above) covered 5,295 farmers, mainly and the household sample was comprised of
focusing on the Palay Check technology (Pinoy 1,082 treatment and 1,022 control households
Rice Knowledge Bank, 2017). randomly selected from within the selected CISs.
Table 8.1 presents the distribution of the CISs
and households covered by the data collection.
Project impact The quantitative data were collected between
March and June 2017. Household questionnaires
Data and methods gathered detailed information about: households’
The analysis of IRPEP’s impact is based sociodemographic characteristics; all agricultural
on quantitative data collected from 2,104 production activities over all seasons during
households and 112 IAs, both evenly spread the previous 12 months at the plot level;
across regions. The quantitative data were agricultural marketing; other income‑generating
supplemented by qualitative information activities; access to basic services; and
gathered through focus group discussions and food security indicators. IA questionnaires
key informant interviews with national, regional gathered information on: the IA structure and
and provincial project staff and IA officers. The improvements since 2010; efficiency in irrigator
key to a robust ex post impact assessment is fee collection; details of irrigation water supply;
to compare treatment (beneficiary) units with post-harvest processing facilities; access to
control (non-beneficiary) units that represent marketing and finance; and membership by
how treatment units would have fared in the gender. The qualitative information gathered
absence of the project. The sampling frames through the focus group discussions and key
of both quantitative and qualitative data were informant interviews helped the research team
designed to ensure that the treatment and better understand the challenges faced during
control households and IAs met this requirement. implementation.

91
 Table 8.1 Distribution of communal irrigation systems (CISs) and household samples

Region Treatment Control

CISs Households CISs Households

Region VI 21 361 19 360

Region VIII 20 359 20 361

Region X 17 362 16 301

Total 58 1,082 55 1,022

The quantitative impacts were estimated using Production and market participation
an inverse probability weighted regression
In contrast to the unanimously positive findings
adjustment model, which reduces important
for irrigation supply, findings for rice production
differences between treatment and control units,
were mixed. First, cropping intensity – a measure
using both propensity scores and a regression
of land use across seasons – was found to
adjustment based on a specification of a model
increase only in Regions VIII and X, whereas
for the outcomes of interest (Austin and Stuart,
expenditure on production inputs increased only
2015). This method was chosen as the main
in Region VI. As shown in figure 8.4, rice yields
model because of its robustness in case of
were found to increase significantly in Region VI,
misspecification of treatment or the outcome
by 13 per cent, and in Region X, by 8 per cent,
models, and it had been used effectively in a
while a significant decrease of 8 per cent was
previous impact assessment of a similar irrigation observed for Region VIII. The qualitative research
project in Madagascar (see Chapter 7). suggests that the lack of impact in Region VIII
was probably caused by the extensive typhoon
damage experienced in the region, as well as
Key impact estimates by a lack of capital for inputs. The value-cost
ratio of rice production – an efficiency indicator
Irrigation water supply that measures the value of harvest divided by
Various indicators were used to assess the the value of inputs – was found to increase

impact of the project on irrigation water supply, only in Region X, suggesting that the increase

and all showed highly positive effects. Overall, in yields in Region VI may have been driven

IRPEP households were 50 per cent more likely by an increased volume of inputs rather than
increased efficiency.
to have a sufficient level of irrigation across both
of the main cropping seasons, and their irrigation Figure 8.5 presents the differential results of
expenditure per hectare was on average 204 IRPEP on market participation, measured in
per cent higher. These positive effects were rice sales revenue. In Region X, the increase in
observed across regions (figures 8.2 and 8.3), yields was translated into a higher proportion
except for irrigation expenditure in Region X, of rice harvest being sold and a higher
which was actually found to decrease despite proportion of processed rice sales, leading to
the apparent improvements in supply. The results an average increase in revenue from sales of
also show that improvements in supply were rice of 128 per cent. However, the same is not
more marked for households based in smaller observed in Region VI, where farmers channelled
IAs and for parcels located downstream from the the yield increase into non-sale uses – mainly
irrigation source. paying back the costs related to their increased

92
Figure 8.2 Differential impacts on likelihood of sufficient irrigation in wet and dry seasons

Region VI 79% (s)

Region VIII 44% (s)

Region X 29% (s)

Large IA 36% (s)

Small IA 88% (s)

Up/Midstream parcels 19% (s)

Downstream parcels 45% (s)

0 50 100
IRPEP impact (change in % probability)
Note: s = statistically significant above 90 per cent confidence level.

Figure 8.3 Differential impacts on irrigation expenditure per hectare

Region VI 316% (s)

Region VIII 297% (s)

Region X -33% (s)

PART II / Chapter 8
Large IA 12% (s)

Small IA 249% (s)

Up/Midstream parcels 139% (s)

Downstream parcels 146% (s)

-50 50 150 250 350

IRPEP impact (% change)
Note: s = statistically significant above 90 per cent confidence level.

inputs – and thus did not experience an increase Although both upstream and downstream parcels
in rice revenue. The ineffectiveness of the experienced a significant increase in yields, the
marketing support provided by IRPEP was widely increase for downstream parcels was found to
noted in the qualitative research, which identified be, on average, about 8 percentage points larger.
the encouragement of collective marketing and However, possibly reflecting their lower need for
the provision of market information services as increased household consumption, upstream
ill‑suited to the operating context and, thus, and midstream households experienced a much
having a low level of uptake. higher effect on rice sales revenue than did
Figures 8.4 and 8.5 also show that the results downstream households.
for yields and market participation continue the
Overall income and other impacts
trend of more favourable results for smaller IAs
and downstream households. A non-significant IRPEP was found to have a significant positive
impact on large IA households is contrasted with impact on household per capita income of
a significant increase in yields of 5 per cent and a 11 per cent. However, as with yields and rice
non-significant increase in rice revenue of 57 per income, this finding masks significant regional
cent for households located in smaller IAs. differences (figure 8.6). Although Region VI

93
 Figure 8.4 Differential impacts on rice yields (harvest per hectare)

Region VI 13.3% (s)

Region VIII -7.9% (s)

Region X 8.1% (s)

Large IA 5.2%

Small IA 4.6% (s)

Up/Midstream parcels 2.5% (s)

Downstream parcels 10.3% (s)

-10 -5 0 5 10 15
IRPEP impact (% change)
Note: s = statistically significant above 90 per cent confidence level.

Figure 8.5 Differential impacts on rice sales revenue per hectare

Region VI -5.9%

Region VIII -37.2%

Region X 127.5% (s)

Large IA -58.4%

Small IA 56.5%

Up/Midstream parcels 92.0% (s)

Downstream parcels 18.6%

-100 -50 0 50 100 150

IRPEP impact (% change)
Note: s = statistically significant above 90 per cent confidence level.

did not experience a significant increase in rice the expense of other income sources. Finally, in
revenue, IRPEP was found to have increased Region VIII, household income was found to have
household income there by about 18 per cent, slightly decreased – although this decrease was
which seems to have been driven by increases in not statistically significant – reflecting the poor
household enterprise and livestock activities. This all‑round project performance in this region.
finding suggests that improvements in irrigation Once again, households based in smaller IAs
supply may allow households to dedicate less showed more favourable results than households
time to rice farming, leaving them more time to in larger IAs (figure 8.6). Household income in
spend on other income-generating activities. small IAs was found to increase significantly by
In contrast, Region X experienced a large increase about 25 per cent, whereas a negative although
in rice revenues but only a small significant not significant effect was observed for larger IA
increase in household income. This suggests households. Assessment of household income
that, unlike households in Region VI, households effects was not done by parcel location because
in Region X chose to devote more time to rice households could have multiple parcels across
farming in response to the IRPEP support, at different locations.

94
Figure 8.6 Differential impacts on household income per capita

Region VI 18.2% (s)

Region VIII -9.1%

Region X 0.4% (s)

Large IA -13.1%

Small IA 25.2% (s)

-20 -10 0 10 20 30
IRPEP impact (% change)
Note: s = statistically significant above 90 per cent confidence level.

The project’s theory of change also anticipated of the project’s effects – is converted into a
positive effects on household nutrition. 102 per cent increase in IA spending per member
A measure of household dietary diversity ranging on operation and maintenance, and an 85 per cent
from 0 to 12 was found to significantly increase increase in overall spending per member.
on average by 0.4, but breaking this figure down

PART II / Chapter 8
by region shows that the result was driven largely
by improvements in Region VI. Also, in contrast
Lessons learned
to most of the other results, dietary diversity was
found to improve in households based in larger IRPEP proved very effective in improving the
IAs but not in smaller IAs. irrigation water supply of households across the
project regions, and this effect translated into
IA-level impacts higher rice yields in two of the regions. However,
Figures 8.7, 8.8 and 8.9 show that IRPEP was mixed results for production, market participation
found to produce positive effects across the and household income highlight the following:
board at the IA level. Although these effects were • Further supplementary support is required
mostly not significant, this is likely to be due to when households are coping with extreme
the small sample size in many cases. The number weather conditions.
of members in IAs was found to have increased • Production efficiency does not automatically
by an average of 32 overall, by an average of increase with improved irrigation supply;
9 for female members, and by an average 3 for other supplementary supports should be
members under 30, with the latter starting from provided to ensure that yields increase in
a low base. There was a significant increase of proportion with increased expenditure on
1.5 in the number of female IA officers, reflecting water and other inputs.
favourably on the IA capacit‑building activities • Capital constraints may have limited the
that focused on increasing women’s involvement beneficiaries’ use of production inputs.
in CIS management. This suggests that the yield impact may
The increased irrigation expenditure found in the be greater if future projects can address
household analysis is reflected at the IA level these constraints.
with a 49 per cent increase in water user fee • Market support must be rethought. More
collection, and an overall increase in IA income research is needed on whether, and how,
per member of 90 per cent. This increase in to encourage collective marketing, and how
income – highly relevant to the sustainability best to provide market information services.

95
 • Future projects must consider household a conducive institutional environment can have
income in its entirety and should be wary of distinct benefits for sustainable improvements
encouraging concentration of livelihoods on in smallholder livelihoods, including significant
a narrow range of activities to the potential increases in IA participation, income,
detriment of livelihood resilience. operation and maintenance expenditure, and
women’s empowerment.
Project performance was notably better for
downstream parcels and households located
in smaller IAs. This highlights the pro-poor
potential of this type of project, and supports
the view that smaller IAs are easier to organize
and mobilize, as found in previous studies.
Strengthening the capacity of IAs combined with

Figure 8.7 Irrigators associations – membership impacts

Number members 31.9

Number female members 9.1

Number female officers 1.5

Number members under 30 3.3

0 10 20 30 40
IRPEP impact (change in members)

Figure 8.8 Irrigators associations – income impacts

Total income
per member 90.1%

Water fee income

per member 49.3%

0 20 40 60 80 100
IRPEP impact (% change)

Figure 8.9 Irrigators associations – expenditure impacts

Total IA expenditure
per member 84.6%

Expenditure on operation
and maintenance per member 102.4%

0 20 40 60 80 100 120
IRPEP impact (% change)

96
References Philippines. Tokyo, http://open_jicareport.jica.
go.jp/pdf/1000008651.pdf
ADB (Asian Development Bank). 2012. Shallow
Kakuta, I. 2014. Impact of participatory irrigation
tubewell irrigation in Nepal: Impacts of the
management on the Bohol Irrigation Project in
Community Groundwater Irrigation Sector Project,
the Philippines. Journal of the Institute for Asian
www.adb.org/sites/default/files/evaluation-
Studies, 41: 123-162.
document/35900/files/ies-nep-final.pdf
Nguyen, M. R., Rola, A. C., Arcala-Hall, R.,
Austin, P. C. 2011. An introduction to propensity
Lizada, J. C., Abansi, C. L. and David, M. E.
score methods for reducing the effects of
2015. Comparative analysis of the national and
confounding in observational studies. Multivariate
communal irrigation systems’ water governance:
Behavioural Research, 46(3): 399-424.
The Philippines case. Journal of Public Affairs and
Austin, P. C. and Stuart, E. A. 2015. Moving Development, 2(2): 1-35.
towards best practice when using inverse
OECD (Organisation for Economic Co-operation
probability of treatment weighting (IPTW) using
and Development). 2017. Agricultural policies in
the propensity score to estimate causal treatment
the Philippines. Paris: OECD Publishing. www.
effects in observational studies. Statistics in
oecd-ilibrary.org/docserver/download/5117011e.
Medicine, 34(28): 3661-3679.
pdf?expires=1513772073&id=id&accname=ocid1
Bandyopadhyay, S., Shyamsundar, P. and Xie, 95767&checksum=B17BBA91326D0491351C32
M. 2007. Yield impact of irrigation management 4F8CBA9855
transfer: Story of the Philippines. Policy Research Palmer-Jones, R., Dilokkunanant, N., Phonyiam,
Working Paper No. 4298. Washington, D.C.: B., Punyaratabandhu, S., Sutthiwongse, T. and

PART II / Chapter 8
World Bank. Hanpongpandh, S. 2012. Impact evaluation of
Bordey, F. H., Moya, P. F., Beltran, J. C. and Mae Lao Irrigation Improvement project, Thailand.
Dawe, D. C., eds. 2016. Competitiveness of 3ie Grantee Final Report. Washington, D.C:
Philippine rice in Asia. Science City of Muñoz, International Initiative for Impact Evaluation (3ie).
Philippines: Philippine Rice Research Institute. Pinoy Rice Knowledge Bank. 2017. Palay Check
Decena, F. L. C. 2016. Analysis of the effects of system for the Philippine irrigated lowland rice,
various irrigation service fees for national irrigation www.pinoyrice.com/palaycheck/
systems in the Philippines. FFTC Agricultural President of the Philippines. 2017. Executive
Policy Article. Taipei, Taiwan Province of China: Order No. 23: Extending the effectivity of the
Food and Fertilizer Technology Center for the most-favoured-nation rates of duty on certain
Asian and Pacific Region (FFTC-AP), http://ap.fftc. agricultural products under Republic Act No.
agnet.org/ap_db.php?id=585 10863, www.officialgazette.gov.ph/2017/04/28/
FAO (Food and Agriculture Organization of the executive-order-no-23-s-2017/
United Nations). 2010. Global agro-ecological PSA (Philippine Statistics Authority). 2016.
zones: Yield and production gaps, www.fao.org/ Highlights of the Philippines population 2015
nr/gaez/about-data-portal/yield-and-production- Census of Population. Quezon City, Philippines,
gaps/en/ https://psa.gov.ph/content/highlights-philippine-
Inocencio, A. B., Ureta, C., Baulita, A. and Baulita, population-2015-census-population
A. 2016. Technical and institutional evaluation Rosegrant, M. W., Perez, N., Pradesha, A. and
of selected national and communal irrigation Thomas, T. S. 2016. The economywide impacts
systems and characterization of irrigation sector of climate change on Philippine agriculture.
governance structure. Discussion Paper Series Climate Change Policy Note 1. Washington, D.C.:
No. 2016-12. Quezon City, Philippines: Philippine International Food Policy Research Institute.
Institute for Development Studies.
WFP (World Food Programme). 2017. Food
JICA (Japan International Cooperation Agency). security and nutrition in the Philippines. Strategic
2012. Impact evaluation of the Bohol Irrigation Review, https://docs.wfp.org/api/documents/
Project (Phase 2) in the Republic of the WFP-0000015508/download/
97
©IFAD/Sarah Morgan

98
PART III

Analysing the evidence on

thematic activities

99
 Chapter 9

Land administration
interventions: a systematic
review by
Daniel Higgins
Tim Balint
Harold Liversage
Paul Winters

The agenda for the Sustainable Development Despite the investments by governments
Goals (SDGs) cites secure access to land as and donor agencies, questions remain over
a key means of meeting SDG 2: “end hunger, the validity of the expected benefits of land
achieve food security and improved nutrition and tenure security. Many observers call for more
promote sustainable agriculture.” With proof of clarity on how best to implement land tenure
ownership and reduced risk of appropriation,
security interventions in complex contexts
rural households with secure tenure are
(Everest‑Phillips, 2008; Gignoux, Macours and
expected to: invest more in production and land
Wren-Lewis, 2013). To help fill this knowledge
conservation (Meinzen-Dick, 2009); have greater
gap, IFAD commissioned a systematic review
access to collateral-based credit (De Soto, 2000);
of high-quality evidence on the effects of
rent and sell their land with more ease (Deininger
and Binswanger, 1999); and experience less strengthening rural peoples’ land tenure security
land-related conflict (Nuesiri, 2014). In addition, in an effort to gain insights for a key area of its
land tenure security is expected to contribute to portfolio. This review aims to test the expected
women’s empowerment when women’s names effects and to provide insights to guide policy
are included on land titles (USAID, 2016). and practice, taking advantage of the significant
number of studies published since a similar
These perceived benefits have led to large
investments in land tenure interventions, review was conducted in 2012 (Lawry et al.,
particularly through national land reform, which 2017).
has been widespread across Asia and Africa The systematic review presented in this
since the early 1990s (Holden, Otsuka and chapter is the only known example of such
Deininger, 2013). Specific activities implemented
a rigorous review being conducted based on
have consisted mainly of:
one organization’s portfolio, and it is expected
• demarcating and issuing formal titles to
to be the first in a series of reviews of various
already-held land parcels;
IFAD‑funded activity types. In considering all
• allocating and redistributing formally
forms of land tenure security activity, the review’s
registered land to those with low
landholdings; insights are also applicable to non-IFAD projects.
• strengthening institutions tasked with Moreover, the review has additional external value
administering land ownership; as a potential best-practice example of how to
• raising awareness of land rights and how to systematically use evidence in decision-making
obtain them. within a development organization.

100
Review methodology study findings in a systematic review is to
combine the quantitative study results through a
Owing to its rigour and neutrality, the systematic meta-analysis, producing aggregate estimates of
review methodology is considered the gold effects on a given outcome (McDonald, 2008).
standard among approaches to literature However, given the review’s coverage of a wide
reviews (Bryman, 2008). This review followed range of land tenure activities, contexts and
the standard procedure for the method (Higgins outcomes, and its aim of drawing rich in-depth
and Green, 2011), first conducting an exhaustive insights, the research team decided that such
search of all relevant sources of published aggregate figures would not suit the nature or
and unpublished literature for relevant studies, purpose of the review.
using a detailed set of predefined inclusion
criteria.21 These criteria restricted eligible studies
to those undertaken after 1990 that: (1) use
Review findings
quantitative or qualitative data to investigate the
impact of land tenure security in a middle- or
Distribution of the research base
low‑income country; and (2) use a methodology
that exceeds a specified level of rigour. For From among almost 30,000 identified studies,
the second criterion, quantitative studies were 278 were selected as eligible based on title and
mainly required to have assessed the effects of abstract screening. After full-text screening and
land tenure security using a well-constructed assessment of methodological quality, a final set
comparison/control group, and qualitative studies of 59 studies – 36 based on quantitative data,
needed to perform well on the widely used and 23 based on qualitative data – were selected
Critical Appraisal Skills Programme checklist for a transparent and neutral synthesis of their
(CASP, 2015). Eligible studies were also given findings.
a ranking for their methodological rigour, which
Figure 9.1 shows the distribution of studies
was then considered when interpreting the
by publication/completion year. No eligible
study findings.
studies from 1990-1999 were identified, and
The research team synthesized the studies a particularly high number were identified from
identified through the search using the narrative the past five years. These two results could be
approach, which was conducted in a way that due to an increasing research focus on land
ensured that the findings of all studies were tenure effects, or to recent improvements in
considered. Another method of synthesizing methodological rigour.

Figure 9.1 Distribution of studies by publication year

12
10 10
10
PART III / Chapter 9

8
8

6
5 5 5
4 4
4
2 2
2
1 1 1 1
0 0 0
0
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016

21. The search was conducted between February and April 2016.

101
 Table 9.1 shows the distribution of studies by In terms of the specific focus of the studies
location, with countries in Asia and the Pacific, and the outcomes they assessed, both the
and in East and Southern Africa receiving the quantitative and qualitative studies investigated
most focus. In terms of individual countries, mainly land reforms, and interventions that issued
Ethiopia, China, Viet Nam and Peru were the
land ownership certificates and conducted land
most covered – unsurprising given their histories
parcel mapping. A number also assessed the
of land reform measures.
effects of varying levels of land tenure status in
Table 9.2 shows the distribution of studies non-intervention settings. The quantitative studies
by their methodology and their ranking for
mainly focused on how interventions affected
methodological rigour. Few studies, either
investment, and some also assessed effects
quantitative or qualitative, scored highly in terms
on credit access and income. Few focused on
of methodological rigour. Among quantitative
agricultural productivity, land rental, land sales,
studies, only two employed the randomized
control trial design for constructing a comparison/ or conflict. Qualitative studies were concerned
control group, and many of the qualitative studies mainly with how the interventions and reforms
did not report enough details of the research were implemented rather than the outcomes
method to justify a higher ranking. themselves.

Table 9.1 Distribution of studies by location

Number of Number of
Region/country Region/country
studies studies

Asia and the Pacific 24 Sub-Saharan Africa 25

China 6 East and Southern Africa 21

Viet Nam 5 Ethiopia 8

India 4 Uganda 4

Cambodia 2 South Africa 3

Fiji 2 United Republic of Tanzania 2

Pakistan 1 Madagascar 1

Philippines 2 Malawi 1

Nepal 1 Rwanda 1

Tajikistan 1 Zimbabwe 1

Latin America and Caribbean 10 West and Central Africa 4

Peru 5 Ghana 2

Bolivia (Plurinational State of) 2 Benin 1

Guatemala 1 Democratic Republic of the Congo 1

Mexico 1

Nicaragua 1

102
Table 9.2 Distribution of studies by methodology and ranking

Study type Number of studies

Quantitative studies

Methodology

Instrumental variable 14

Propensity score matching (PSM) 8

Difference-in-difference (DD) 6

PSM and DD 4

Randomized control trial 2

Regression discontinuity 2

Risk of bias level

Low 2

Medium-low 15

Medium-high 19

Qualitative studies

Grade

A 3

B 6

C 14

Themes in the study findings of the included quantitative studies on these

three outcomes, separating them by study focus.
The findings of the 59 identified studies can be
The findings show a clear divergence between
separated into those focusing on the economic,
investment and income effects, with the majority of
social and environmental effects of land tenure
studies showing positive effects on investment but
security. Although the studies analyse different
PART III / Chapter 9

not on income. With most of the studies focusing

settings (ranging from different land tenure
security interventions to situations where land on reasonably short periods, it is unclear whether

tenure security already exists), evidence for this divergence occurs because the expected

positive effects across such different settings income effect is not valid – which would be
can provide an evidence-based rationale for surprising given the support for positive investment
investments in land tenure security. effects – or because the studies did not have
time to capture the income effects that may have
Economic effects followed increased investment.

Productive investment, productivity and income Some qualitative studies highlight factors that
were among the most heavily studied effects in the may mitigate the economic effects. Many
identified studies. Figure 9.2 presents the findings studies noted that formally titled households still

103
 Figure 9.2 Quantitative results of land tenure security on productive investment, productivity and income

No effect Positive effect

Effect of land formalization

intervention on:
Productive investment 2 6

Commercial investment 0 2

Productivity 1 1

Income 5 0

Effect of varying level of tenure

security in a natural setting on:
Productive investment 0 2

Productivity 2 1

Effects of vouchers to
purchase own land on:
Productive investment 1 0

Productivity 0 1

Income 0 2

Effect of strengthened
collective rights on:
Productive investment 1 0

Productivity 0 1

Income 1 0

Effect of strengthened
forest rights on:
Income 0 2

Effect of title against renting

or sharecropping on:
Productive investment 0 1

Effect of varying knowledge

of land law on:
Productive investment 0 1

Number of studies

perceived their tenure security to be low, most corruption, elite capture, and expensive,
often as a result of past experience of state-led complex application procedures within local land
land appropriation, and as a result they did not administration institutions.
change their investment behaviour. In addition, The other main expected economic effect of
many qualitative studies found that certain land tenure security is improved credit access,
groups – mainly women, minorities and the but it received little support from the quantitative
very poorest – were often hindered or excluded or qualitative studies that investigated this link.
from obtaining titles through interventions and According to the qualitative studies, the main
reforms, mainly because of discrimination, explanation was that lending institutions often

104
employ non-collateral-based lending policies, approach to farming, in the knowledge that their
thus nullifying the mechanism through which land was safe from appropriation. However, other
credit access is expected to improve. In addition, qualitative studies noted that poorly designed or
households were found to face other barriers to poorly implemented interventions can actually
credit access unrelated to land tenure security, induce overfarming of titled land. This result
including excessive distance from lending occurred mainly when households were required
institutions, and long and complex loan application to provide ongoing payments for their land title or
processes. were threatened with eviction if the land was not
used. In these cases, neither actual nor perceived
Social effects security was increased, leading to detrimental
effects on the environment.
The quantitative studies found unanimously
positive effects on gender empowerment for
interventions that specifically target women’s
secure land access. However, three of the Key implications of review
four studies were based in one country – India findings
– raising doubts about whether the effect crosses
This review aimed to use high-quality evidence to
contexts. Moreover, one qualitative study of a
test the expected benefits of land tenure security
joint-titling programme, which found positive
and to generate insights to inform future work
effects for joint-titled women, noted that women
using land tenure security to reduce poverty and
were often excluded from participating, either by
to promote rural development. The synthesized
their spouses or by local institutions.
findings of 59 studies identified through an
Another social effect commonly expected from exhaustive search point to the following policy,
land tenure security is a reduction in conflict practice and research implications:
over land. Although the handful of quantitative • It is valid to expect increased land
studies that tested this effect produced tenure security to lead farmers to invest
inconclusive outcomes, some of the qualitative more in agricultural production and land
studies provided important insights. The most conservation. However, interventions must
salient was that interventions that do not pay ensure that farmers’ perceptions of tenure
proper consideration to the pre-existing land security increase along with their actual
administration system can actually increase tenure status; whether farmers trust the
land contestation, either through ineffective land state to protect their access rights is a
mapping or by disrupting and weakening existing key consideration. Interventions must also
mechanisms for conflict resolution. address the potential economic and social
barriers – discrimination, long and expensive
Environmental effects application procedures, and elite capture
of land administration institutions – facing
Eight quantitative studies assessed the effect of
intended beneficiaries.
land tenure security on environmentally beneficial
• The expected increase in credit access
PART III / Chapter 9

investment, using indicators such as organic

seemingly relies on a number of contingent
fertilizer use and frequency of land fallowing. Few
factors. Practitioners hoping to improve
focused on any other environmental outcomes,
credit access using land tenure security
such as soil degradation or deforestation.
should assess whether local lending
Of the studies focused on conservation institutions have collateral-based lending
investments, all but one observed a positive policies, are within spatial reach of
effect. A number of qualitative studies suggest beneficiaries, and have manageable
that these effects were probably caused by application processes. On the demand
households being induced to take a longer-term side, in light of the positive findings on

105
 investment in the absence of a credit effect, References
practitioners should also consider whether
beneficiaries are actually capital constrained General references
and, therefore, in need of credit.
Bryman, A. 2008. Social research methods. Third
• There appears to be strong potential for
edition. Oxford: Oxford University Press.
boosting women’s empowerment by
improving their access to secure land, CASP (Critical Appraisal Skills Programme). 2017.
although more research is needed on CASP qualitative checklist, http://docs.wixstatic.
interventions beyond India. However, to com/ugd/dded87_25658615020e427da194a325
ensure that all women can benefit, the e7773d42.pdf
research highlights the need to understand
De Soto, H. 2000. The mystery of capital:
community and household gender dynamics,
Why capitalism triumphs in the West and fails
and the other potential barriers women face
everywhere else. New York, USA: Basic Books.
in obtaining secure access to land.
• To avoid increasing the level of land conflict Deininger, K. and Binswanger, H. 1999. The
or producing other unintended effects, land evolution of the World Bank’s land policy:
tenure security interventions should seek to Principles, experience, and future challenges.
fully understand existing land administration World Bank Research Observer, 14(2): 247-276.
systems, and to align activities with them Everest-Phillips, M. 2008. The myth of “secure
accordingly. property rights”: Good economics as bad history
• Further research is required in a number and its impact on international development.
of areas: (1) capturing the longer-term SPIRU Working Paper No. 23. London: Overseas
effects of land tenure security; (2) assessing Development Institute, Strategic Policy Impact and
the effect of women-focused land tenure Research Unit.
interventions on gender empowerment
beyond India; and (3) filling the research Gignoux, J., Macours, K. and Wren-Lewis, L.
gaps relating to effects on land rental, land 2013. Evaluating the impact of land administration
sales and conflict. programs on agricultural productivity and
rural development. IDB Technical Note No.
The results of this review will feed into the Global IDB‑TN-506. Washington, D.C.: Inter-American
Land Indicators Initiative to help produce better Development Bank.
impact measurement indicators and techniques.
In collaboration with the Millennium Challenge Higgins, J. P. T. and Green, S., eds. 2011.
Corporation, IFAD is also developing an Cochrane handbook for systematic reviews of
interactive study database that the development interventions. Version 5.1.0 [updated March 2011].
community can use as a one-stop source for The Cochrane Collaboration, www.handbook.
high-quality land tenure security studies. cochrane.org

Holden, S., Otsuka, K. and Deininger, K. 2013.

Land tenure reforms, poverty and natural
resource management: Conceptual framework.
In S. Holden, K. Otsuka and K. Deininger, eds.
Land tenure reform in Asia and Africa: Assessing
impacts on poverty and natural resource
management. London: Palgrave Macmillan.

106
Lawry, S., Samii, C., Hall, R., Leopold, A., Hornby, Brulé, R. 2010. Changes in India’s rights regime
D. and Mtero, F. 2017. The impact of land property and the implications for gender parity: Evidence
rights interventions on investment and agricultural from regression discontinuity analysis of panel data
productivity in developing countries: A systematic for 17 Indian states. American Political Science
review. Journal of Development Effectiveness, Association 2010 annual meeting paper.
9(1): 61-81. Deininger, K. and Jin, S. 2002. The impact of
McDonald, J. H. 2008. Handbook of biological property rights on households’ investment, risk
statistics. Baltimore, USA: Sparky House. coping, and policy preferences: Evidence from
China. Economic Development and Cultural
Meinzen-Dick, R. 2009. Property rights for
Change, 51(4): 851-882.
poverty reduction? DESA Working Paper No. 91.
New York, USA: United Nations Department of Deininger, K., Ali, D. A. and Alemu, T. 2009.
Economic and Social Affairs. Impacts of land certification on tenure security,
investment and land market participations:
Nuesiri, E. O. 2014. Secure tenure must be the
Evidence from Ethiopia. Land Economics, 87(2):
foundation of adaptation in East and Southern
312-334.
Africa. Conflict sensitive adaptation: Use human
rights to build social and environmental resilience. Deininger, K., Ayalew, D. and Yamano, T. 2006.
Brief 4. Indigenous Peoples of Africa Co- Legal knowledge and economic development: The
ordinating Committee and IUCN Commission on case of land rights in Uganda. World Bank Policy
Environmental, Economic and Social Policy. Research Working Paper 3868. Washington, D.C.:
World Bank.
USAID (US Agency for International Development).
2016. Fact sheet: Land tenure and women’s Deininger, K., Goyal, A. and Nagarajan, H. 2010.
Inheritance law reform and women’s access to
empowerment. Washington, D.C., www.land-links.
capital: Evidence from India’s Hindu Succession
org/issue-brief/fact-sheet-land-tenure-womens-
Act. World Bank Policy Research Paper No. 5338.
empowerment
Washington, D.C.: World Bank.
References for quantitative studies identified Do, Q.-T. and Iyer, L. 2007. Land titling and rural
through the search transition in Vietnam. Economic Development and
Ali, D. A., Deininger, K. and Goldstein, M. 2014. Cultural Change, 56(3): 531-579.
Environmental and gender impacts of land Dower, P. C. and Pfutze, T. 2013. Specificity of
tenure regularization in Africa: Pilot evidence from control: The case of Mexico’s ejido reform. Journal
Rwanda. Land and Property Rights, 110: of Economic Behaviour and Organisation, 91:
262-275. 13-33.
Ameha, A., Nielsen, O. J. and Larsen, H. O. Fatema, N. and Kibriya, S. 2015. Can land and
2014. Impact of access and benefit sharing on food entitlement reduce conflict? Evidence from
livelihoods and forest: Case of participatory forest violence prone Eastern DR Congo. Selected paper
management in Ethiopia. Ecological Economics, prepared for presentation at the 2015 Agricultural
PART III / Chapter 9

97: 162-171. and Applied Economics Association and Western

Agricultural Economics Association Joint Annual
Bellemare, M. F. 2013. The productivity impacts
Meeting, San Francisco, CA, July 26-28.
of formal and informal land rights: Evidence from
Madagascar. Land Economics, 89(2): 272-290.

107
 Field, A. J., Field, E. and Torero, M. 2006. Property Keswell, M. and Carter, M. 2011. Poverty and land
rights and crop choice in rural Peru, 1994-2004. distribution: Evidence from a natural experiment.
MTID Discussion Paper No. 100. Washington, UNU-WIDER Working Paper No. 2011/46.
D.C.: International Food Policy Research Institute. Helsinki: United Nations University – World Institute
for Development Economics Research.
Fort, R. 2008. The homogenization effect of
land titling on investment incentives: Evidence Kousar, R. and Abdulai, A. 2015. Impact of
from Peru. NJAS – Wageningen Journal of Life non-farm work and land tenancy contracts on
Sciences, 55(4): 325-343. soil conservation measures. Paper prepared for
the 88th annual conference of the Agricultural
Gelo, D. and Koch, S. F. 2014. The impact of
Economics Society, April 9-11, 2014, Paris.
common property right forestry: Evidence from
Ethiopian villages. World Development, 64: Kumari, R. and Nakano, Y. 2015. Does land lease
395-406. tenure insecurity cause decreased productivity
and investment in the sugar industry? Evidence
Goldstein, M., Houngbedji, K., Kondylis, F., from Fiji. Tsukuba Economics Working Papers No.
O’Sullivan, M. and Selod, H. 2015. Formalizing 2015-001. Tsukuba, Japan: University of Tsukuba.
rural land rights in West Africa: Early evidence from
Lawson-Remer, T. 2013. Do stronger collective
a randomized impact evaluation in Benin. Policy
property rights improve household welfare?
Research Working Paper No. 7435. Washington,
Evidence from a field study in Fiji. World
D.C.: World Bank.
Development, 43: 207-225.
Gordoncillo, P. U. 2012. The economic effects of
Li, G., Rozelle, S. and Huang, J. 2000. Land
the Comprehensive Agrarian Reform Programme
rights, farmer investment incentives, and
in the Philippines. Journal of ISSAAS (International
agricultural production in China. Working
Society for Southeast Asian Agricultural Sciences),
Paper No. 00-024. Davis, USA: University of
18(1): 76-86.
California, Davis, Department of Agricultural and
Hagos, H. G. and Holden, S. 2013. Efficiency and Resource Economics.
productivity differential effects of land certification
Ma, X., Heerink, N., van Ierland, E. C., van den
program in Ethiopia. IFPRI Discussion Paper
Berg, M. and Shi, X. 2013. Land tenure security
01295. Washington, D.C.: International Food
and land investments in Northwest China. China
Policy Research Institute. Agricultural Economic Review, 5(2): 281-307.
Ito, J., Bao, Z. and Ni, J. 2016. Land rental Macours, K. 2009. Land titles and conflicts
development via institutional innovation in rural in Guatemala. IDB Working Paper CS 1-164.
Jiangsu, China. Food Policy, 59: 1-11. Washington, D.C.: Inter-American Development
Kemper, N., Ha, L. V. and Klump, R. 2015. Bank.
Property rights and consumption volatility: Markussen, T. 2008. Property rights, productivity,
Evidence from a land reform in Vietnam. World and common property resources: Insights from
Development, 71: 107-130. rural Cambodia. World Development, 36(11):
Kemper, N., Klump, R. and Schumacher, H. 2277-2296.

2011. Representation of property rights and credit Mendola, M. and Simtowe, F. 2015. The welfare
market outcomes: Evidence from a land reform in impact of land redistribution: Evidence from a
Vietnam. Proceedings of the German Development quasi-experimental initiative in Malawi. World
Economics Conference, Berlin, No. 45. Development, 72: 53-69.

108
Nakasone, E. 2011. The impact of land titling on forestry regimes in Nepal. PhD dissertation,
labor allocation: Evidence from rural Peru. IFPRI Massey University.
Discussion Paper 01111. Washington, D.C.:
Bezu, S. and Holden, S. 2014. Demand for
International Food Policy Research Institute.
second-stage land certification in Ethiopia:
Petracco, C. K. and Pender, J. 2009. Evaluating Evidence from household panel data. Land Use
the impact of land tenure and titling on access to Policy, 41: 193-205.
credit in Uganda. IFPRI Discussion Paper 00853. Bose, P. 2013. Individual tenure rights, citizenship,
Washington, D.C.: International Food Policy and conflicts: Outcomes from tribal India’s forest
Research Institute. governance. Forest Policy and Economics, 33:
Reyes-Garcia, V., Orta-Martinez, M., Gueze, M., 71-79.
Luz, A. C., Paneque-Galvez, J., Macia, M. J. Bradstock, A. 2006. Implementing land reform in
and Pinto, J. 2012. Does participatory mapping South Africa’s Northern Cape Province. Network
increase conflicts? A randomized evaluation in the Paper No. 137. London: Overseas Development
Bolivian Amazon. Applied Geography, 34: 650-658. Institute, Agricultural Research and Extension
Santos, F., Fletschner, D., Savath, V. and Network.
Peterman, A. 2014. Can government-allocated Broegaard, R. B. 2009. Land access and titling
land contribute to food security? Intrahousehold in Nicaragua. Development and Change, 40:
analysis of West Bengal’s Microplot Allocation 149-169.
program. World Development, 64: 860-872.
Cardona, W. C., de Jong, W., Zuidema, P. A.
Wiig, H. 2013. Joint titling in rural Peru: and Boot, R. 2014. Diverse local regulatory
Impact on women’s participation in household responses to a new forestry regime in forest
decision‑making. World Development, 52: 104-119. communities in the Bolivian Amazon. Land Use
Policy, 39: 224-232.
Xie, L., Berck, P. and Xu, J. 2016. The effect on
forestation of the collective forest tenure reform in Castella, J.-C., Boissau, S., Thanh, N. H. and
China. China Economic Review, 38: 116-129. Novosad, P. 2006. Impact of forestland allocation
on land use in a mountainous province of Vietnam.
References for qualitative studies identified Land Use Policy, 23(2): 147-160.
through the search
Chen, J. and Innes, J. L. 2013. The implications
Ali, D. A., Collin, M., Deininger, K., Dercon, S., of new forest tenure reforms and forestry property
Sanderfur, J. and Zeitlin, A. 2014. The price of markets for sustainable forest management
empowerment: Experimental evidence on land and forest certification in China. Journal of
titling in Tanzania. World Bank Policy Research Environmental Management, 129: 206-215.
Paper 6908. Washington, D.C.: World Bank.
Gondo, T. and Kyomuhendo, V. 2011. Between
Antwi-Agyei, P., Dougill, A. J. and Stringer, L. C. reality and rhetoric in land conflicts: An anecdotal
2015. Impacts of land tenure arrangements on anatomy of the lawful, bonafide occupants and
PART III / Chapter 9

the adaptive capacity of marginalised groups: The customary tenants in Kyenjonjo district, Uganda.
case of Ghana’s Ejura Sekyedumase and Bongo Land Tenure Journal, 1: 27-50.
districts. Land Use Policy, 49: 203-212.
Keating, N. B. 2013. Kuy alterities: The struggle to
Bajracharya, B. 2008. Institutional factors that conceptualise and claim Indigenous land rights in
influence access of the poor to forest benefits: neoliberal Cambodia. Asia Pacific Viewpoint, 54(3):
Case studies of community and leasehold 309-322.

109
 Kuusaana, E. D. and Bukari, K. N. 2015. Land Vista B. M., Nel, E. and Binns, T. 2012. Land,
conflicts between smallholders and Fulani landlords and sustainable livelihoods: The impact
pastoralists in Ghana: Evidence from the Asante of agrarian reform on a coconut hacienda in the
Akim North District (AAND). Journal of Rural Philippines. Land Use Policy, 29(1): 154-164.
Studies, 42: 52-62.
Yami, M. and Snyder, K. A. 2015. After all, land
Mazhawidza, P. and Manjengwa, J. 2011. The belongs to the state: Examining the benefits of
social, political and economic transformative land registration for smallholders in Ethiopia. Land
impact of the Fast Track Land Reform on the Degrading Development, 27(3): 465-478.
lives of women farmers in Goromonzi and
Vungu‑Gweru Districts of Zimbabwe. Harare:
Women Farmers Land and Agriculture Trust and
Center for Applied Social Sciences, University of
Zimbabwe.

Mukhamedova, N. and Wegerich, K. 2014. Land

reforms and feminization of agricultural labor
in Sughd province, Tajikistan. IWMI Research
Report No. 157. Colombo: International Water
Management Institute.

Pedersen, R. H. 2015. A less gendered access

to land? The impact of Tanzania’s new wave of
land reform. Development Policy Review, 33(4):
415-432.

Rakolojane, M. J. 2013. The gender dimensions

of land reform in South Africa: A case study of
Daggakraal Rural Housing and Resettlement
project. PhD dissertation, University of South
Africa.

Sikor, T. 2006. Politics of rural land registration

in post-socialist societies: Contested titling in
villages of northwest Vietnam. Land Use Policy,
23(4): 617-628.

Shibeshi, G. B. 2012. Lessons from systematic

evaluation of land administration systems: The
case of Amhara National Regional State of
Ethiopia. World Development, 68: 282-295.

Teklu, A. 2005. Land registration and women’s

land rights in Amhara region, Ethiopia. London:
International Institute for Environment and
Development.

Van Leeuwen, M. 2015. Localising land

governance, strengthening the state:
Decentralization and land tenure security in
Uganda. Journal of Agrarian Change, 17(1):
208-227.

110
Chapter 10

Smallholder irrigation in a
broader productivity and value
chain framework: implications
for IFAD investments by
Chandra Madramootoo
Rui Benfica

Irrigation is critical for smallholder agriculture. Irrigation covers about 16 per cent of the world’s
Insufficient rainfall, changing weather patterns, cropland, and these irrigated lands make a
and climatic variability all reduce crop yields disproportionate contribution to global food
and often lead to crop failure and food scarcity. security, producing 36 per cent of the world’s
In arid and semi-arid regions, prolonged drought food (World Food Summit, 1996). However,
has led to hunger and malnutrition, and in some irrigated area is unevenly distributed worldwide,
cases to famine. and net irrigated area is declining. Whereas
agricultural land under irrigation was estimated to
Irrigation offers protective insurance against
be about 263 million hectares in the mid-1990s,
variations in rainfall and drought. It meets
in 2013 it was estimated at about 255 million
cropwater demands by increasing soil moisture
hectares (Amede, 2015). More than 70 per cent
and, if properly managed, enhances plant uptake
of the irrigated area is in Asia. Sub-Saharan
of nutrients. It leads to changes in cropping
Africa, a highly food-insecure region, has just
patterns – farmers who depend on rainfall
over 3 per cent of the irrigated area (FAO, 2016).
generally have only one cropping season per
While the area that could be exploited globally
year, whereas irrigation permits at least two
for irrigation could be significantly more 300
cropping seasons a year. Irrigation supports the
million hectares, there are several constraints
production of high-value crops and improves
to achieving this potential, particularly in Africa.
overall crop quality. Recent studies show that
These constraints include: inadequate technical
beneficial agricultural management practices
skills and institutional capacity; lack of financial
in combination with irrigation increase food
capital (You et al., 2010); problems with water
production and minimize the impacts of climate
and land rights (Gunda et al., 2013; Wanvoeke
change on food security (Verge, De Kimpe
PART III / Chapter 10

et al., 2016); transboundary constraints; future

and Desjardins, 2007; Heard et al., 2012).
climates (Muller et al., 2015); and difficulties
Apart from food security, irrigation systems, if
obtaining the necessary environmental permits.
properly designed and managed, can contribute
to poverty alleviation and environmental Over the years, IFAD has made loan and grant
sustainability (Walker, 1989; Madramootoo, Wiyo funds available for the development of new
and Enright, 1992; Hussain and Hanjra, 2004; irrigation schemes, rehabilitation of existing
Hanjra, Tadele and Gutta, 2009; Dukhovny, schemes, the capacity-building of water
2011), and drive economic development (Sezen user associations (WUAs), and strengthening
et al., 2010; Gao et al., 2012). processes such as irrigation management

111
 Box 10.1 IFAD’s programmatic approach to water/irrigation investments by geographic region

Region Programmatic focus

Small-scale garden irrigation; watershed management; soil and water

conservation; water harvesting, small dams and ponds, tube wells; lowland
West and Central Africa
development for rice irrigation; climate-smart agriculture; economic poles/
clusters; value chains.

Irrigation infrastructure expansion, rehabilitation and modernization;

watershed and land management; water resources and basin
East and Southern Africa
development; soil and water management; adaptation to climate change;
farmer capacity- and institution-building; markets and value chains.

Improvements in irrigation infrastructure; expansion of irrigation facilities;

water harvesting, pond and tube-well development; drought proofing;
Asia and the Pacific drinking-water supply and sanitation; strengthening of water user
associations; farmer capacity- and institution-building; markets and
value chains.

Limited involvement in irrigation in the region. Irrigation improvements;

capacity-building of water user associations; natural-resource protection
Latin America and Caribbean
and management; water harvesting. Limited indication about markets and
value chains in these water/irrigation projects.

Irrigation development, expansion and rehabilitation; strengthening of water

Near East and North Africa user associations; watershed management; soil and water conservation;
climate-smart agriculture; markets and value chains.

transfers. This chapter argues that IFAD can these 11 projects focus primarily on smallholder
significantly increase the water-use efficiency, irrigation systems, they include complementary
irrigation system performance, and crop water components that cover capacity and institution
and economic productivity of smallholder building, management, extension and technical
irrigation schemes, as well as the financial returns advice, markets, inputs, rural infrastructure
to irrigation, by linking the design, operation and and credit. Without these other functions and
management of irrigation systems to markets, services, the irrigation projects could not achieve
value chains, and input supplies and services. their full potential. The largest investments are in
It seeks to explore business opportunities Asia and the Pacific. The primary characteristics
and technological innovations in smallholder of IFAD’s irrigation investments in its five
irrigation, and to develop a framework to inform operational regions are shown in box 10.1.
programmatic development relevant to IFAD in
building irrigation value chains.
IFAD’s approach to value chain
development
IFAD’s irrigation portfolio
IFAD recognizes that poverty reduction,
IFAD has few projects focused exclusively on economic resilience among the rural poor, and
water, and even fewer focused exclusively rural transformation are increasingly dependent
on irrigation. It currently has some 76 active on a value chain approach. Whereas in 1999 only
projects with investments in irrigation/water, 3 per cent of IFAD projects had a value chain
with only about 11 devoted primarily to both the development component, today that figure is
hard and soft components of irrigation. While more than 50 per cent. This significant increase

112
is linked to the adoption of IFAD’s Private Sector capacity of value chain actors, policy framework
Development and Partnership Strategy in 2005. and the business environment. IFAD observes
While earlier value chain projects centred on that, for commodity value chains where
cooperatives, today public‑private-producer marketing arrangements or prices are controlled
partnerships (4Ps) are being promoted. by governments, these tend to distort markets
and displace the private sector.
IFAD projects use three types of value chain
models: producer-driven, buyer-driven and
intermediary-driven. In the producer-driven
model, producers are organized in groups, IFAD and irrigation value chains
associations or cooperatives that take on
Of the 76 active IFAD projects with a water/
production and, sometimes, post-production
irrigation component, 43 have a component
functions aiming at capturing the largest share
on value chains. Although in some projects the
of the retail value. Through IFAD and other
value chain model is not specified, most of the
project support, these groups become strong
projects seem to favour the producer‑driven
enough to network, search for buyers and
and the intermediary-driven models, or a
negotiate contracts with them. In many cases,
combination of the two. For example, the
they become the leaders of the value chains
economic clusters that are part of the small
and are able to transform, process, transport
garden irrigation projects in Benin and the Niger
and market their commodities themselves,
require producers and intermediaries, such
thereby earning a higher share of value addition
as brokers and retailers, to work in tandem.
along the value chain. Examples of such
The Market Access Alliances in Ethiopia’s
models are most prominent in Latin America,
Participatory Small-Scale Irrigation Development
where farmers’ organizations are strong and
Programme II (PASIDP II) are a combination of
well established. In the buyer-driven model,
commodity organizations, farmers’ organizations
the private sector (e.g. processors, retailers,
and retailers. The Swaziland Market-led Project
traders and wholesalers) organizes producers
links irrigated producers to small local traders
into suppliers through various contractual
and larger wholesalers, but is not a complete
arrangements (e.g. outgrower schemes, contract
buyer-driven model. At present, the full
farming and joint ventures). This model is
buyer‑driven model seems to be more oriented
particularly appealing when a private company
to outgrower schemes such as the Bagamoyo
is identified from the start of the project and the
Sugar Infrastructure and Sustainable Community
value chain is a large infrastructure or processing
Development Programme in the United Republic
investment that only a private-sector entity can
of Tanzania.
undertake. In the intermediary-driven model,
the most common intermediary is usually an
NGO or a service provider that brings the parties
together, provides them with services (e.g.
Emerging opportunities in
capacity building of farmers, information and irrigation value chains
PART III / Chapter 10

sourcing to private companies, and networking

Global demand for food is expected to at least
between the parties), and brokers and monitors
double over the next 25 years. An emerging
value chain linkages. This model is useful
middle class, particularly in Asia, will also
when the value chain linkages are not very
demand more higher-value food products.
strong, the private‑sector presence is weak,
Therefore, smallholder irrigators will have multiple
and value chains are identified during project
opportunities to contribute to local and regional
implementation.
food security, and to manage environmental
There is no single right or optimal way to services within watersheds. These scenarios
organize a value chain. Models will vary will lead to more economic and financial
according to country, commodity, local context, opportunities for small irrigators.

113
 However, to date, investments in value-chain Strengthening IFAD’s
irrigated agriculture are severely lacking in
investments in irrigation
public smallholder irrigation schemes. Such
investments have the potential to drive economic value chains
growth, reduce poverty, and fulfil domestic and An analysis of IFAD’s work in irrigation value
global demand for agricultural products (Seville, chains has revealed the following shortcomings:
Buxton and Vorley, 2011). Linking smallholders • Few projects have significant investments
to markets and identifying which markets have
in agroprocessing, cold storage and
the potential to benefit certain producers are
transportation.
crucial steps towards improving the livelihoods
• Most projects appear to lack investments
of the rural poor. The small irrigator is at the
in marketing infrastructure and marketing
base of the economic pyramid, and the objective
information systems.
is to lift such smallholders from the bottom of
• There are few successful examples of the
the pyramid.
4Ps in current project execution.
A new development pathway for the design of • With the exception of the PASIDP II project,
smallholder irrigation projects is proposed in there is no strong attempt to link the irrigation
order to enhance crop water productivity and to water user associations to marketing
significantly augment the economic, social and organizations and value chains.
environmental benefits of irrigated agriculture. • Few projects address gender and inclusivity,
The emphasis should be not just on upstream and financial inclusivity, in the value chain.
irrigation hardware components but also on the
Based on these observations, IFAD can improve
value chain – and particularly on lengthening
its activities in irrigation value chains in several
the chain from irrigated production of high-value
ways, as outlined below.
commodities to local, national, regional and
international markets. Increase private-sector involvement.
Apart from the traders, wholesalers and
Stronger links between irrigators, farmers
outgrower schemes, there is little full
organizations and marketing systems could
private‑sector involvement in irrigation value
lead to a stronger asset base, new employment
chains. More private-sector involvement should
opportunities, and increased income generation
be encouraged given that, in conjunction with
in rural areas. Moreover, a more robust irrigation
the public sector, private companies can help
business enterprise model offers potential
cover capital investments and logistics along the
to leverage co-investments from the private
cold chain, including storage and transportation
sector, impact investors, venture capitalists,
infrastructure. Moreover, the private sector can
microcredit agencies, and emerging NGOs with
provide an integrated package of services,
a strong interest in agricultural value chains and
ranging from irrigation equipment and pumps to
financial inclusivity.
improved seed availability, agronomic advisory
This development pathway is rooted in the services, transport, grading and packaging,
inclusion of women and rural youth in irrigated storage, agroprocessing and, ultimately,
agriculture, and in the emergence of new high‑end markets. In addition, the private sector,
economic and employment opportunities for mainly large commercial firms and multinationals,
the people who depend on rural areas and has experience with contract farming, whereby
agriculture for their livelihoods. Building value farmers are provided with seed, inputs and
chains and markets will raise the asset base of technical advice to grow specific high-quality
women, indigenous groups and the rural poor. products (cocoa, coffee, horticulture and
Through new entrepreneurship programmes potatoes). Contract farming with private firms
around irrigation value chains, poorly educated also allows farmers to know in advance what
youth, who represent almost 40 per cent of the their prices will be, and may also include
agricultural workforce in rural areas, will be in a mechanisms to pay farmers a premium price for
better economic and financial position. top-quality products.

114
Expand the scope of the 4Ps. Given the direct communication and coordination along
high capital costs associated with irrigation, the supply chain. Although these requirements
transportation and storage infrastructure, the raise the barrier to entry for new producers,
4Ps model should be explored much more they also present potential opportunities for
significantly in irrigation value chains. It can be diversification, income generation and high-end
used to fund the high upfront investment costs value-added production. Poor households in
of agricultural value chains, cover the risks, rural areas can also participate in formal supply
provide a stronger regulatory and contractual chains as wage labourers in production, grading,
framework, and ensure fuller farmer participation. sorting, labelling, handling or processing, and
When farmers are involved structuring and as providers in the service markets that support
implementing the value chains, they could be value chains (Seville, Buxton and Vorley, 2011).
more fairly compensated.
Small and medium-sized companies have had
Enhance the intermediary-driven model. mixed success with smallholder value chains.
NGOs and companies are working to link small These companies are risk averse and need the
farmers to regional and global formal markets. financial backing of governments, which is not
An advantage of NGOs is that they put the always forthcoming. In many instances, they
farmer first, and they have the ability to raise are not able to pay the high costs of cold chain
funds from private investors, angel investors, logistics, and they depend on government to
venture capitalists, impact investors, financial pay for roads, electricity, transport and storage
intermediaries and microcredit agencies. Another facilities. In some cases, small farmers do not
emerging marketing mechanism consists of honour contracts with these firms if they can
commodity exchanges. Buyers employed by suddenly obtain a better price elsewhere. In
the commodity exchange and working on other cases, the firms do not honour the prices
commission can link to farmers in the field and to small farmers if they suddenly have a glut of a
offer prices based on commodity trading prices specific commodity.
in markets in other countries. The goal is to
have a pyramid whereby food producers are
aggregated upward in the pyramidal structure, A new irrigation business model
and the commodity exchange then purchases
the raw commodities. The exchange deals The proposed business model for new irrigation
with the aggregators at the top of the pyramid projects is tied to value chains, and it frames
rather than with hundreds of small farmers at smallholder irrigation in a more sustainable
the bottom of the purchasing pyramid. This financial, economic, environmental and social
significantly reduces transaction costs. context (figure 10.1). By incorporating aspects of
youth entrepreneurship, women’s empowerment
and gender inclusivity, the proposed business
model can create new economic opportunities,
Challenges to successful value jobs and wealth in rural areas.
chains
PART III / Chapter 10

In both smallholder irrigation projects and value

Efforts to establish successful value chains chain projects, IFAD has a unique opportunity
face several hurdles. One is the lack of strong to support women with access to training,
market intelligence systems in many developing productive assets (including rights to land and
countries. There are few institutionalized water) and financial tools so that they can
mechanisms for linking irrigated production to more easily access concessionary credit. Any
market demand, in terms of timely supplies of development of irrigation value chains should
product in the right volumes to specific markets. be firmly anchored in educating and promoting
Formal markets have requirements – such as women as business managers. This will require
quality, consistency, traceability, food safety and a suite of training programmes not just on
third-party certified standards – that necessitate the technical aspects but in areas such as

115
 purchasing, retailing, keeping financial records, practical training courses at high schools,
understanding balance sheets, business technical schools, and agricultural, technical and
development, financial planning, marketing, vocational education and training schools for
procurement, entrepreneurship and banking. disenfranchised youth. Training is also required in
post-harvest handling techniques, food handling,
More than 40 per cent of the population in rural
preparation and recipe creation to promote
regions of developing countries are under the
consumption of traditional vegetables across
age of 18, and a large number of these people
cultures. At training hubs, farmers learn how
are unemployed. Creating irrigation value chains to identify markets, grow vegetables that meet
will stimulate the need for technicians, truck quality attributes required by specific consumers,
drivers and handlers, mechanics, packers and work collaboratively with other key players in
and graders. IFAD is uniquely positioned to a given value chain. The outcomes will be more
implement a signature programme in youth opportunities for profitable self‑employment
empowerment and inclusivity in irrigation value in peri-urban and rural areas, and better
chains. Such a programme would require production and consumption of nutritious
investments in short-term, hands-on, applied, value‑added commodities.

Figure 10.1 Proposed irrigation business model

Government participation System benefits Private-sector participation

Establishment of land and Increased irrigation Increased technological and
water rights, institutions, productivity, water savings, economic productivity,
policies, regulations, new irrigation technologies, financial investments,
participatory irrigation production of high-value building sustainable irrigation
management, water user crops financing for operation
association, standards, and maintenance
funding

Improved infrastructure Irrigation value chains Improved inputs (e-wallets,

(Transport, electricity, roads, Create value chains for agroprocessing equipment
storage); strengthened farmer smallholders, new market and facilities), public-private
cooperatives, market access opportunities, improved partnerships, contract farming,
alliance, economic poles, logistic chains financial inclusivity, gender and
commodity markets, training youth inclusivity, innovative
programmes, farmer & field financial platforms
centres, farmer research
groups

Capacity-building, training and education

Increased technical know-how, innovation platform,

job and wealth creation, and entrepreneurship

Increased income, youth and gender empowerment, poverty reduction

Economic and environmental sustainable irrigation value streams

116
Innovation and undertake small cottage-level agroprocessing
– they will need training in food processing,
entrepreneurship
food safety and quality control. New microcredit
As part of the new business model for vehicles are required so that these skilled people
smallholder irrigation, there is an opportunity can take out small loans to purchase hand tools
to develop a unique programme on innovation and other items.
and entrepreneurship. Figure 10.2 shows a
Technological innovations will also be
series of activities within the innovation and
necessary. For example, high energy costs
entrepreneurship pathway that are necessary for
and scarce energy availability in rural villages
successful irrigation value chains.
are impediments to prolonged shelf life
A range of entrepreneurial capabilities will be and marketability of high-value, perishable
required. Skilled people will be needed as pump commodities. Solutions could include solar-
and irrigation system technicians, truck drivers powered irrigation pumps, solar-cooled storage
and handlers, mechanics, packers and graders. tents in the fields, and solar- or biofuel-powered
Local technical colleges and agricultural, transport systems. Another innovation is the use
technical and vocational education and training of e-wallets to procure credit and agricultural
schools can train unemployed youth in such inputs, and to sell commodities, thereby
skills. In addition, women and young people minimizing the handling of cash, and cutting
can be employed in packinghouses and can transaction costs.

Figure 10.2 The development pathway towards building irrigation value chains

Grading, Sale of fresh products

Harvest and
VALUE
CHAIN

Irrigated horticulture packaging, Transport to to consumers,

transport to
production labelling and economic centers agroprocessors
collection points
storage and retailers

Providing inputs, Logistics and Supply of crates, Logistics and Creation of commercial
ENTREPRENEURSHIP

irrigation pumps transport bags and labels; transport services distribution points for
AND INNOVATION

and equipment services; creating standards for sale of fresh and

collection depots grading and processed products;
sorting; low-cost creation of local cottage
storage points industries for light
agroprocessing;
creating markets and
attracting clients

Training and Training and Training in Training and Training and financing
financing financing grading, sorting, financing programmes for youth
INCLUSIVITY AND

programmes for programmes for packing and programmes for entrepreneurs in

SOCIAL IMPACT

youth in the drivers, containerized logistics services, marketing, sales, light

provision of input mechanics, storage drivers, agroprocessing, and
FINANCIAL

supplies, irrigation logistics agents mechanics distribution

equipment
PART III / Chapter 10

Reduced rural poverty and youth unemployment; higher economic returns from irrigation; new market
opportunities; wealth creation; educated youth workforce; new skills and jobs created; new educational
programmes for youth; opportunities for youth entrepreneurship and innovation; increased farm and rural incomes

117
 Recommendations to advance technical and institutional frameworks for
groundwater use in irrigation in order to
IFAD’s investment programme
protect the resource and the terrestrial
in smallholder irrigation and aquatic ecosystems that depend on
Based on the analyses conducted, meetings groundwater.
held with IFAD staff and leading international • Adapt to climate change, improve
irrigation experts, this chapter makes several water management in rainfed areas,
recommendations for advancing IFAD’s and adopt climate-smart practices.
investment programme in smallholder irrigation. Drought, water scarcity and desertification
will limit agricultural productivity. Therefore,
Irrigation water management investments in climate-smart agriculture
These recommendations pertain to all should reflect funding of water conservation
geographic regions and are of equal importance: and water reuse, and make use of new
• Harvest and use rainwater efficiently. sources of climate-resilient and green
More than 80 per cent of cultivated cropland corridor funds.
worldwide relies on rainwater. Therefore, • Establish an IFAD study group on water
water harvesting, water conservation, productivity. A study group on water
soil moisture management, water-use productivity should identify the exact causes
efficiency, dryland cropping and conservation of low irrigation efficiencies by region,
agriculture in the rainfed areas must continue and define a package of technological
to receive investments. improvements for enhancing efficiencies.
• Continue to emphasize the multiple uses In addition, the study group could use data
of water. In rural areas, irrigation is the from the Performance Management System
largest water user and is often the stimulus for various IFAD water/irrigation projects to
for providing water for other uses such as quantify both physical water productivity and
aquaculture, livestock, agroprocessing, water economic water productivity for selected
supply and sanitation. It would be good projects, and determine the scope for
to formalize the broader water allocation, possible productivity increases.
pricing and institutional framework for
multiple-use systems, and to ensure that Technological innovations
irrigation water supplies are not jeopardized.
These recommendations pertain to all regions
• Fit irrigation design and management into
and are prioritized as follows:
a holistic framework. Irrigation practices
must be in support of natural-resource High priority
management and integrated watershed
management, and more emphasis must be • Promote water-saving technologies.
placed on impacts of irrigation schemes on These technologies include, among others,
downstream water quality. This will mean improved on-farm water management,
working with environmental regulators and land levelling, and drip and microsprinkler
agencies to establish water-monitoring irrigation. They can be combined with
networks and to develop environmental conservation agriculture and climate-smart
management plans. agriculture to reduce greenhouse gas
• Play a stronger role in managing emissions and sequester soil carbon.
groundwater irrigation. Given concerns • Control soil salinity in arid and semi-arid
about the injudicious pumping of areas. Mitigation measures are required in
groundwater and the depletion of aquifers, order to prevent and control soil salinity in
IFAD should place greater emphasis on both irrigated and rainfed schemes.

118
 • Introduce irrigation scheduling • Place increasing emphasis on
techniques. Simple soil moisture sensors private‑sector involvement. Apart from
and weather stations installed in irrigation in sugar cane outgrower schemes, there
districts with support from WUAs can help appears to be limited larger private-sector
conserve water and improve water-use involvement in irrigation value chains.
efficiency. It will be increasingly important to engage
• Investigate the potential for water table larger private-sector interests, through
control/subirrigation. This technology is the buyer‑driven model or 4Ps, in order
especially suited to lowland and river valley to finance more costly infrastructure
irrigation developments, particularly when investments of storage, transport, shipping
adopted by several farmers in a drainage/ and logistics chains.
irrigation district. • Support the public-private-producer
partnership (4Ps) model. This model
Lower priority allows farmers (or their representatives)
to participate more in securing markets,
• Examine new models for allocating and
engaging in contract negotiations and
delivering water. This includes shifting from
obtaining fairer compensation.
a supply-driven irrigation system to one
• Use private-sector partnerships to
that is demand driven. To reduce current
increase investments in agroprocessing,
problems of over- and under-irrigation,
cold storage, marketing facilities and
existing irrigation systems will need to be
transportation. Existing projects invest
re-engineered as part of investments in
little in agroprocessing and other stages
irrigation modernization.
of the value chain. These activities add
• Use modern water assessment models.
value to perishable, high-value horticultural
Apply integrated decision-making platforms
commodities.
comprising climate forecasting models,
• Partner with the private sector to provide
geographic information systems (GIS), and
an integrated package of services.
hydrological and crop models to assess
These services range from selling irrigation
water availability and water allocations for
equipment and pumps to improved seed
irrigation project designs.
availability, to fertilizer and other chemicals,
• Apply new irrigation sensor technologies
to soil nutrient testing, to agronomic advisory
for precision irrigation management.
services, and to value chain logistics, such
This involves the use of automated weather
as transport, grading and packaging,
stations, soil sensors, and remote sensing
storage, agroprocessing and ultimately
of soil, climate and crop stress to schedule
high-end markets. This provides a one-stop
irrigation, thereby conserving water and service for irrigators.
improving water-use efficiency. • Build stronger marketing information
systems in value chains. There is a weak
Value chain development
link between market demand and farmer
PART III / Chapter 10

These recommendations pertain to all supplies. Most projects appear to lack strong
geographic regions and are of equal importance: marketing information systems and market
• Tie irrigation investments more closely intelligence, and this aspect should be
to value chains. Appropriate business strengthened.
models need to be developed, and each • Strengthen the linkage between the
project will need to be designed to fit the irrigation water user associations and
marketing situation, institutional and financial marketing organizations. This can be a
circumstances, and stakeholder interests. natural link to enhance the producer-driven

119
 value chain model, as the associations References
comprise the irrigation farmers, and they
can operate within a legal and administrative Amede, T. 2015. Technical and institutional
framework. attributes constraining the performance
• Introduce new service providers and of small‑scale irrigation in Ethiopia. Water
financial models. These include agroclimate Resources and Rural Development, 6: 78-91.
agencies, mobile phone companies, Dukhovny, V. 2011. Water and food security in
insurance companies, information Central Asia. In C. A. Madramootoo, ed. Water
technology companies, digital banking, management for global food security. Montreal,
e-wallets, venture capitalists, impact Canada: McGill Institute for Global Food Security.
investors and village-based microcredit
FAO (Food and Agriculture Organization of the
agencies, in order to support indigenous
United Nations). 2016. The State of Food and
entrepreneurial development of value chains.
Agriculture: Climate change, agriculture and food
• Build a new class of rural entrepreneurs,
security. Rome.
and promote stronger gender and
inclusivity in irrigation value chains. Gao, X., Shi, Y., Zhang, D. and Giorgi, F. 2012.
This would focus on capacity-building for Climate change in China in the 21st century as
women and young people through targeted simulated by a high-resolution regional climate
hands‑on, applied training programmes, model. Chinese Science Bulletin, 57(10):
thereby enabling them to create jobs, wealth 1188-1195.
and assets. Gunda, C., Gasasira, J., Sithole, S., Liversage,
• Implement a robust monitoring, H. and Jonckheere, S. 2013. Securing
evaluation and learning programme. smallholder farmers’ land and water rights and
This is necessary in order to verify the promoting equitable land access in irrigation
value chain approach and models, and use and watershed management in Malawi, Rwanda
lessons learned to make necessary course and Swaziland. Paper presented for the annual
corrections. World Bank Conference on Land and Poverty.
Washington, D.C.: World Bank.

Hanjra, M. A., Tadele, F. and Gutta, D. G.

2009. Reducing poverty in sub-Saharan Africa
through investments in water and other priorities.
Agricultural Water Management, 96(7):
1062-1070.

Heard, J., Porker, M., Armstrong, D., Finger,

L., Ho, C., Wales, W. and Malcolm, B. 2012.
The economics of subsurface drip irrigation on
perennial pastures and fodder production in
Australia. Agricultural Water Management,
111: 68-78.

Hussain, I. and Hanjra, M. 2004. Irrigation and

poverty alleviation: Review of the empirical
evidence. Irrigation and Drainage, 53(1): 1-15.

Madramootoo, C. A, Wiyo, K. A. and Enright,

P. 1992. Nutrient losses through tile drains
from two potato fields. Applied Engineering in
Agriculture, 8(5): 639–646.

120
Müller, C., Elliott, J., Chryssanthacopoulos, J.,
Deryng, D., Folberth, C., Pugh, T. and Schmid,
E. 2015. Implications of climate mitigation for
future agricultural production. Environmental
Research Letters, 10(12): Article 125004.

Seville, D., Buxton, A. and Vorley, W. 2011.

Under what conditions are value chains effective
tools for pro-poor development? Report for
the Ford Foundation. London: Sustainable
Food Laboratory and International Institute for
Environment and Development.

Sezen, S. M., Celikel, G., Yazar, A., Tekin, S. and

Kapur, B. 2010. Effect of irrigation management
on yield and quality of tomatoes grown in
different soilless media in a glasshouse. Scientific
Research and Essays, 5(1): 41-48.

Verge, X. P. C., De Kimpe, C. and Desjardins, R.

2007. Agricultural production, greenhouse gas
emissions and mitigation potential. Agricultural
and Forest Meteorology, 142(2): 255-269.

Walker, W. 1989. Guidelines for designing and

evaluating surface irrigation systems. FAO
Irrigation and Drainage Paper No. 45. Rome:
Food and Agriculture Organization of the United
Nations.

Wanvoeke, J., Venot, J. P., De Fraiture, C. and

Zwarteveen, M. 2015. Smallholder drip irrigation
in Burkina Faso: The role of development
brokers. Development Studies, 52(7):
1019-1033.

World Food Summit. 1996. Rome Declaration

on World Food Security and World Food Summit
Plan of Action. Rome: Food and Agriculture
Organization of the United Nations.

You, L., Ringler, C., Wood-Sichra, U., Robertson,

PART III / Chapter 10

R., Wood, S., Zhu, T., Nelson, G., Guo, Z. and

Sun, Y. 2010. What is the irrigation potential
for Africa? A combined biophysical and
socioeconomic approach. IFPRI Discussion
Paper No. 993. Washington, D.C.: International
Food Policy Research Institute, Environment and
Production Technology Division.

121
 Chapter 11

Assessment of IFAD’s policy

impact in four countries in Asia

by
Anna McCord
Edward Heinemann
Lauren Phillips

The possibility of achieving inclusive and improving the impact of IFAD’s projects as well.
sustainable rural transformation is determined Implementation of projects may be constrained
in large part by the policies that national by a mismatch between the project objectives
governments put in place and implement. and the policy framework, or a gap between
Thus, the policy framework can have a the policy and its implementation. Therefore,
dramatic impact – positive or negative – on the addressing policy bottlenecks or weaknesses
opportunities open to rural men and women. in policy implementation can help to create an
IFAD recognizes that it has a key role to play enabling environment for project implementation,
in shaping the policy framework in its Member thereby improving the chances that outcomes
States, drawing on its specialized expertise are realized, and development impacts achieved.
to promote enabling country-level policies for
IFAD has defined country-level policy
rural people. For this reason, beyond the more
engagement (CLPE) as a process in which the
traditional activities embedded in IFAD’s lending
Fund can collaborate, directly and indirectly, with
programme, an increasing number of loans have
partner governments and other country‑level
the specific objective of engaging in country-level
stakeholders to influence policy priorities or
policy, to enable poor rural men and women
the design, implementation and assessment of
and their organizations to participate in policy
formal policies that shape the opportunities for
processes, to stimulate the production and use
inclusive and sustainable rural transformation.
of evidence for policy processes, and to enhance
It is a broad definition that is rooted in an
the policy capacity of governments.
understanding of a “policy cycle” and one that
Today, it is widely understood that projects alone enables a relatively broad range of activities
cannot eradicate rural poverty or generate rural to be considered as contributing to a policy
transformation because, inevitably, they reach engagement agenda. IFAD’s approach is to
a limited number of people. However, projects facilitate, support and inform nationally owned
can serve as a space in which to innovate and policy processes so as to enable governments
experiment with new possible policy solutions. and other national stakeholders themselves
Moreover, by feeding the lessons learned back to to determine the policy change required. Its
local, national and international actors, projects approach is characterized by building national
can also be a lever for influencing public policies capacities for dialogue and the design,
and national-level programmes, thus bringing implementation and assessment of policies, and
about systemic change. Policy engagement by bringing evidence to those processes where
can serve the more immediate purpose of appropriate and useful. It includes, but goes

122
beyond, policy dialogue. IFAD does not impose of the intervention under review on national policy
policy change as a condition for its support for and formed the hypothesis to be tested during
investment projects, and rarely does it seek the second phase. It also identified potential key
specific policy change. informants from across a range of sectors who
were knowledgeable about the policy process
In 2016, as part of a broader series of activities
under review. The national expert then arranged
aimed at facilitating a consistent approach to
an interview schedule for the research visits.
monitoring and evaluating its CLPE efforts, IFAD
commissioned a study to assess the impact of Second, once this preparatory phase had been
its policy engagement on policy change in four completed, the review team (the international
countries in the Asia and the Pacific region. consultant and the national expert) made
This chapter is based on the findings of this five‑day research visits to each country. In three
review (McCord, Heinemann and Phillips, 2018). of the four countries, the research took place
The study had three objectives: exclusively in the capital, although in India
• Review the experience of sampled IFAD both the capital and the state were visited.
CLPE activities associated with five The research visits started with an orientation
IFAD‑supported projects in four countries session with IFAD staff that covered the national
(India, Indonesia, Nepal and Viet Nam), and programme, the programme associated with
analyse their outcomes and impact. the CLPE, and the CLPE activity itself, including
• Identify the factors that contributed to these the IFAD contribution narrative under review.
outcomes and those that limited success, A bellwether approach was adopted, using
drawing out lessons for future policy work. semi-structured interviews to explore the CLPE
• Propose a credible, cost-effective and intervention and the policy-related change under
replicable methodology that can be used to review. This entailed interviewing 15-30 of the
undertake similar exercises. previously identified informants in each country,
drawing on civil servants, politicians, civil society
representatives, international development
partners, academics and media actors.
Methodology
In cases where IFAD was mentioned as a
The study was implemented by a two-person
driver or contributor, the research team sought
team comprising an international evaluation
evidence in support of this assertion. When it
and policy specialist, and a national expert in
was not mentioned as such, they probed the
the area of agricultural policy under review. The
context and sought evidence to identify and
case study countries and CLPE interventions for
assess the relative importance of other drivers,
review were selected to focus on activities linked
as well as asked questions in order to assess
to policy development, policy outcomes or policy
the informant’s knowledge and perception of
implementation, in line with the IFAD definition of
IFAD. To elicit further insights, open-ended
CLPE, as part of a narrative that identifies IFAD
questions regarding IFAD’s role in relation to
as a contributing agent in that change.
the broader policy debate and its potential
PART III / Chapter 11

A methodology was devised to enable a rapid contribution were also asked. This approach
review of a single CLPE case study in each of to identify the main drivers of a change in
the four countries. This entailed a two-phase discourse or policy did not in all cases identify
process. First, in each country, the national IFAD as a factor, and thus not all cases were
expert prepared a background report providing consistent with the IFAD narrative. The review
summary information on IFAD programming and team reviewed findings after each interview and
the case study CLPE intervention, with guidance analysed the set of responses overall in order
from the IFAD country office. This report included to appraise CLPE performance in relation to
a statement of the IFAD contribution narrative, the IFAD change narrative, and also to explore
which articulated IFAD’s perception of the impact the specific questions set out in the terms of

123
 reference. The findings were validated with IFAD and national policy processes, and promoting
country staff at the end of the research period policymaker capacity and programme innovation.
where possible. A final review team workshop
Over a sustained period covering two projects,
was held to finalize the findings, which were then
IFAD made the case for pro-poor leasehold
shared with the IFAD country staff in draft format
forestry. It identified a programming and policy
for verification.
gap, and financed a pilot and a follow-up project
The methodology adopted could not provide any to demonstrate the potential role of leasehold
formal scoring of CLPE performance to allow forestry in rehabilitating land and creating
for geographical or intertemporal comparisons. institutions to represent the most marginalized.
However, it did provide insights and draw out The projects’ most innovative feature was the
nationally and internationally relevant themes that handing over of degraded forestland to the
could not be extracted using more quantitative poor, together with the introduction of livelihood
or formal approaches. The approach also had improvement planning. These steps helped
the advantage that it could also be applied raise people’s incomes through commercial
consistently, despite the extreme differences in
agricultural and livestock farming. Despite
the case studies under review. Finally, because
the 1996-2006 insurgency, which severely
the methodology relied on interpretation and
hindered programme implementation and policy
synthesis by the authors and was influenced by
engagement, the successive leasehold forestry
the selection of informants, there was a degree
projects supported by IFAD demonstrated
of subjectivity to the findings. However, extensive
the leasehold forestry concept, and showed
discussion and triangulation were used to
that transferring land with degraded forests
minimize this challenge.
to the poor could both reduce poverty and
reforest the hills. The projects also showed
that this approach could empower women and
Nepal: Transforming an reduce the time they spent collecting fuelwood
IFAD‑financed pilot into and fodder.
a national programme for The two leasehold forestry programmes also
leasehold forestry created platforms for dialogue at local and
national levels, as well as institutions to facilitate
The IFAD-supported projects associated with
poor people’s representation and participation
the policy engagement experience under review
in forestry dialogue. These institutions included
were the Hills Leasehold Forestry and Forage
local farmer-based institutions, notably leasehold
Development Project (1992-2002) and its
forestry user groups and cooperatives, and
successor, the Leasehold Forestry and Livestock
village financial institutions, which were
Programme (2005-2014). IFAD introduced
these projects to address the lack of a pro-poor subsequently used by other government and
dimension in the community forestry approach development partner projects to replicate
prevalent in Nepal. The initial Hills Leasehold and adapt the leasehold forestry models.
Forestry and Forage Development Project The projects had a significant effect on land
programme did not have a policy engagement rehabilitation and poverty. They addressed
agenda. However, CLPE relating to leasehold the pro-poor deficit in the community forestry
forestry is consistent with the strategic goal of discourse dominant in the 1980s and 1990s, and
the 2013-18 Country Strategic Opportunities the fact that within this dominant approach “the
Programme (COSOP), which explicitly includes voice of the poor was not heard” (former district
policy engagement objectives relating to the forestry officer). IFAD support for CLPE relating
programme in Nepal overall and to leasehold to leasehold forestry ceased after programme
forestry in particular. These policy objectives funding came to an end in 2014, despite the
include incorporating farmer participation in local priority status it was given in the COSOP.

124
IFAD played a catalytic role in promoting was recognized as a successful model. As one
leasehold forestry within Nepal. It established the key informant stated, before IFAD “there was
concept of pro-poor leasehold forestry, which nothing in the policy [about leasehold forestry],
combined agriculture, forestry and livestock. The we did not know how to do it – we were starting
projects developed a model that later influenced from zero” (former district forestry officer).
legislation and government structure. Before the Achieving these outcomes is particularly
intervention, the concept of leasehold forestry significant given that IFAD is not a major player
was not widely accepted, and was poorly in Nepal’s donor context in terms of the size of
addressed in national forestry sector policy its programme or its country representation.
and legal instruments. A number of legal and It had not participated in the development of
technical hurdles delayed the process (including community forestry programming in Nepal, but
a lengthy process of handover of leasehold joined the forestry debate later, championing
forestry to centralized authorities at the Ministry the concept of pro-poor leasehold forestry.
of Forests and Soil Conservation, and the Moreover, the leasehold forestry model initially
long-standing prioritization of community over met strong opposition from the forestry DPs,
leasehold forestry). However, the amendment which were committed to promoting community
of the Forests Act (1992) in early 2017 finally forestry. The DPs and the government accepted
resolved the legal barriers restraining leasehold leasehold forestry only after a decade of
forestry implementation. Under the amendment, successful project implementation. Thus, IFAD
leasehold forestry has technical parity with was obliged to work alone in the initial decade of
community forestry, the legal authority to issue this initiative.
lease certificates has been decentralized,
and provisions integrating leasehold forestry
into community forestry have been enacted, India: Approaches to
with leasehold forestry formally identified as
a mechanism for implementing the national
community-based resource
community-managed forestry system. As such, management for scaling up
leasehold forestry has been institutionalized in The North Eastern Region Community Resource
the forestry sector, and the National Planning Management Project for Upland Areas
Commission has officially recognized the role (NERCORMP) – a government project that was
of leasehold forestry by giving it Priority 1 supported by IFAD – aims to transform the
programme status, which should mean that lives of poor and marginalized tribal families in
it will be mainstreamed by donors working selected districts of three states in the North
in interventions related to forestry and Eastern Region of India. It seeks to do so
poverty alleviation. through improved and environmentally sound
In addition, the leasehold forestry concept management of their resource base. CLPE is not
has been widely replicated in other parts of formally mentioned in the NERCORMP project
the country. It has influenced the design of design report or the associated logframes, and
PART III / Chapter 11

subsequent IFAD-financed projects and informed neither phase had explicit CLPE objectives or
the leasehold forestry components of a variety activities. In addition, the 2010-2015 COSOP
of interventions by other development partners contained no CLPE objectives relating to the
(DPs). Support for leasehold forestry by the DP NERCORMP intervention. However, the project
community largely occurred after 2007, when offers some successes and lessons on CLPE.

Australian support to the forestry sector came to IFAD had limited contact with government
an end. Before this, the position of most donors actors both nationally and regionally relating to
was in conflict with IFAD’s leasehold forestry NERCORMP. During annual loan reviews, there
position, and it was only after a ten-year struggle was limited space for CLPE with Delhi-based
with the donor community that leasehold forestry policymakers, given that NERCORMP was

125
 only one of multiple projects under discussion. development interventions to increase
Annual IFAD supervision missions to the North impact and promote efficiencies in delivery,
Eastern Region provided a greater opportunity an agenda that is central to the policy
for policy-related contact, through government debate in the North Eastern Region. The
officials’ participation in the missions and the IFAD innovations contributed directly to
debriefing process to the North Eastern Council, convergence by harmonizing programme
but neither was strategically used to promote delivery for key community institutions and
CLPE. As such, IFAD was perceived by those processes. One example is the adoption
involved as not advocating for the project, of NERCORMP-created institutions and
resulting in missed opportunities to celebrate processes by the national employment
achievements and share innovations arising from guarantee scheme.
the project more widely within the North Eastern • Promoting government buy-in, scaling up
Region and beyond in other areas of the country and sustainability: The central government
where conventional project implementation funded the NERCORMP Secretariat directly
modalities were not viable. IFAD instead during the unfunded period between the
allowed NERCORMP successes to “speak for first and second phases. It did so in order
themselves”, an approach that was consistent to prevent staff attrition and protect the
with the COSOP statement that advocacy was investment made previously. The government
not part of IFAD’s role. This stance also reflected used the second loan to expand the
the fact that NERCORMP is a government geographical coverage of NERCORMP, while
programme, from which the government a third phase exclusively funded by the North
itself learns directly, arguably rendering IFAD’s Eastern Council is currently being initiated,
engagement in policy dialogue redundant. indicating programme sustainability.
Nonetheless, the outcomes of the NERCORMP • Creating technical assistance capacity:
intervention, in terms of policy impact in its In recent years, the NERCORMP Secretariat
broadest sense, were fourfold: has been taking on a role as a regional
• Contributing new modalities for resource, providing technical assistance
programme implementation: NERCORMP for project implementation. Formalization
had multiple influences in terms of community of this role is currently under discussion.
mobilization and participation processes, The NERCORMP Secretariat is expected
NGO participation, and the creation of to become a government-financed regional
community institutions. Subsequent resource centre to provide technical
IFAD‑supported projects and other DP assistance for programme management
and government programmes adopted or and implementation, thereby capturing
replicated these approaches, as did national‑ NERCORMP implementation experience as a
and state-level government programmes resource for the region.
implemented in the region. The project Thus, although CLPE was not part of the
also mainstreamed women’s participation NERCORMP concept, the project successfully
in community decision-making, promoted devised new ways of delivering “development” in
natural resource management and community a resource-poor and remote region. Because its
ownership of a livelihoods agenda, and success was recognized by key officials, it was
engaged NGOs in offering alternative replicated and expanded, and other states and
institutional options for programme delivery donor programmes adopted key programme
in addition to government departments. All components. NERCORMP delivery models were
these ideas were subsequently adopted in also used in other government programmes.
government programming in the region. In the absence of a policy agenda, it was the
• Promoting convergence: Convergence spontaneous transmission of knowledge about
involves harmonizing diverse government the programme, derived from programme

126
experience (the programme “speaking for itself”) community development and trained facilitators
that led to these positive outcomes, and IFAD drawn from the local community. As of 2016,
presents this as an indicator of successful CLPE. the objectives of the CLPE were to successfully
However, if IFAD had had a policy engagement implement the second phase of the agriculture
agenda at the design stage and had actively component of the PNPM in Papua and West
mandated IFAD personnel to engage in CLPE in Papua, and to promote the nationwide adoption
support of the programme, it is probable that the of innovations to help villages select appropriate
impact could have been greater in terms of policy economic infrastructure and ensure the provision
influence, with impacts beyond the three states in of technical support, resulting in improved
which the programme was implemented. livelihoods. These are the aspirations against
which future achievements should be appraised.

IFAD was able to be opportunistic, capitalize on

Indonesia: Informing the successful institutional repositioning, and engage
government’s approach to in responsive ad hoc CLPE. It built on the
success of a programme that largely “spoke for
implementing the Village Law
itself”, and drew the government’s attention as a
From 2009 to 2014, IFAD piloted and cofinanced potential model for Village Law implementation.
the agriculture component of the National It is premature to assess the sustainability of
Programme for Community Empowerment the outcome of the CLPE initiative, as PNPM
(PNPM) in Papua and West Papua. The implementation in Papua and West Papua has
objective of the larger PNPM was to reduce only recently recommenced. Whether IFAD can
poverty and improve local-level governance and capitalize on the invitation to influence national
socio-economic conditions in rural Indonesia. implementation will depend on its capacity
The agriculture component aimed to improve to mobilize appropriate technical assistance.
agricultural livelihoods by developing new models However, the invitation to provide inputs, which
of participatory community participation in brought an IFAD pilot to the national agenda,
development planning and resource allocation, may itself be taken as a CLPE success. Whether
and in this way influence PNPM implementation. the IFAD-supported agriculture PNPM will
have broader influence and affect Village Law
Following elections in 2014, the new government
outcomes in the medium or long term remains
suspended the project, along with all similar
to be seen. Equally, it is not known whether the
programmes, as it sought to move away
government will continue to fund the Village Law
from the PNPM. In its place, the government
investment in infrastructure, which is at the heart
focused on implementing the Village Law, a
of the IFAD model beyond elections in 2019.
national policy that promotes the autonomy of
villages and serves as a framework for village This case study offers three main insights: (1)
development and community empowerment. evidence of programme outcomes can be less
Villages are allocated resources, which they important in driving CLPE than institutional
manage through participatory community relationships, project exposure and perceptions
PART III / Chapter 11

development plans. In 2016, the government of programme performance; (2) effective

invited the agriculture component to restart processes for engagement can be as important
programming in Papua and West Papua, and to for success as final programme outcomes; and
advise on modalities for improving participatory (3) access to financial and human resources and
planning in the national implementation of the programming flexibility are key to responsive
Village Law. This represented an opportunity programming and opportunistic engagement.
for the component to draw on its initial These results reflect the non-linear relationship
implementation experience to create a model for between policy and evidence that has been
livelihood activities related to infrastructure and documented in Indonesia and elsewhere (see
related economic activities, using participatory McCord, Yablonski and Winder Rossi, 2016).

127
 Viet Nam: Supporting the community-level planning and decentralization
in place of the top-down socio-economic
National Targeted Program on
development planning process from the district
New Rural Development to the commune level. The IFAD-supported
(NTP-NRD) MOP-SEDP was put forward as a model of
participatory commune-level planning that could
IFAD supports the market-oriented participatory
be adopted.
socio-economic development planning
(MOP‑SEDP) component of Viet Nam’s National The recommendations from the IFAD/WB
Target Program on New Rural Development assessment were presented in March 2016 at a
(NTP-NRD). The NTP-NRD is a national workshop co-hosted by IFAD and the National
programme implemented in 9,071 communes Office of the NTP-NRD. Workshop participants,
to consolidate the multiple social, economic, including key representatives from government
cultural and political interventions taking place and DPs, agreed to the recommendations, and
there. The New Rural Development (NRD) the government charged the National Office
approach seeks to promote community of the NTP-NRD to use those findings and
ownership of key development processes at the recommendations as inputs into the design
commune level alongside government policy of the second phase of the NRD. Many of the
orientation and support. recommendations were subsequently endorsed
in the Prime Minister’s Decision 1600/QD-TTg
In 2013, in preparation for the second phase
of the NTP-NRD (2016-2020), the government – notably, the implementation of agricultural
approached key DPs, including IFAD, seeking restructuring to develop value chains for high
support and financial assistance for expanding added value and sustainable development, and
the NRD rollout. In response, the DPs proposed the shift from 19 fixed performance criteria to a
carrying out a review of NTP-NRD performance smaller number to be applied flexibly depending
to date before making financial commitments. on local context. To advance the assessment
The partners’ objective in this CLPE intervention report’s recommendations on strengthening NRD
was thus to contribute to the formulation of the monitoring and evaluation in phase two, IFAD
second phase of the NTP-NRD, revising the provided a grant for technical assistance to the
conceptualization and design to accommodate NRD National Office, starting in December 2016.
a reduction in centralized control, with more In February 2017, the Ministry of Planning and
market and community engagement and Investment issued the Draft Circular on Manual for
context-specific programming. Participatory Planning at Commune Level under
the Implementation of NTP-NRD, using evidence
In late 2015, IFAD worked with the World
from the MOP-SEDP along with other DP
Bank to conduct a detailed assessment of the
interventions. The intervention served to further
NTP-NRD strategy, reviewing its performance
consolidate IFAD’s relationship with the national
and proposing recommendations for moving
NRD implementing agency, and to position
forward – drawn in part from IFAD programme
IFAD for further engagement in developing
experience. The assessment confirmed
future monitoring and evaluation aspects of the
the NRD’s contribution to socio-economic
NTP‑NRD. These are results that in themselves
development in rural Viet Nam in terms of rural
may be perceived as positive CLPE outcomes.
infrastructure and services, but concluded that
it had done little to promote rural economic In terms of the overall impact of the CLPE activity,
transformation. For the second phase of the a view outside IFAD is that the changes outlined in
NRD, it identified the need to invest in planning, the Prime Minister’s Decision 1600/QD-TTg would
investment modalities and monitoring, and have occurred regardless of the intervention
it recommended using more flexible and because the government already recognized the
context-specific performance criteria to assess need for policy revision. This may be true, but it is
performance. It also called for a greater role for also the case that the evidence provided through

128
the study facilitated this process. It is also possible Conclusions and way forward
that the intervention could have had a greater
impact if it had extended the existing discourse The variety of outcomes highlighted in the four
into new areas and provided an agenda for more case studies illustrates the different functions
progressive programming and policy change, of CLPE. On the one hand, CLPE can create
based on a longer-term engagement in the policy opportunities for increased IFAD engagement,
debate and deeper engagement in the political in which case it is essentially instrumental – it
positions the agency for future engagement,
discourse. However, the political feasibility of this
resulting in requests for technical assistance
approach is open to question.
(as in Indonesia and Viet Nam). On the other
Informants perceived several untapped hand, CLPE can lead governments and DPs
opportunities for constructive CLPE, such to adopt innovative IFAD-financed policy or
as greater sectoral collaboration by donors, implementation models (as in India and Nepal).
potentially under IFAD leadership. Informants It is useful to divide the activities reviewed in this
also noted that IFAD could also play a larger report into two distinct CLPE approaches. The
role in central-level policy discussions related first is direct and active engagement by IFAD
to promoting smallholder agriculture within the staff, carried out to complement programme
NTP-NRD, drawing on areas of IFAD expertise activity. The second is indirect CLPE, where
and experience. This is a critical area in which programmes “speak for themselves” by
few DPs with access to significant resources demonstrating the success of implementation
are working and where informants identified modalities or policy innovation (which may
important gaps. Closing these gaps would involve be described as indirect CLPE). The studies
providing ongoing support to the working group reviewed here include both approaches,
on smallholder farming that IFAD had initiated, although the majority of CLPE resulted from
participating regularly in and supporting the donor programme performance, and only a small
policy forum, and creating strategic initiatives number of CLPE activities were directly carried
out by IFAD staff.
among like-minded agencies.
Analysis of the four case study interventions
Overall, this review identifies CLPE in Viet Nam
identified a number of common factors
as an appropriate investment for IFAD at this
affecting CLPE performance. Above all, there
juncture. Government policies are in flux, and
are clear tensions between IFAD’s in-country
the country is exploring market mechanisms.
staffing levels, financing modalities, operational
However, there is a lack of investment, technology
instruments and historic mandate on the one
and markets, and there is a need to support
hand, and increased CLPE activity, including the
the use of markets by farmers. As Viet Nam
facilitation of domestic national policy discourse,
undergoes significant economic change,
on the other. In the case study countries, IFAD
becoming an increasingly important exporter,
does not have the resource base needed to
there is potential for IFAD to support Viet Nam
participate in ongoing policy discourses or
given its comparative advantage in the area of
PART III / Chapter 11

to carry out significant direct CLPE requiring

smallholder farming. Viet Nam is currently open IFAD staff engagement. As a result, IFAD has
to the provision of counterparts, consultants focused on indirect policy engagement related
and policy inputs in support of decentralization to its implementation models, drawing on
and market-based development, and support IFAD’s specific institutional competence and a
in this area may be more relevant now than key aspect of IFAD programming. Elsewhere,
project‑based support. Referring to the current this approach might not be considered policy
need for institutional reform at the grass-roots engagement, given that it is not directly linked
level and for policy change at the central level, to either engagement or policy. It might more
one informant stated: “Now is not time for projects typically be considered good development work
but for policy.” and provision of good models for development

129
 processes, which are recognized and then References
replicated and scaled up by governments and
DPs, and in other IFAD programmes. McCord, A., Yablonski, J. and Winder Rossi, N.
2016. The political economy of cash transfer
Despite these constraints, some direct CLPE did
evaluations. In B. Davis, S. Handa, N. Hypher,
take place. In Viet Nam, the country programme
N. Winder Rossi, P. Winters and J. Yablonski,
manager (CPM) and a consultant reallocated
eds., From evidence to action: The story of cash
staff time and used additional funds to overcome
transfers and impact evaluation in Sub Saharan
the resource constraints outlined above. They
Africa. Oxford: Oxford University Press.
directly engaged in CLPE, and their focus on
general institutional repositioning using existing McCord A., Heinemann E. and Phillips L. (2018).
CPM resources encouraged the government Exploration of a methodology for assessing the
invitation to extend programming. Overall, impact of policy engagement: what impact and
however, IFAD’s institutional structure limited the how to assess it? IFAD Research Series 26.
potential for active, explicit policy engagement in Rome: IFAD.
each of the case studies.

If IFAD seeks to play an increased role in

CLPE, in line with an extended mandate that
incorporates policy engagement alongside
programme implementation, it will need to: make
changes to its resource allocation modalities and
country-level programming practices; increase
both its financial and human resources at the
country level; and provide additional support
from IFAD headquarters in Rome. IFAD policy
engagement is unlikely to yield significant results
unless country offices, which are currently
engaged primarily in managing loan and project
portfolios, are provided with these additional
resources and skills. As such, increased CLPE
implies an associated realignment of human,
financial and procedural resources within the
institution. The key question is the extent to
which this can be accommodated within IFAD’s
existing business model and funding modality.

130
Chapter 12

The impact of adoption of

CGIAR’s improved varieties on
poverty: a systematic review
by
Alessandra Garbero
Pierre Marion

It is widely acknowledged that agricultural possible causal impacts of technology adoption

research has played a key role in improving rural on poverty reduction. To date, only two reviews
livelihoods in the developing world (Thirtle, Lin – Fan et al. (2007) and Alene et al. (2009)
and Piesse, 2003). Donors and governments – have sought to quantify impacts on poverty,
have invested heavily in agricultural research and specifically movements out of poverty.
institutions, particularly the Consultative Group The first study examined the macroevidence of
on International Research (CGIAR) and its the impact of modern rice varieties developed
network of research centres and partners. by the International Rice Research Institute on
Specifically, IFAD funded many CGIAR poverty in China and India. The study found that,
improved‑seed interventions from 2007 to 2014 between 1981 and 1999, the institute’s research
through its grant-funding programme. However, contributed to moving more than 6.75 million
at a time when some donors are increasing Chinese out of poverty. In India, 14 million people
pressure on development institutions to show exited poverty between 1991 and 1999. Alene
evidence-based results, the causal linkages et al. (2009) estimated that improved maize
between agricultural research, poverty reduction, seed adoption in West and Central Africa moved
and welfare, and other development outcomes 740,000 people out of poverty annually, with the
more broadly, have not been sufficiently and rate of exit increasing over time.
convincingly demonstrated (Renkow and
Relative to global assessments, CGIAR
Byerlee, 2010). This has cast doubt on the
system‑level assessments date back to
effectiveness of CGIAR’s investments and those
Anderson (1985). Nelson and Maredia (1999),
of its donors.
Evenson and Gollin (2003), Maredia and Raitzer
A large literature has documented substantial (2006), and Raitzer and Kelley (2008) have
PART III / Chapter 12

pro-poor impacts resulting from international validated the perception that the CGIAR has
agricultural research and development generally, had, over its lifetime, a sustainable impact on
and from CGIAR-led research in particular poor people by helping to develop technologies
(Thirtle, Lin and Piesse, 2003; Adato and and agricultural management tools that have
Meinzen-Dick, 2007). However, this “impact” increased food security and dramatically lowered
literature has focused only on aggregate returns the cost of producing the world’s major staple
and on the efficiency of research investments, food crops. However, this perception has not
neglecting a thorough examination of the been supported by a rigorous appraisal of the

131
 “counterfactual”-based evidence available to This systematic review appraises and
date – that is, rigorous impact evaluations from quantitatively synthesizes the findings of studies
the CGIAR centres. assessing the direct impacts of improved seed
varieties on the welfare and poverty of adopting
This study aims to fill this gap. To this end,
households. While these direct impacts can be
a systematic review was conducted and
rigorously measured, evaluating indirect impacts
a quantitative synthesis was performed to
is more challenging because they affect both
assess the poverty and welfare impacts
adopting and non-adopting households, and
of the adoption of improved seed varieties
they depend on local market conditions.
developed and disseminated by CGIAR from
2007 to 2015. The study used meta-analysis In terms of direct effects, improved varieties
to compute the global estimate of the impact have characteristics that can lead to greater
of agricultural research on poverty reduction agricultural production, on average, than
and intermediate outcomes such as income traditional seeds. The improved seeds have
and expenditure. A meta‑analysis is a powerful higher yield potential, are more responsive to
technique that allows one to summarize the fertilizer and irrigation, have shorter maturation
quantitative evidence on intervention effects periods, have longer storage capabilities, are
from different environments in a comprehensive more tolerant of environmental stresses, or have
and unbiased way. Compared with traditional a higher nutrient content.
ways of aggregating research results, such as For households that produce and sell on the
vote counting, which do not remove the risk of local market, higher agricultural yields at a
purposively selecting studies based on subjective constant cost level may be associated with
criteria, meta-analyses are scientific. They greater income and greater profits. Similarly,
take into account precision, sample size, the for subsistence households – which purchase
magnitude of the effect, and the research design. the food they need but cannot produce
The rationale for this study of CGIAR – higher yields benefit farmers by reducing their
interventions is IFAD’s large investments in spending on food. Farmers may thus reallocate
CGIAR’s improved seed variety interventions expenditures away from food to assets, which
may further increase productivity. With higher
over the period. Out of the 95 grants provided
production, farmers can also enter local markets
to CGIAR (equivalent to US$96.4 million), 30
to sell excess production (Irz et al., 2001; de
(amounting to US$40.5 million, or 42 per cent
Janvry and Sadoulet, 2002). Thus, improved
of the total amount provided to CGIAR in
varieties can reduce poverty through higher
grants) were allocated to improved seed
yields for adopting households if the income gain
variety interventions.
is large enough to allow them to exit poverty.

However, farmers’ access to markets may be

Conceptual framework hindered by a rise in costs associated with higher
production, such as storage and transportation
When farmers adopt improved varieties, it costs. Moreover, as improved varieties are
can lead to poverty reduction through several disseminated, prices and costs in the food
mechanisms, both direct and indirect. Improved market may change, leading to indirect impacts
seed varieties can generate gains in crop in other markets. With higher potential income
productivity, raising farmers’ incomes and gains, large landowners may have the incentive
subsequently reducing poverty (direct impacts). to increase rents or to expand their cropped
Improved varieties can also benefit both adopting area by cultivating land previously rented out,
and non-adopting households by increasing thereby eliminating income gains for tenants
employment opportunities and wages, and by and increasing landlessness (Hazell and Haddad
lowering food prices as a result of the rise in 2001). Increased income may not translate into
agricultural production (indirect impacts). poverty reduction if, as often happens, credit

132
is required to purchase improved varieties. impact of adoption of modern seed varieties
This credit may impose a significant financial on welfare outcomes. These modern seed
burden on poor households, and in the case of variety interventions had to be supported by
a negative shock, debt repayments may instead CGIAR institutes.
increase poverty (Diagne and Zeller, 2001). • Comparison: The impact needed to be
Numerous external factors, such as political evaluated through real or reconstructed
instability or extreme climate shocks, may control groups (the latter correspond to the
also eliminate the productivity improvements. sampled households that did not adopt
Moreover, households may lack the education improved varieties).
and experience to adequately cultivate these • Outcomes: Outcomes were poverty impacts
improved varieties. and/or estimates on intermediate monetary
development outcomes, such as income and
expenditure for adopting households.
Methodology • Study design: The studies included used
either a “microlevel” experimental study
This study methodology begins with a systematic
design (whereby improved varieties were
review of studies that measured the poverty
randomly assigned to potential beneficiaries)
and welfare impacts of adoption of improved
or a quasi-experimental study design
varieties developed and disseminated by
(whereby modern varieties were not
CGIAR in the period 2007-2015, and it then
randomly assigned but groups of adopting
employs a meta‑analysis. In contrast with a
and non‑adopting households could be
standard literature review, a systematic review
is a clear protocol for systematically searching identified). Such study designs reduce the

defined databases over a defined period, with risk of biased results.

transparent criteria for including or excluding The study search was conducted on the largest
studies, and with an analysis and reporting of relevant databases (Science Direct, Google
study findings (Hedges and Cooper, 1994). Scholar, the International Initiative for Impact
The three steps required to perform a rigorous Evaluation’s impact evaluation database, the
systematic review are described below. CGIAR Standing Panel on Impact Assessment
publications, and the CGIAR Library). This search
Step 1: Study selection
strategy followed the methodology indicated
First, a search for all relevant studies was by Waddington et al. (2014) and Stewart et al.
conducted, following a strict protocol and (2015). For each of the above criteria, search
search strategy. The selection of studies was terms were chosen. Each search term from
governed by the so-called PICOS study selection the “interventions” criterion was inputted in the
criteria (participants, interventions, comparison, database search and combined with a search
outcomes and study design). Studies were term of one or more criteria. For all databases,
included in the review when they satisfied the the search was repeated in order to combine
PART III / Chapter 12

following criteria: all or most of the search terms from each

• Participants: Participants were small criterion. The bibliographies of included studies
farmers growing food crops and living in any and of existing systematic reviews on related
country except in high-income economies. topics were also screened for completeness.
They owned less than 2 hectares of land, Finally, key researchers from the eligible studies
and agriculture was their primary source of and colleagues working on impact evaluation
income. They also allocated a significant initiatives were contacted for additional
proportion of their land to the growth of published and unpublished studies. Two analysts
food crops. independently performed this search. A third
• Interventions/exposure: The studies analyst repeated this search to ensure that all
included were those that examined the relevant studies were included.

133
 Step 2: Critical appraisal of selected studies This methodology also guided the selection of
the estimates when multiple studies existed (for
The second step entailed appraising the
quality of each individual estimate presented example, a working paper and a journal article
in each study. The framework summarized covering the same study by the same authors) or
in Waddington et al. (2014) was used to when multiple studies were based on the same
structure the analysis. Studies were evaluated dataset. However, other econometric methods
based on the probable risks of bias (internal were more difficult to rank. Thus, a “synthetic
validity or causal identification) and external effect”22 was computed when the choice of the
validity (generalizability of the study’s results). best estimate was ambiguous. This synthetic
Each paper was assigned a bias score, a effect is based on the sample weighted average,
generalizability score and a total score (the sum using the procedure described in Borenstein et
of the two previous scores). This step was highly al. (2009), which calculates the variance and the
important as it assessed the reliability of the standard error of the estimate (these are both
studies’ estimates and the extent to which the measures of the spread in the magnitude of the
estimated impact of improved seed varieties
outcome variable across surveyed households
could constitute a reliable measure of the actual
in each study). For papers that reported results
causal impact.
over several years of follow-up, the results were
Step 3: Meta-analysis reported for the most recent years. In papers
where subgroup analysis was conducted,
To perform a meta-analysis, the research team
estimates were combined into a single number
computed standardized estimates across all
as in Waddington et al. (2014).
included studies. The final estimates were thus
expressed in terms of effect sizes. Response The response ratios from each study can
ratios were chosen as the appropriate metric of therefore be combined using two meta-analytic
effect size. The response ratio is defined as the methods: (1) the fixed-effect model (where
ratio of the mean outcome for the households studies are weighted in the meta-analysis only
adopting the improved varieties divided by according to the amount of information they
the mean outcome for the households not contain); and (2) the random-effects model
adopting the improved varieties. It has the same (where an estimate of between-study variation
interpretation as a risk ratio: 1 is the point of
is incorporated in the weighting). The choice
“no effect,” and any movement above or below
of model ultimately affects the distribution of
the “no effect” point represents a percentage
weights given to studies in the meta-analysis.
change in the adopting households’ outcome
In the fixed-effect model, the main assumption
compared with the non-adopting households.
is that the true effect is the same across studies.
For example, a response ratio of 1.3 translates
In a random-effects model, the true effect is
into a 30 per cent increase in the outcome
variable for the adopting households compared assumed to differ across studies. Under the
with the non‑adopting households. latter, each study presents new information that
is equally important, and therefore the weights
Many studies provided several estimates as they
are more balanced across studies with varying
presented different methods and estimators.
effect sizes. Given the fact that adoption of
To avoid double counting studies in the
improved varieties may have different impacts in
meta‑analysis, a single estimate was derived per
different settings, a random-effects model was
study. Following the discussion in Waddington
et al. (2014), the method with the lowest risk chosen to derive the final estimate.
of bias was chosen. Some methods were
considered unambiguously superior, and the
choice of the best estimate was straightforward.

22. Details on effect-size calculations for specific papers can be provided upon request.

134
Results were not representative of the population). Many
papers chose non-adopting households that
Step 1: Study selection were geographically close to adopting ones,
which may have produced spillover bias. One
Based on the search strategy, about 25,000
subset of papers failed to report all outcomes
titles in the selected databases were screened.
and omitted some necessary statistics. For
The search strategy identified 21 studies for
the generalizability score, many received lower
inclusion in the systematic review. Of these,
scores on rigour of analysis, context description
16 were found initially, and the other 5 were
and data collection methods.
identified by screening the bibliographies of
included studies and of existing systematic
Step 3: Meta-analysis
reviews. Out of these 21 studies, 7 study
poverty, while 12 report the effects on income A final aggregate estimate was computed and
and 8 on expenditure outcomes over different provided a weighted average of all estimates
crops (rice, wheat, maize, bananas, chickpea, (response ratios), where each study was
pigeon pea and groundnut); some studies weighted according to its variance (a measure
examine more than one type of impact. of the spread in the magnitude of the outcome
variable across surveyed households). Studies
Step 2: Critical appraisal of selected studies that reported a more precise estimate carried a
As described above, each study received three larger weight than studies with lower precision.
scores: bias, generalizability and a total score. The precision of the study estimate reflected the
None received the maximum score. For the bias sample size. However, it is important to note
score, most studies suffered from selection bias that the weight does not reflect the validity of
(e.g. selected households included in the sample methodology presented in the paper.

Figure 12.1a Results for poverty, disaggregated by region

Study %
ID ES (95% CI) Weight

Asia and the Pacific

Mendola 2007 − Nigeria 0.55 (0.32, 0.94) 0.99
Subtotal (I−squared = .%, p = .) 0.55 (0.32, 0.94) 0.99
.
East and Southern Africa
Asfaw et al. 2012 − Tanzania 0.78 (0.31, 1.95) 0.35
Simtowe et al. 2012 − Malawi 0.83 (0.33, 2.08) 0.34
Khonje et al. 2014 − Zambia 0.85 (0.50, 1.44) 1.03
Kassie et al. 2011 − Uganda 0.90 (0.29, 2.75) 0.23
PART III / Chapter 12

Mathenge et al. 2014 − Kenya 0.94 (0.93, 0.96) 47.02

Smale et al. 2014 − Zambia 1.00 (1.00, 1.00) 50.04
Subtotal (I−squared = 84.6%, p = 0.000) 0.97 (0.92, 1.02) 99.01
.
Overall (I−squared = 83.9%, p = 0.000) 0.96 (0.91, 1.02) 100.00

.292 1 3.42
Decrease Increase
Note: Based on inverse variance weighting/dependent effect sizes removed.

135
 Figure 12.1b Results for income, disaggregated by region
Study %
ID ES (95% CI) Weight

Asia and the Pacific

Mendola 2007 − Nigeria 1.32 (0.96, 1.82) 7.82
Wu et al. 2010 − China 1.34 (1.16, 1.55) 13.11
Subtotal (I−squared = 0.0%, p = 0.938) 1.34 (1.17, 1.53) 20.93
.
East and Southern Africa
Smale et al. 2014 − Zambia 1.01 (1.00, 1.01) 15.85
Mathenge et al. 2014 − Kenya 1.07 (1.00, 1.14) 15.25
Kassie et al. 2011 − Uganda 1.47 (1.13, 1.90) 9.43
Bezu et al. 2014 − Malawi 1.62 (1.31, 2.00) 10.91
Kikulwe et al. 2012 − Kenya 2.38 (0.75, 7.61) 1.10
Khonje et al. 2014 − Zambia 2.75 (1.86, 4.07) 6.28
Subtotal (I−squared = 91.4%, p = 0.000) 1.33 (1.14, 1.56) 58.83
.
Latin America
Rovere et al. 2009 − Mexico 1.28 (0.99, 1.64) 9.80
Subtotal (I−squared = .%, p = .) 1.28 (0.99, 1.64) 9.80
.
West and Central Africa
Dibba et al. 2012 − Gambia 1.01 (0.03, 31.10) 0.13
Nguezet et al. 2011 − Nigeria 1.46 (0.60, 3.56) 1.78
Wiredu et al. 2014 − Ghana 1.51 (1.13, 2.03) 8.53
Subtotal (I−squared = 0.0%, p = 0.972) 1.50 (1.14, 1.98) 10.44
.
Overall (I−squared = 87.4%, p = 0.000) 1.35 (1.19, 1.53) 100.00

.0322 1 31.1
Decrease Increase
Note: Based on inverse variance weighting/dependent effect sizes removed.

Figure 12.1c Results for expenditures, disaggregated by region

Study %
ID ES (95% CI) Weight

East and Southern Africa

Shiferaw et al. 2014 − Ethiopia 1.14 (1.11, 1.16) 82.14
Debello et al 2015 − Ethiopia 1.15 (1.09, 1.20) 17.24
Simtowe et al. 2012 − Malawi 1.18 (0.47, 2.96) 0.05
Asfaw at al. 2012 (2) − Ethiopia 1.25 (0.68, 2.28) 0.11
Khonje et al. 2014 − Zambia 1.32 (0.66, 2.63) 0.08
Asfaw et al. 2012 − Tanzania 1.36 (0.53, 3.53) 0.04
Subtotal (I−squared = 0.0%, p = 0.994) 1.14 (1.12, 1.16) 99.66
.
West and Central Africa
Adekambi et al. 2009 − Benin 1.41 (0.22, 8.89) 0.01
Audu & Aye 2014 − Nigeria 1.50 (1.06, 2.12) 0.33
Subtotal (I−squared = 0.0%, p = 0.953) 1.49 (1.06, 2.10) 0.34
.
Overall (I−squared = 0.0%, p = 0.897) 1.14 (1.12, 1.16) 100.00

.112 1 8.89
Decrease Increase
Note: Based on inverse variance weighting/dependent effect sizes removed.
Source: Authors’ analysis.
136
Figure 12.1 presents results by region. The of the world for adopting households relative
reported coefficients under the effect-size to non-adopters. A systematic review protocol
column are the response ratios, representing was followed in order to find relevant studies
the percentage change between adopting and appraise and quantify the direct aggregate
households and non-adopting households. impact. The methodology encompassed three
Figure 12.1 also reports the probable steps: (1) a rigorous search of relevant studies;
range of the estimated impact (95 per cent (2) a critical appraisal of the selected studies;
confidence interval) and the individual study and (3) a meta-analysis. A comprehensive search
weight. The 95 per cent confidence interval led to a final sample of 7 papers for poverty
of both the subtotal and overall effect size is outcomes, 12 for income and 8 for expenditure
represented by a diamond. Also reported is outcomes over different crops, and included
the I-squared statistic, which describes the unpublished papers.
percentage of variation across studies that is
The meta-analysis findings showed that
due to heterogeneity (this refers to the variation
adoption of CGIAR’s improved varieties has led
in study outcomes between studies). A higher
to statistically significant increases in income
I-squared suggests large differences in effect
and expenditures by 35 and 14 per cent,
sizes between the studies. The p value provides
respectively, for adopting households relative to
an indication as to whether one can consider
a valid comparison group. In addition, although
that all studies in the meta-analysis give a similar
improved varieties may have reduced poverty
effect size. A p value below 0.05 indicates that
by 4 per cent, this finding is not statistically
it is very likely not the case. The numbers on the
significant, and therefore one cannot rule out
horizontal axis delineate the range within which
the possibility of no impact, given both the
the estimates are contained.
paucity of studies and the large imprecision of
The forest plot shown in Figure 12.1a – a typical study estimates.
representation of meta-analysis results – shows
It is worth noting a number of limitations. First,
that adoption of improved varieties resulted
owing to a high risk of bias for some studies,
in a 4 per cent decrease in poverty. However,
results may not be perfectly representative of
this estimate is not significant, owing both
causal impacts. In addition, results may be
to the paucity of the studies and to the large
underestimated because of the period analysed
imprecision of the reported study estimates.
in some of the included impact evaluations.
The effect size for income indicates instead a
The studies’ timespan may be too short to
significant impact – about a 35 per cent increase
fully capture the welfare impacts of improved
for adopting households relative to non-adopting
varieties or agricultural research in general, given
households, with the largest increase occurring
that such impacts may take several years to
in West and Central Africa (50 per cent) (Figure
occur. Finally, this analysis may be subject to
12.1b). Finally, results for the expenditure
publication bias – that is, studies with negative or
outcome are also positive and significant (Figure
low impacts may be less likely to be published or
1c), with a 14 per cent increase on average, and
available in the databases.
PART III / Chapter 12

the largest increase being found in West and

Central Africa (49 per cent). Nevertheless, some implications for policy can
be derived from this study. Barriers to access
to improved seeds may exist, preventing poorer
Conclusion and policy farmers from fully benefiting from the potential
of such technologies. The same structural and
recommendations
contextual constraints to adoption emerge
In this systematic review, the aggregate direct across the included studies. These constraints
impact of CGIAR-related improved seed varieties are related to: human capital endowments (the
on welfare was synthesized across all regions education and experience of the household

137
 head); access to knowledge and the technology References
itself; information asymmetries (such as proximity
to the extension office and to markets, and Adato, M. and Meinzen-Dick, R. 2007.
information about the technology in general); Agricultural research, livelihoods, and poverty:
financial and physical capital endowments (initial Studies of economic and social impacts in
assets, farm size, livestock holdings, and off-farm six countries. Baltimore, USA: Johns Hopkins
income sources); social capital endowments University Press.
(group membership); and behavioural aspects Alene, A. D., Menkir, A., Ajala, S. O.,
such as risk aversion. Given the potential for Badu‑Apraku, B., Olanrewaju, A. S., Manyong,
strong welfare benefits for smallholder farmers, V. M. and Ndiaye, A. 2009. The economic and
policy interventions should aim at reducing poverty impacts of maize research in West and
the many constraints that households face in Central Africa. Agricultural Economics,
adopting improved varieties. More effort should 40: 535‑550.
be made to increase the scale and efficiency
of agricultural extension services and input Anderson, J. R. 1985. Assessing the impact
supply systems. The studies included in this of farming systems research: Framework and
analysis also point to the need for greater problems. Agricultural Administration, 20(4):
investments in developing local markets and 225-235.
road infrastructure in order to facilitate diffusion. Borenstein, M., Hedges, L. V., Higgins, J. P.
In addition, economic incentives for more T. and Rothstein, H. R. 2009. Introduction to
constrained farmers should be considered in meta‑analysis. Chichester, UK: John Wiley & Sons.
order to encourage early adoption and the use
de Janvry, A. and Sadoulet, E. 2002. World
of improved varieties; such incentives could be
poverty and the role of agricultural technology:
packaged with technology development as part
Direct and indirect effects. Journal of
of CGIAR research projects.
Development Studies, 38(4): 1‑26.

Diagne, A. and Zeller, M. 2001. Access to credit

and its impact on welfare in Malawi. Research
Report 116. Washington, D.C.: International
Food Policy Research Institute.

Evenson, R. E. and Gollin, D. 2003. Crop variety

improvement and its effect on productivity: The
impact of international agricultural research.
Wallingford, UK: CABI Publishing.

Fan, S., Chan-Kang, C., Qian, K. and Krishnaiah,

K. 2007. National and international research and
rural poverty: The case of rice research in India
and China. In M. Adato and R. Meinzen-Dick,
eds. Impacts of agricultural research on poverty
reduction: Studies of economic and social
impact in six countries, 285‑305. Baltimore,
USA: Johns Hopkins University Press.

Hazell, P. and Haddad, L. 2001. Agricultural

research and poverty reduction. IFPRI 2020
Discussion Paper 34. Washington, D.C.:
International Food Policy Research Institute.

138
Hedges, L. V. and Cooper, H. M., eds. 1994. The
handbook of research synthesis. New York, USA:
Russell Sage Foundation.

Irz, X., Lin, L., Thirtle, C. and Wiggins, S. 2001.

Agricultural productivity growth and poverty
alleviation. Development Policy Review, 19:
449‑466.

Maredia, M. K. and Raitzer, D. A. 2006. CGIAR

and NARS partner research in sub-Saharan
Africa: Evidence of impact to date. Rome:
Science Council Secretariat.

Nelson, M. and Maredia, M. 1999. Environmental

impacts of the CGIAR: An initial assessment.
Rome: Consultative Group on International
Agricultural Research (CGIAR), Technical
Advisory Committee Secretariat, Food and
Agriculture Organization of the United Nations.

Raitzer, D. A. and Kelley, T. G. 2008. Benefit-cost

meta-analysis of investment in the international
agricultural research centers of the CGIAR.
Agricultural Systems, 96(1‑3): 108‑123.

Renkow, M. and Byerlee, D. 2010. The impacts

of CGIAR research: A review of recent evidence.
Food Policy, 35: 391‑402.

Stewart, R., Langer, L., Rebelo Da Silva, N.,

Muchiri, E., Zaranyika, H., Erasmus, Y., Randall,
N., et al. 2015. The effects of training, innovation
and new technology on African smallholder
farmers’ economic outcomes and food security:
A systematic review. 3ie Grantee Final Review.
London: International Initiative for Impact
Evaluation (3ie).

Thirtle, C., Lin, L. and Piesse, J. 2003. The

impact of research-led agricultural productivity
growth on poverty reduction in Africa, Asia and
Latin America. World Development, 31(12):
PART III / Chapter 12

1959-1975.

Waddington, H., Snilstveit, B., Hombrados,

J. G., Vojtkova, M., Anderson, J. and White,
H. 2014. Farmer field schools for improving
farming practices and farmer outcomes in
low‑and middle-income countries: A systematic
review. London: International Initiative for Impact
Evaluation (3ie).

139