Impact evaluation methods: Qualitative Methods
Download as PPTX, PDF3 likes2,450 views
The document discusses the significance of qualitative methods in impact evaluation, emphasizing their ability to capture complex, nuanced information that quantitative methods may overlook. It highlights the flexibility and people-oriented nature of qualitative approaches, making them especially relevant for assessing climate adaptation projects and their unexpected outcomes. Additionally, it advocates for mixed methods to enhance the breadth and depth of evaluations and to effectively engage various stakeholders.
Impact evaluation methods: Qualitative Methods
- 1. Lini Wollenberg, CGIAR Research Program on Climate Change, Agriculture and Food Security and Gund Institute, University of Vermont LORTA Program Inception Workshop, 24-26 July 2018 Impact Evaluation Methods: Qualitative Methods
- 3. Why qualitative methods? (1) Numbers reduce information! Use qualitative methods to capture things that cannot be easily measured with numbers - Where there is a need for assessing • system-level impacts • complexity • context, meaning, • depth, detail, nuance, • open-ended answers, unintended effects e.g. adaptation v. mitigation - Where things cannot be observed, and description is needed, e.g. historical extreme events, scaling potential of intervention, project contributions in complex impact pathways
- 4. (2) Flexibility and reduced expenses • Rapid assessment, e.g. exploring potential counterfactual explanations, activity data for mitigation, dealing with shifting adaptation strategies (3) People-oriented research • People are “in, not out” of assessment system: Participatory research, formative assessment, adaptive management • Ethics of measurement e.g. value of a life, value of ancestral lands, sensitive questions in humanitarian/emergency relief programs, • Where truth or knowledge is determined by social discourse and discussion - e.g. meaning, values, political issues, bridging multiple knowledge systems; critical social analysis Why qualitative methods? (2)
- 6. Examples of qualitative methods in impact assessment Bamberger 2012
- 7. Role of qualitative methods in the research process • Conceptual frameworks concerning social values, differences or power: human rights, legal, feminist or social inclusion frameworks • Research design: comparison; single case analysis; hypotheses may develop during research; flexibility for iterative, adaptive inquiry • Sampling: smaller numbers, purposive, representation of stakeholders • Data collection: may be less structured and open ended, more flexible, evolves with increased knowledge
- 8. Limits to qualitative methods Fit with scientific method • Limited statistical representativeness: can be highly context specific, small samples, difficult to generalize • Methods may not be as transparent or reproducible • Rich, but complicated information (spaghetti diagrams) • Causality can be difficult to isolate • Aggregation of results or comparison can be difficult Researcher as a person • Can require rapport or trust for optimal results • Researcher’s identity, perspective and behavior may affect methods (e.g. participant observation) Communicating results • Demand for numbers from policy makers, donors, media
- 9. Mixed methods Integrate approaches, not only in data collection, but in conceptual framework, design, analysis, interpretation Use multiple methods to • complement each other • triangulate results to test validity • inform each other (e.g. indicators for surveys) • support diverse ways of knowing or communicating and buy in from multiple audiences - Worry less about qualitative-quantitative divide and more about fit of method for aims - Problems arise more because of expectations and differences in team capacities and management
- 10. Example of mixed methods Bamberger 2012
- 11. Example of mixed methods Bamberger 2012
- 12. Qualitative assessment of climate action projects Some conclusions • Especially relevant to climate projects, e.g. assessing adaptation impacts, unexpected outcomes, complex impact pathways • Can facilitate learning and ownership to augment project outcomes. • Focus on quantitative or qualitative approach alone will limit results • Mixed method approaches, especially with sequential approaches, can ensure good mix of breadth and depth and effective project engagement and communication of results.
- 13. Some resources • GIZ. Impact evaluation guide for climate chage adaptation projects https://www.adaptationcommunity.net/?wpfb_dl=260 • Bamberger, M., Rugh, J., Church, M., Fort, L., Shoestring Evaluation: Designing Impact Evaluations under Budget, Time and Data Constraints, American Journal of Evaluation, 25(1), 2004, pp. 5 – 37. • Silvestrini, S. , Organizational Aspects of Evalu- ations in: Stockmann, R. (ed.), A Practitioner Handbook on Evaluation, Edward Elgar, Cheltenham, 2010. • Stockmann, R. (ed.), A Practitioner Handbook on Evaluation, Edward Elgar, Cheltenham, 2010.
Editor's Notes
- #3: The common distinction between Quant and Qual is you can use a thermometer to measure a fever (or the climate) but you can also put your hand on the forehead (or take this photo/ask people) as a qualitative measure. Or some say that qualitative methods are what social scientists do and hard scientists don’t like, but in fact the differences are much more blurry in practice as I hope to show you and mixed methods approach allows the best of both worlds.
- #4: So why qual methods in the first place? At least three major reasons
- #6: participatory planning workshop with communities in Myanmar using @CAFOD @IIED energy delivery models approach Engagement usually ensures relevance and ownership of results. Development projects are usually strong on stakeholder engagement and participatory methods because they work.
- #7: ** Indicates that these techniques can be used both quantitatively and qualitatively. They are placed in the column where they are most commonly used.
- #9: But these limits are not necessarily always the case
- #11: https://www.interaction.org/sites/default/files/Mixed%20Methods%20in%20Impact%20Evaluation%20(English).pdf
- #12: https://www.interaction.org/sites/default/files/Mixed%20Methods%20in%20Impact%20Evaluation%20(English).pdf