Nominet trust projects theory of change presentation 2016
Download as PPTX, PDF5 likes639 views
This document provides an overview of theory of change approaches for evaluating social impact. It defines theory of change as describing how specific changes are expected to occur as a result of interventions and actions. The document discusses best practices for developing theories of change such as having plausible, doable, and testable causal links between activities and outcomes. It also addresses challenges such as the complexity of social systems and limitations of research available. Throughout, it provides exercises and examples to illustrate key concepts for developing and critiquing theories of change.
Nominet trust projects theory of change presentation 2016
- 1. www.nominettrust.org.uk Theory of Change & Measuring Social Impact
- 2. www.nominettrust.org.uk Theory of Change – Background and Use • EVALUATION • Planning • Engagement • Communication (verbal and visual)
- 3. www.nominettrust.org.uk Evaluation – 3 questions What? So What? Now What? Michael Quinn Patton
- 4. Exercise 1: What is my project/organisation trying to change? (Team Discussion – 15mins)
- 5. www.nominettrust.org.uk Theory of Change - Definition “No matter how it is defined, at its heart, a theory of change lays out what specific changes the group wants to see in the world, and how and why a group expects its actions to lead to those changes. “ (Guthrie et al. 2005 – emphasis by Lamb, 2011, p6)
- 6. Exercise 2: Theory of Change – Elevator Pitch (Groups – 20mins) 1. What are we (i.e. my project/organisation) trying to change? 2. What are we doing to bring about that change? 3. How and why do we believe our activities will bring about the change we want to see?
- 7. Outputs Outcomes Tangible results of the your work The changes that result (new behaviours, changes in well being, increased skills etc... Outputs vs Outcomes
- 8. Logic Models
- 9. Causality – ‘Black Box’ Evaluations Programme or Intervention (e.g. parenting course) Outcomes (e.g. happy, well-adjusted children)
- 11. www.nominettrust.org.uk Theory of Change approach – In a word REALLY?
- 12. www.nominettrust.org.uk Questioning Causality (aka ‘exploding the arrow’) i.e. asking whether an organisation’s activities will bring about the changes they intend in the way they describe.
- 13. www.nominettrust.org.uk Causality ‘Exploding the Arrow’ (1) 1) Evidence – what evidence is there the intervention will lead to the desired outcome? 2) Single or multiple steps – will the intervention lead directly to the desired outcome* *....or are there a number of micro steps along the way? Is there a chain of connected outcomes? (aka: ‘so that’ chains) (Funnell and Rogers, 2011)
- 14. Outcomes Chain – Example (Text)
- 15. Causality – Outcomes Chains (blank) Programme or Intervention (e.g. parenting course) Outcomes (e.g. happy, well-adjusted children) ? ? ? ?? ?
- 16. Exercise 3: Discussion – Outcomes Chain (Team – 10mins)
- 17. www.nominettrust.org.uk Causality ‘Exploding the Arrow’ (2) 3) Linear or multifaceted – is the outcomes chain a simple linear process or is it multifaceted?* 4) Internal and External factors – does the outcomes chain take into account the influence of internal and external factors? *Do separate outcomes chains need to run in parallel in order to combine towards the final, ultimate outcome? Necessary vs. Sufficient. (Funnell and Rogers, 2011)
- 18. www.thinknpc.org
- 19. www.nominettrust.org.uk Causality ‘Exploding the Arrow’ (3) 5) Plausibility and coherency – ultimately any plan that details a causal link between an intervention and an outcome needs to be scrutinised, ideally by those who have knowledge relevant to the intended change (e.g. practitioners, academics and intended beneficiaries). (Funnell and Rogers, 2011)
- 20. www.nominettrust.org.uk Making the implicit explicit “..takes for granted that social programs are based on explicit or implicit theories about how and why the programme will work. The evaluation should surface those theories and lay them out in as fine detail as possible, identifying all the assumptions and sub- assumptions built into the program. … The aims is to examine the extent to which the program theories hold.” (Weiss, 1995, pp66-7)
- 21. Assumptions Programme or Intervention (e.g. parenting course) Outcomes (e.g. happy, well adjusted children) ? ? ? ? ? ? ?
- 22. Exercise 4: Assumptions – Outcomes Chain (Groups – 15mins)
- 23. www.nominettrust.org.uk What makes a good theory of change? (1) 1) Plausible - stakeholders believe the logic of the model is correct: if we do these things, we will get the results we want and expect 2) Doable - human, political and economic resources are seen as sufficient to implement the action strategies in the theory Source: https://communities.usaidallnet.gov/fa/system/files/Applying+Theory+of+Change+Approach. pdf
- 24. www.nominettrust.org.uk What makes a good theory of change? (2) 3) Testable - stakeholders believe there are credible ways to discover whether the results are as predicted) 4) Meaningful - stakeholders see the outcomes as important and the magnitude of change in these outcomes being pursued as worth the effort
- 25. www.nominettrust.org.uk Theories of Change – Some Examples
- 28. From Mayne, J (2008)
- 29. One Project – Several theories of change Some projects use a limited number of activities to serve multiple purposes. For example, a project involving the construction of a water well in a village in Burundi could be linked to the following theories of change: • By providing access to a closer water source, we will improve safety (less danger of assault or rape, as is the case in many refugee camps) and health (improved sanitation, no contamination in drinking water, ability to cook with water). • If there is closer access to water, then children can attend school instead of having to travel for hours to provide water for the family. • If we teach locals how to build water wells, then they can repeat the process in neighbouring villages, thus spreading access to water throughout the region. Practical Approaches to Theories of Change in Conflict, Security & Justice Programmes. Department for International Development
- 30. www.nominettrust.org.uk Defining success – the ToC approach: 1) Offers guidance on how to get to your ultimate goal, it will not tell you what that goal should be 2) Is dependent on having a clearly articulated ultimate goal, but defining, agreeing and sticking to such a clear can be a challenge
- 31. www.nominettrust.org.uk Theory of Change Toolbox ToC approach contains different elements and levels of detail that will appeal variously to organisations depending on their interests, values, and size.
- 32. Theory of Leverage “Give me a place to stand and with a lever I will move the whole world.” Archimedes
- 33. Lean Start Up and Theories of Change: Interesting parallels e.g. Testing assumptions (ToC) and validating hypotheses (LSU). www.theleanstartup.com/
- 34. www.nominettrust.org.uk Criticisms and Challenges • Limited research available – especially for non-US contexts and smaller charities • Most documentation produced by ToC ‘champions’ • ToC still an emerging approach (no orthodoxy)
- 35. www.nominettrust.org.uk Criticisms and Challenges Complexity Science and Complex Adaptive Systems 1. Complex systems can be defined as having various characteristics including a large number of interacting elements and being dynamic compositions, where the whole is larger than sum of parts (Snowden & Boone, 2007, p71). 2. Cause and effect pathways often unclear, and visible only in hindsight
- 36. Further reflections The more complex (and less linear) the social change the more difficult it is to develop a ToC ToC approach about unearthing assumptions, but also values, beliefs and paradigms Tension - evaluators like stable theories and models, whilst innovators happier working with still evolving models Half-life/strategic decay. Also applies to ToC, which need to be refreshed/reviewed. ToC change. The three main reasons are: 1) context changes 2) new players/change makers, and 3) new evidence. Complex change = variety (Gladwell’s pasta sauces), i.e. not 'what works', but 'what works for who, in what context and why?'
- 37. ToC Approach – better as a thinking tool than a planning tool?
- 38. Resources Papers & Books • Brest, P. (2010). The Power of Theories of Change. Available: http://www.kipp.org/news/stanford-social-innovation-review- the-power-of-theories-of-change- • Clark, H and Anderson, A. (November 2004) Theories of Change and Logic Models: Telling Them Apart. Presentation at American Evaluation Association, Atlanta, Georgia. http://www.theoryofchange.org/wp- content/uploads/toco_library/pdf/TOCs_and_Logic_Models_forAEA.pdf • Funnell, S. C. and Rogers, P. J. (2011) Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. Jossey-Bass • Keystone Accountability. (2008). Developing a theory of change: A guide to developing a theory of change as a framework for inclusive dialogue, learning and accountability for social impact. Available: http://www.researchtoaction.org/theory-of- change-useful-resources/ • Lamb, B. (February 2011) Campaigning for Change: Learning from the United States. Campaigning Effectiveness, NCVO. http://www.ncvo-vol.org.uk/sites/default/files/campaigning_for_change_-_learning_from_the_us.pdf or http://www.advocacyinitiative.ie/resource/campaigning-change-learning-united-states • Mayne, J (2208) Contribution Analysis: An approach to exploring cause and effect, ILAC, http://www.cgiar- ilac.org/files/ILAC_Brief16_Contribution_Analysis_0.pdf • Stachowiak, S. (2009) Pathways for Change: 6 Theories About How Policy Change Happens. Seattle, Organizational Research Services.http://www.organizationalresearch.com/publicationsandresources/pathways_for_change_6_theories_about_how_ policy_change_happens.pdf • Weiss, C. (1995) ‘Nothings as practical as a good theory: Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families’. In Connell, J. P. et al. (eds) Approaches to Evaluating Community Initiatives: Concepts, Methods and Contexts. Washington; Aspen Institute. Available: http://www.eric.ed.gov:80/PDFS/ED383817.pdf Links: http://www.theoryofchange.org http://www.thinknpc.org/publications/theory-of-change/ http://www.ces-vol.org.uk/tools-and-resources/Evaluation-methods/making-connections-tools http://www.aecf.org/resources/theory-of-change/ www.whatleads.to For a critical view: • Reusga, A. (2011). Philanthropy’s Albatross: Debunking Theories of Change. http://postcards.typepad.com/Philanthropy%20s%20Albatross.doc
Editor's Notes
- #3: Evaluation – both progress against end goal and process (i.e. Did it work, did it work as we thought it would?) Theory of Change (ToC) approaches to evaluation of community-based change initiatives also emerged in the mid-1990s. Notable early exponents include the Aspen Institute (Weiss, 1995) and The Annie E. Casey Foundation (1997). Although the focus of these early discussions was evaluation the argument that misinformed or conflicting theories of change could reduce the impact of a given initiative had direct implications for programme planning (Weiss, 1995, pp67, 71-2). History (h/t Mark Cabaj http://www.naturalstep.ca/mark-cabaj): Evaluators in the 1960s concluded the policies and programs they were asked to evaluate did not come with clear enough goals/objectives and designs to develop an evaluation. Evaluators created “evaluability” assessment to help surface these issues early in the planning process. Evaluators began to get involved in helping policy makers and program designers to conceptualize the “intervention”.
- #4: Michael Quinn Patton – Essentials of Utilization Focused Evaluation pgs 3-4 Evaluation leads to action
- #5: Assumption that something needs to be changed i.e. things could be better than they are? There is a job to be done.
- #6: Defined as three key questions (‘What are you trying to change?’, ‘How will you affect change?’, and ‘Why do you believe your actions will affect this change?’) the ToC approach can provide a useful structure to frame important conversations.
- #7: Short statement (40-60secs) that contains all three elements. Clear, memorable, and makes sense to a stranger. If you can’t summarise your ToC then it’s probably too complicated, and unclear.
- #9: W. K. Kellogg Foundation. (2004). Logic Model Development Guide. Battle Creek, MI: W. K. Kellogg Foundation. 2004. Available: http://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg-Foundation-Logic-Model-Development-Guide.aspx Common in planning practice to use a reverese logic model process i.e. Start with end result/impact and work backwards.
- #10: Funnell, S. C. and Rogers, P. J. (2011) Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. Jossey-Bass Black box – “In science, computing and engineering a black box is a device, system or object, which can be viewed in terms of its input, output and transfer characteristics without any knowledge of its internal workings.” Wikipedia
- #11: If there’s no explanation or prediction of how and why change will occur then it is very hard to evaluate a project’s claims/effectiveness
- #12: Scrutiny, challenges assumptions – esp. around causation. Will explain in further slides.
- #13: One of the key features of the ToC approach is the emphasis on scrutinising the causal linkages that exist between an organisation’s activities and its intended goals i.e. asking whether an organisation’s activities will bring about the changes they intend in the way they describe. Important, but can be difficult. Requires time and access to both information, expertise and partnerships
- #18: Internal factors (e.g. peoples, systems, and resources) External factors (e.g. changes in government policy, economy, or population) Mayne, J (2208) Contribution Analysis: An approach to exploring cause and effect, ILAC, http://www.cgiar-ilac.org/files/ILAC_Brief16_Contribution_Analysis_0.pdf Also, stuff happens - wider implication for ToC approach as it demonstrates problems of predicting future events (how do you factoring unexpected events into your outcomes chain).
- #21: Weiss, C. (1995) ‘Nothings as practical as a good theory: Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families’. In Connell, J. P. et al. (eds) Approaches to Evaluating Community Initiatives: Concepts, Methods and Contexts. Washington; Aspen Institute. Available: http://www.eric.ed.gov:80/PDFS/ED383817.pdf
- #22: What are the assumptions? E.g. first ‘?’ resources, need, design, etc. Can also add risks and measures. Risks can be rated similar to risk register exercise: Score 1-10 how important is this assumption / Score 1-10 How confident are we the assumption is correct?
- #23: What are the main assumptions underlying each step of outcomes chain? Also what are the main risks? What are the indicators or measures i.e. how will we know that each step, each change, has occurred in the way that we predicted?
- #24: Plausible aka the sniff test e.g. This app will eradicate malaria Can be a challenge as well i.e. Radical innovations may not seem plausible to entrenched or embedded practice
- #27: Confidential – please do not share without permission. Nominet Trust
- #28: https://vinspired.com/media/W1siZiIsIjIwMTQvMDYvMTAvMDkvMDQvMjMvODg4L3ZJbnNwaXJlZF9JbXBhY3RfUmVwb3J0XzIwMTQucGRmIl1d
- #29: Mayne, J (2208) Contribution Analysis: An approach to exploring cause and effect, ILAC, http://www.cgiar-ilac.org/files/ILAC_Brief16_Contribution_Analysis_0.pdf
- #30: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/304613/Practical-approaches-theories-change-conflict-security-justice-prog.pdf
- #31: Fixing a clear (and measurable) end goal/destination is crucial, but can be a time-consuming, complex and sensitive process
- #33: Joel Orosz? – complementary question to TOC Particularly relevant to funders as often faced with challenges/opportunities that vastly outsize their resources – where can we make the most different with what we have?
- #34: Reis, E The Lean Startup: How Constant Innovation Creates Radically Successful Businesses (2011)
- #36: Snowden, D. and Boone, M (2007) ‘A Leader's Framework for Decision Making.ʼ Harvard Business Review. November 2007 85 (11) p68-76 Human beings (both individual and especially collectively) are not machines, difficult to crank a handle and get predictable results. Connection between cause and effect often unclear, inconsistent and disproportionate.
- #38: Perhaps impossible to cover every possibility and predict sequence of outcomes in advance. However ToC could be seen as a very useful, open process, for constanttly scrutinising and testing out assumptions in order to improve what we do.