Download to read offline
Uploaded byPractical Playbook
Validating & Promoting your Program's Success Using ROI and other Evaluation Methods
The document outlines steps for developing a performance measurement and evaluation plan for public health programs. It discusses developing logic models and selecting process and outcome measures, collecting interim data, and calculating return on investment. The goal is to show stakeholders like funders and policymakers how programs are performing and achieving impacts in a manner that promotes continuous program improvement.
More Related Content
Similar to Validating & Promoting your Program's Success Using ROI and other Evaluation Methods
More from Practical Playbook
Validating & Promoting your Program's Success Using ROI and other Evaluation Methods
- 1. June 1st, 2017 KarlEnsign, Chief, Performance Improvement, Research & Evaluation (PIRE) Association of State & Territorial Health Officials (ASTHO) Validating & Promoting your Program’s Success Using ROI & other Evaluation Methods 2nd Annual Practical Playbook National Meeting Westin City Center --Washington, DC
- 2. What’s keeping youup at night…??? II need to show how my programs are performing. How do I develop performance measures??
- 3. Things to remember •Takes time and effort – doesn’t “just happen” • Build in staff time and resources • Data collection can’t be sprung on people last minute – quality and reliability will suffer • Make this an expectation from the beginning • Important that measures aren’t seen as “imposed from above” – may not reflect a full understanding of implementation • Involve key stakeholders in all aspects of design and interpretation • Most successful as a learning process – knowledge development • Not a thumbs up or thumbs down but rather continuous improvement
- 4. Differences between monitoring& evaluation • Monitoring – Is the program, policy, or initiative functioning as intended? • Evaluation – What is the impact of the program, policy, or initiative?
- 5. Beginning the journey EvaluationRoadmap Phase I: Planning and conceptualization Phase II: Data collection and analysis Phase III: Reporting and action
- 6. Phase I: Planning& Conceptualization First Steps… Get up to speed Engage the input of key stakeholders •Policymakers •Funders •Program Administrators •Data collectors Build a logic model
- 7. But what exactlyis a logic model…???
- 8. Inputs Activities OutputsOutcomes Short-term Intermediate Long-term Resources What we do What is produced What changes Logic model components Contextual Variables
- 9. Advantages of a logicmodel • Interactive, collaborative process • Makes the theory of change explicit • Highlights areas of agreement, ambiguity • Gets everyone on the same page • Becomes a living, orienting framework throughout the project
- 10. • Slide withoutswoosh and logo.
- 11. •Process Measures (Monitoring) •Whatwas produced? •Outcome Measures (Impact) •Change in knowledge, status, behavior, function, values A word about process measures & outcomes
- 12. Select evaluation measures& definitions from logic model Measure Definition Instructions Process Measure: Partnerships formalized for Million Hearts initiative # of MOUs Choose measure and each definition that applies to your initiative. Numerator will reflect current number of partnerships formalized. Denominator will reflect goal. # of contracts signed # of informal partnerships established Process Measure: Protocols developed and put in place # of state data exchange protocols developed Choose measure and each definition that applies to your initiative. Numerator will reflect current number of protocols developed. Denominator will reflect goal for protocols for each year. # of local data exchange protocols developed # of local client referral & follow-up protocols developed # of state client referral & follow protocols developed Outcome Measure: Patients with reduced blood pressure # of patients with measureable reduction in BP over baseline (for instance, 5 mm diastolic and 10 mm systolic change). Should not include patients with reduced and controlled BP, only those with reduced BP. Numerator is hypertensive patients referred to services with reduced BP. Should not include patients with reduced AND controlled BP, only those with reduced BP. Denominator is patients referred as hypertensive.
- 13. Phase II: DataCollection & Analysis – Get going! Require interim data collection and reporting Data collection and reporting templates help Need to conduct frequent quality and consistency checks Feedback interim results to advisory group and questions to data collectors
- 14. A good measureis SMART! measure Example – By the end of the project year, the 4 regional collaboratives will include 4 – 6 states on average. Example – By the end of the project year, increase estimated state response time for certain emergencies by 25%. Example – By the end of the project year, 4 states will have implemented most aspects of the comprehensive strategies outlined in the environmental health toolkit.
- 15. A word aboutperformance measures or…“process measures with an expectation” 0 1 2 3 4 5 6 Category 1 Category 2 Category 3 Category 4 Chart Title Series 1 Series 2 Series 3 Goal Red Flag
- 17. “ROI is oneway of measuring and communicating public health effectiveness in a manner that is particularly salient for policymakers, funders, administrators and the general public” • Helps answer the following questions: • Are we making the right investments? • Are we becoming more efficient? • What is our budget accomplishing? • What returns do different investments yield? • Must be done thoughtfully and carefully A Word about ROI
- 18. ROI calculated: Netbenefit = Benefits – Costs ROI = Benefits – Costs Costs • Hypothetical values: ̶ $5 = ($400 + $500 + $300) – ($150 + $50) ($150 + $50) Or …
- 19. “…a dollar spenton pediatric immunization is estimated to save $5 in treating preventable illness.”
- 20. ASTHO’s Web-Based ROITool Let’s see how it works!
- 21. Tool structure – Processimprovements or project implementation • Define process/project time periods • Plan Phase – Pre-Implementation, Baseline • Do – Initial Implementation • Study – Mid-course Corrections • Act – Full Implementation
- 22. ROI tool makescomparisons over time (PDSA) • Investment costs • Planning and tracking implementation of the initiative (labor, other direct, and indirect costs) • Routine operating costs • Delivery of project or program that is implemented or altered (labor, other direct, and indirect costs) • Outputs or outcomes • Additional benefits (health outcomes) realized through the initiative
- 23. Planning and DevelopmentCosts Planning & Tracking Cost Category Plan (Time1) Do (Time2) Study (Time3) Act (Time4) Personnel Costs Non-Personal • Contracted Services • Office Operations • Facilities/Maint/Rent • Communications • Equipment • Construction/ Renovation Other Direct Costs Indirect Costs
- 24. Routine Operating Costs Delivery CostCategory Plan (Time1) Do (Time2) Study (Time3) Act (Time4) Personnel Costs Non-Personal • Contracted Services • Office Operations • Facilities/Maint/Rent • Communications • Equipment • Construction/ Renovation Other Direct Costs Indirect Costs
- 25. Phase III: Reporting& Action • Engage stakeholder group in reviewing the findings, shaping message • Don’t bury findings in lengthy reports • Executive Summary – short on analysis process and big on findings and implications • Have short, concise, stand-alone documents for key messages and audiences • Develop action plan for dissemination and communication • Who • Where • How
- 26. Getting from processto impact Target Population Patients Identified Patients Referred Patients Followed-Up General Population Undiagnosed Patients Screened Patients with Reduced/Controlled Blood Pressure Patients Adhering to Treatment Plan
- 27. Help us tellyour story… Since we began our CDC/ASTHO Million Hearts Learning Collaborative work, we have identified _#_ people in our target population. By implementing our intervention, _#_ of undiagnosed patients were screened for hypertension. Of those, _#_ were identified as hypertensive. Of those identified as hypertensive, _#_ were referred to services. Of those referred to services, _#_ followed-up on their referral. Of those who followed-up, _#_ are adhering to a treatment plan. Of those adhering to a treatment plan, _#_ have reduced blood pressure. Of those with reduced blood pressure, _#_ have their hypertension under control.
- 28. Key points forthe evaluation journey… Select the strategy that is right for you and the situation Evaluation – impact, ROI Monitoring Don’t shortchange the time and effort needed The 10% rule Build expectations into project from beginning Involve a wide range of stakeholders early and often Program administrators and end-users of the analysis Can help foster a dialogue about intent of policy/program and data interpretation Especially when starting, less can be better A few key measures and data collection points can go a long ways Process measures form the foundation Outcome measures show the impact Think about the end first!!
- 29.
Editor's Notes
- #30 Bertha