Maria Raciti: Evaluating Indigenous outreach programs in five steps

Maria Raciti: Evaluating Indigenous outreach programs in five steps

• 1.
  Evaluating Indigenous Outreach Programsin Five Steps Maria Raciti Kalkadoon-Thaniquith/Bwgcolman Professor of Marketing, University of the Sunshine Coast Director of the Indigenous and Transcultural Research Centre Adjunct Fellow, National Centre For Student Equity in Higher Education
• 2.
  Why evaluation matters Demonstrates successand provides evidence to support ongoing funding Provides an ongoing check that the chosen strategies are working as anticipated Improves program design, implementation and impact
• 3.
  Indigenous outreach success isdefined differently by different stakeholders
• 4.
  Figuring out what, where,when and how to evaluate is not always easy
• 5.
  Five steps toevaluation Step 1: Visualise the big picture Step 2: Conduct a self-evaluation Step 3: Plan Step 4: Implement and analyse Step 5: Share
• 6.
  Step 1: Visualisethe big picture Adaptation of : Http://www/cdc.gov./framework; https://www3.epa.gov/evaluate/lm-training/ • We used these resources Inputs • For these activities Activities • To produce these outputs Outputs • To encourage participants to change their behaviour Outcomes • Leading to long term results (change in the condition or situation) Impact The ‘logic model’ is a simple picture of the Indigenous outreach program that illustrates the relationship between the inputs and its intended results.
• 7.
  Example Adaptation of :Http://www/cdc.gov./framework; https://www3.epa.gov/evaluate/lm-training/ •$  staff, catering, accommodation, travel and incidentals Camp Inputs •Camp activities: #1 Welcome; #2 Academic activity; #3 Cultural activity etc. Camp Activities •Participant numbers, confidence, clarity, engagement, satisfaction etc. Camp Outputs •Intention to attend your university; positive word-of- mouth University Outcomes •Intention to attend any university WP Impact Indigenous Outreach Camp
• 8.
  Step 2: Conducta self-evaluation Do we fully understand the expectations of our camp participants and their families? 01 Do we fully understand how ‘going to university’ is situated within the lives of our targets? 02 Do we fully understand the barriers and enablers facing our camp participants? 03 What did we learn from the last camp and have made changes to address these? 04
• 9.
  Step 3: Plan Evaluationoccurs across all phases of an outreach program Evaluation includes outreach program participants and partners Compare to previous outreach programs to show change over time There is no one ‘best’ method of collecting data
• 10.
  Plan using anevaluation table Phase What you are evaluating? Who is the sample? When will you evaluate? How will you evaluate? Pre-camp Confidence in academic ability and in cultural identity Participants Confirmation of attendance Brief online survey Day #1: Welcome Settling in and making connections Ambassadors and staff End of day #1 Observation notes Day #2: Academic activities Engagement with academic activities Academics End of day #2 Brief phone interview Day #3: Cultural activities Enrichment of cultural identity Participants End of activity Survey via mobile phones Day #4: Farewell Satisfaction and intentions Participants Beginning of day #4 Survey via mobile phones Day #8: Follow-up Experience and impact Parents and school Week following camp Brief online survey Brief phone interview
• 11.
  Example questions for participants •Did thecamp meet your expectations? •Will you stay in contact with the other students you met at the camp? •What aspect of the camp did you find most useful? What did you learn that surprised you? •Did the camp help you to clarify what you want to do when you finish school? How likely is it that you will go to university when you finish school? •How likely is it that you will come to our university when you finish school? •How likely are you to recommend the camp to others? •If you could change just one thing about our camp, what would it be?
• 12.
  Example questions for internal partners: Ambassadors and academics What workedwell? How engaged were participants during your time with them? What were the top three questions you were asked by participants? Were the resources or training that we provided for you helpful? What could we do better next time?
• 13.
  Example questions for external partners: Parents and schools Howsatisfied were you with our communication about the camp? Was the camp a good experience for your child/student? •Do you feel the camp helped your child/students to decide what they want to do when they finish school? • Do you feel the camp helped enrich your child/students cultural identity? •How likely is it that your child will attend university after they finish school? •How likely is it that you child will attend our university after they finish school? • What did we do well? • If you could change just one thing about our camp, what would it be?
• 14.
  Step 4: Implementand analyse
• 15.
  Step 5: Share