Evaluating an open research project: Benefits and challenges from the ROER4D evaluation
Download as PPTX, PDF1 like880 views
The document evaluates the ROER4D project aimed at enhancing the use and impact of Open Educational Resources (OER) in the Global South through a utilization-focused evaluation framework. It discusses the benefits of being open, such as fostering an inclusive team dynamic and addressing educational needs, alongside challenges like the limitations of sharing evaluation work openly. Key findings emphasize the importance of ethical considerations and adaptability in evaluation design and communication.
![Further reading:
About ROER4D:
• Hodgkinson-Williams, C. (2013). Research on Open
Educational Resources for Development in Post-secondary
Education in the Global South (ROER4D) - Scoping Document.
Available online: http://hdl.handle.net/11427/8430 [Last
accessed 10 February 2015].
• Hodgkinson-Williams, C. and Cartmill, T. (2014). Research on
Open Educational Resources for Development in the Global
South: 1st Technical Report 23 June 2013 to 27 August 2014.
Available online: http://hdl.handle.net/11427/9695 [Last
accessed 11 February 2015].
About evaluation:
• Patton, M. Q. (2008). Utilization-focused evaluation.
California: Sage Publications Inc.
• Ramirez, R. and Brodhead, D. (2013). Utilization Focused
Evaluation: A Primer for Evaluators. Penang: Southbound.
• Scriven, M. (1991). Evaluation Thesaurus. California: Sage
Publications Inc.](https://image.slidesharecdn.com/oeglobalpresentation10march2017finalslideshare-170310105836/85/Evaluating-an-open-research-project-Benefits-and-challenges-from-the-ROER4D-evaluation-22-320.jpg)
Evaluating an open research project: Benefits and challenges from the ROER4D evaluation
- 1. Sarah Goodier ROER4D Evaluation Advisor Centre for Innovation in Learning and Teaching University of Cape Town OEGlobal 2017 March 2017 Evaluating an open research project: Benefits and challenges from the ROER4D evaluation 3/10/20171
- 2. Outline • The project being evaluated: ROER4D • Evaluation framework • Benefits and challenges of being “open” • Discussion & conclusion
- 4. in the Global South • In what ways, and under what circumstances can the adoption of OER address the increasing demand for accessible, relevant, high-quality and affordable education and what is its impact in the Global South? Research on Open Educational Resources for Development http://roer4d.org/
- 5. ROER4D has 100+ researchers across 18 sub-projects in 26 countries across 16 time zones
- 6. 1. Build an empirical knowledge base on the use and impact of OER in education 2. Develop the capacity of OER researchers 3. Build a network of OER scholars 4. Communicate research to inform education policy and practice 5. Curate output as open content ROER4D Objectives
- 8. 1. Build an empirical knowledge base on the use and impact of OER in education 2. Develop the capacity of OER researchers 3. Build a network of OER scholars 4. Communicate research to inform education policy and practice 5. Curate output as open content ROER4D Key Evaluation Areas EVALUATION Using the utilization focused evaluation (UFE) framework
- 9. What is utilization focused evaluation (UFE)? UFE is centered around intended use by intended users - What do the users (key stakeholders) want to know? - How will answers to the evaluation questions potentially help to improve the project?
- 10. UFE in 12 steps: Steps are iterative, not linear Utilization Focused Evaluation Framework (Adapted from Ramirez & Brodhead, 2013)
- 11. Evaluation work guided by… • Intended use by intended users • Propriety = a key guiding standard in evaluations – Is it legal? – Is it ethical? – Is it fair?
- 12. Project work guided by… • Principle that research is only valuable if it is used • Make open – by default, keep closed by exception – if it adds value – if it is ethical – if it is legal
- 13. The team dynamic in the network hub is very open and inclusive. (Transparency, Adaptability) The open and dynamic nature of the project can also result in opportunities - new activities and outcomes related to the priority objectives will provide new windows of insight (Adaptability) Benefit from the other components of the evaluation work being undertaken (Reciprocity) Benefits of being “open”
- 14. 2. Develop the capacity of OER researchers - Evaluation data showed that the initial series of webinars (run in 2014) experienced varying degrees of success: - Decreasing attendance over the series - Several barriers to attendance (e.g. timezones) - Researchers found supporting materials & session recordings helpful Changes in timing and number of sessions implemented in 2015 webinars Examples
- 15. Examples 4. Communicate research to inform education policy and practice - Tracking analytics on a regular basis has helped to shape the project direction in terms of which communications platforms are working and what it may be good to focus on
- 16. Challenges to being “open” Evaluation work, by its nature, cannot always be conducted in the open - what can and can’t be shared openly and at what stage? (Propriety, Vulnerability, Liability) The geographical distance between the hub and many of the sub-projects - differences in time-zones, language, ICTs available (Connectivity, Temporality)
- 17. General Examples - Assumption of keep closed by default, make open by exception - Many evaluation outputs have not been shared openly; different process to other project outputs - Where possible, preparing evaluation outputs specifically for different audiences (internal/ external): what level of data could be shared? e.g. processes, methods and/or high-level findings and recommendations
- 19. Key points Key benefit: good fit between open and inclusive team dynamic and the participatory UFE framework Key challenge: evaluation work cannot always be conducted in the open
- 20. Take care in evaluation output production and consider carefully what can (and can’t) be shared openly and at what stage. Propriety needs to be at the heart of Evaluation design and implementation
- 21. This work is licensed under a Creative Commons Attribution 4.0 International License. Website: http://roer4d.org Contact author: Sarah Goodier [email protected] @SarahGoodier Excluding images, screenshots and logos and/or unless otherwise indicated on content Thank you!
- 22. Further reading: About ROER4D: • Hodgkinson-Williams, C. (2013). Research on Open Educational Resources for Development in Post-secondary Education in the Global South (ROER4D) - Scoping Document. Available online: http://hdl.handle.net/11427/8430 [Last accessed 10 February 2015]. • Hodgkinson-Williams, C. and Cartmill, T. (2014). Research on Open Educational Resources for Development in the Global South: 1st Technical Report 23 June 2013 to 27 August 2014. Available online: http://hdl.handle.net/11427/9695 [Last accessed 11 February 2015]. About evaluation: • Patton, M. Q. (2008). Utilization-focused evaluation. California: Sage Publications Inc. • Ramirez, R. and Brodhead, D. (2013). Utilization Focused Evaluation: A Primer for Evaluators. Penang: Southbound. • Scriven, M. (1991). Evaluation Thesaurus. California: Sage Publications Inc.