Evaluating Distance Education (Or ICT) Programs, Clayton R. Wright
Uploaded by
crwrEvaluating Distance Education (Or ICT) Programs, Clayton R. Wright
Uploaded by
crwrEvaluating Distance Education Programs
Items for Consideration
Clayton R. Wright, PhD
crwr77[@]gmail.com
Purpose: To assess whether a distance education or training program is accessible,
appropriate, effective, affordable, and accepted. This series of questions can be used to
evaluate distance education programs. If you substitute distance education with the term
“information and communication technologies (ICTs)”, you could use the questions to guide ICT
implementation and/or evaluation.
In order to complete this assessment, we need to determine answers to these basic questions:
What rationale was established for the program?
What has the program achieved? What worked and did not work?
Are the achievements congruent with the rationale?
What changes should be made to improve the program? What recommendations are
feasible given the current environment? What conditions need to change before other
recommendations can be implemented?
Institutional, Ministry, or Business Priorities and Policy Directions
Rationale for the program and its fit with current provincial, national, and/or business
priorities
Impact the program has or will have on local communities or business and/or the
province, or nation
Fit with institutional or business goals, vision, and policies, e.g., openness, accessibility,
equality, and internationalization or global reach
Existing opportunities for individuals to upgrade themselves and/or obtain accreditation
in the community, business, province, or country. What is the competition?
Rationale for selecting a distance delivery mode for this course or program. Is this
rationale congruent with the mission of the institution or business?
Cost effectiveness of the distance learning program
Commitment of the institution or business to the program
Concerns the institution or business may have with the current program. What are the
sources of this concern? Are the sources reliable? Are the concerns widespread? What
indications are present that the concerns are valid?
Extent to which the institution, at all the appropriate levels, is committed to change
Administrative Structures
Centralized vs. decentralized structures, pros and cons for the given situation
Administrative structure of the program and the interconnections among academic and
service departments. Are the linkages set out in policy or are they primarily dependent on
the willingness of those in the linked positions to collaborate?
Administrative capabilities regarding planning, communication, support, and monitoring.
Do they have the expertise and resources required to conduct their administrative work
effectively?
Financial capabilities of the administration. Are adequate financial resources available to
administer the distance education program and do administrators have control over the
resources that they require?
Willingness of administration to operate in a transparent manner; to listen to and
respond to faculty, staff, and student concerns, especially the concerns raised by students
in rural areas; to provide needed resources; and to adjust to changing circumstances
Capability of the administration to make decisions and to act expediently
Presence of job descriptions for each type of personnel involved in the administration,
development, delivery, and support of the program. Do the job descriptions contain the
mission of the institution or business, tasks to be performed, the percentage of time to be
spent on each task, the level to which each task is to be performed, the number and type
of positions that report to this position, how individuals will be evaluated, the immediate
supervisor, and the conditions under which an individual may be reprimanded or
dismissed?
Collaborative initiatives administration has fostered in an effort to involve various
stakeholders and to use limited resources wisely
The accreditation procedures that are accepted by students and the community. These
may involve provincial, national, or international bodies.
Faculty, full-time vs. part-time/contract personnel and their connection with the academic
department or business unit and service departments
Student recruitment process. Emphasis placed on how it is administered to attract those
in rural areas or small centres of a company’s operation and to provide students with
accurate information that they can use to make a decision about their educational or
training pursuits.
Registration procedures that are accessible, simple, and flexible to meet the needs of
distance or independent learners
Procedures employed to give recognition for prior learning. An indication that potential
students have been informed of the existence of a prior learning program.
Fee payment methods and scheduling that are accessible, flexible, and affordable
Library and learning resources – accessibility, variety, depth, and currency
Counselling and guidance – accessibility, variety of services including job placement
Help desk services including availability and type of services offered; training for help
desk staff; the existence of problem-solving rubrics
Possible re-configurations of the existing administrative structure in order to gain an
increase in effectiveness and efficiencies
Learner, Graduate, and Non-Graduate Profiles
Profile of the learners including past, current, and future students. Particularly, their age,
gender, current educational level, where they live, and competencies or current
employment or position within the company. The primary focus is to gather student data
that will enable administrators to track trends and predict future possibilities rather than to
focus on one individual.
Completeness of the information program personnel have regarding the attributes and
circumstances of the learners and the availability of this information
Perceptions students have about the current program – both positive and negative
Challenges learners face with the existing program, such as accessibility and the feeling
of isolation
How the challenges faced by learners can be addressed. If technology is employed as
part of the solution, will the learners have the knowledge/experience and technical
competencies to use it effectively? Will the learners have access to the technology?
An indication of whether the graduates would recommend this program to their
colleagues
Privacy policies and care taken to limit access to sensitive information.
Employers of Program Graduates
Profile of those who hire graduates, e.g., position, type of institution or company, size of
company
The attributes employers are seeking in graduates
Shortcomings of the existing program as perceived by those who hire graduates
The induction or training programs employers must implement in order to address the
shortcomings of the program
Suggestions regarding how the existing program could be improved by those who hire
graduates
An indication of whether the employers will continue to hire program graduates
Program Design and Development
Structure of the academic program and its congruency with local or national needs as
well as its flexibility to achieve learning outcomes
Clarity of the learning objectives and learning outcome statements as well as whether
they are suitable for the intended audience and the content to be studied
A qualitative determination of whether the content of the program meets the needs of the
learners and those who hire graduates
A quality assessment of the learning materials to ensure that the objectives are
congruent with the instructional/learning strategies and assessment procedures and that
they engage the learners
Contextualization of the material to ensure that it meets the needs of all learners
particularly those in rural areas or those who live in foreign countries and may have
different life experiences than those who live in urban areas or in one nation
Packaging of the materials for delivery to a rural or international audience
Use of media or technology, particularly how it facilitates learning and supports
interactivity, and its appropriateness for use in a particular social, economic, and
geographical context
Adherence to intellectual property laws and the treatment of the work of others
The assessment component of the learning program including its congruency with the
learning objectives and the number and feasibility of formative and summative
assessments
Quality measures that are used in course design and development including the review
process that ensures that the materials are accurate and current, e.g., the use of editors
and external content reviewers
Suggestions for improving the quality of the learning materials in order to address the
needs of the students and ensure the stated outcomes of the program are met
Program Delivery
Program delivery process and procedures including the quality assurance systems that
are in place prior to the distribution of materials
Delivery mode of the materials and its appropriateness for the audience and congruency
with the instructional intent
Record keeping procedures that track the distribution of resources and the progress of
the learners
Distribution of materials in a timely manner and the arrival of the resources in a useable
form at its destination
Emphasis program administrators place on learning support
Role and effectiveness of the learning or study centres and their staff
Distribution of learning centres and their accessibility to learners
Availability of learning resources
Extent and quality of learning support including communication options available to
students who need to contact faculty, tutors, counsellors, registration personnel, library
resource staff, help desk staff, and their peers
The identification of students who fall behind on their studies and the timely and effective
support that is provided to these students
Turnaround time for student feedback
The type of feedback provided to learners. Is it constructive, positive, and nurturing? Is
detailed feedback provided?
Administration of assessments with due regard to security and timeliness
Human Resources
An assessment as to whether current staff are experienced and qualified for their
positions
Hiring procedures that are fair and transparent
Faculty and tutor issues regarding workload, professional development, and support
received from administration
Recognition and reward systems available to faculty, tutors, and staff
Recruitment and training of faculty, tutors, and staff
The support provided to faculty, tutors, and staff who work in rural areas or overseas
Quality Assurance
Quality assurance methods or systems that are in place and their relationship to future
planning
Mechanisms used to collect information from various stakeholders and to provide
feedback to stakeholders
Visibility of the quality assurance system and its associated criteria or standards
Faculty and staff buy-in to quality
Affiliation and accreditation issues as they relate to quality assurance
Improving the existing quality assurance methods with particular emphasis on the
collection, analysis, and use of data to improve operations especially in rural areas and/or
overseas
Overall Recommendations
Is the program accessible and appropriate for the intended learners?
Does the program meet its overall objectives? Is it effective?
Is the current program affordable and sustainable?
Does the program have credibility with the learners and prospective employers?
What, if anything, needs to be done to improve the current administration, development,
delivery, and assessment of the program?
Are there suitable benchmarks which could be employed?
