3PD Approaches to Evaluation: Discovering Instructional Design 16

We're approaching the 40th anniversary of the first moon landing. I’ve no doubt that there will be a bombard of documentaries, retrospectives, and "why aren't we there now?" features coming this July, surrounding the big day itself. This will brighten up my summer no end. Despite its Cold War beginnings, I happen to think that the Apollo-era US Manned Space Program represents the epitome of human vision and endeavor.

What has this got to do with instructional design, say you?

Well, read on...

NASA wouldn't have got to the Moon, or even to the next town, without gimbals. Not only does NASA use gimbals for orienting rocket engines, but also when designing navigational systems and instrument panels. Without gimbals, it would have been very difficult for NASA to find a way to send astronauts safely into space.

A gimbal is a mechanism that helps to keep an object on target: it's built into the platform's systems to correct deviations from a pre-determined goal.

On the Saturn V rocket, for example, gimbals were used to set the rocket at the correct pitch and yaw angles to safely "clear the tower" - that is, not bump into the rocket's support gantry on lift-off. Later in the flight, gimbals pitched the rocket's trajectory to align with the Earth's curve for it's journey into orbit (rockets don't go "straight up" but rather ascend in an arc until they attain the required altitude).

So what space nerd. What has this to do with instructional design, say you again, losing patience?

In my view, the task gimbals* perform space flight is similar to the role evaluation performs in instructional design.

According to Donald Clark (2009)

Evaluation is the systematic determination of merit, worth, and significance of a learning or training process by using criteria against a set of standards.The evaluation phase is ongoing throughout the ISD process. The primary purpose is to ensure that the stated goals of the learning process will actually meet a required business need. Thus, it is performed during the first four phases of the ISD process.

Indeed, we can see that this strategy is codified in Dick and Carey's approach (see Figure 1), where an ongoing review process indicated during the first six phases of the process.

Figure 1. Dick and Carey's Model
[Click to enlarge]

Formal evaluations proper are undertaken in steps 7-9 of their model:

1. Determine the instructional goal
2. Analyze the instructional goal
3. Analyze the learners and contexts
4. Write performance objectives
5. Develop assessment instruments
6. Develop instructional strategy
7. Design and conduct formative evaluation
8. Revise instruction
9. Undertake summative evaluation

Dick and Carey (2001) recommend three categories of of formative evaluations to support this process: one-to-one (or clinical) evaluation, small-group evaluation, and field evaluation, but in my view they don't suggest a mechanism for evaluation per se, as the activities they suggest are standard ethnographical research methodologies. Similarly, while they consider on-going reviews to be a component the their ID model, the research suggests that In her 1989 article Evaluation of training and development programs: A review of the literature, Marguerite Foxon describes herself as "surprised" at the "general" and "superficial" nature of the research undertaken on evaluation, and considered that what was there was "difficult to understand and apply."

She continues:

Where evaluation of programs is being undertaken it is often a 'seat of the pants' approach and very limited in its scope. ...trainers often revert to checking in the only way they know - post-course reactions - to reassure themselves the training is satisfactory.

If the literature is a reflection of general practice, it can be assumed that many practitioners do not understand what the term evaluation encompasses, what its essential features are, and what purpose it should serve. ...Many practitioners regard the development and delivery of training courses as their primary concern, and evaluation something of an afterthought."

She suggests that many practitioners prefer to "remain in the dark," concerned that any actual evaluation will "confirm their [the instructional designers'] worst fears" about the educational quality of the courseware they deliver, with the result that they "choose to settle for a non-threatening survey” of Kirkpatrick Level 1-style trainee reactions.

As we have seen in our look at the Three-Phase Design (3PD, in this model evaluation is not viewed as a post-delivery activity (Sims, 2008 p.5): the nature of Web-based education is such that changes can be made immediately (that is, during Phase 2 - Evaluate, Enhance, Elaborate), as long as those changes don't affect the integrity of the learning program's objectives. The second phase can be

"conceptualised to take place during course delivery, with feedback from both teachers and learners being used to modify and/or enhance delivery.

(p5)

Sims and Jones (2003) call this process "proactive evaluation" (see Figure 2).

Figure 2 Proactive evaluation in 3PD
[Click to enlarge]

Using this approach, formative "feedbacks" occur between instructor and students during course implementation. The authors assert that this mechanism continues the dynamic collaboration between the members of the development team enhances. The second phase enables

generational changes in the course structure, with emphasis on the production (completion) of resources, and where learners can take a role of research and evaluation assistants. By developing and building effective communication paths between each of these three roles, a shared understanding of the course goals and learning outcomes can be established, thereby minimising and compromise in educational quality and effectiveness.

In my view, (as shown in Figure 3), the evaluation in this model is founded upon recursion. The enhancement process is undertaken by the actors (instructors, designers, and learners) using a strategy similar to the concept of optimal (or dynamic) programming, where complex problems are solved by breaking them down into simpler sub-problems.

Figure 3 Recursive evaluation in the 3PD Model
[Click to enlarge]

In essence, the enhancement process is repeated until the learning program is considered complete.

Even during the Maintenance Phase, the ongoing process of

gathering and incorporating evaluation data caters for the sustainability of the course.

(Sims, 2008 p.6)

Unlike the Dick and Carey and Kemp Models, 3PD supports overlapping roles, skills, and responsibilities. These contributions may well change through the lifecycle of a learning program, as the model promotes and supports the development of instructors and students' knowledge, skill and experience via the virtuous circle of ongoing collaboration and communication between the actors, and the development of working relationships. The inclusion of learners in the content development process differentiates 3PD from the other models discussed here.

More...

*(Note to hardcore design-heads: this is a metaphor†: I'm not suggesting they're literally equivalent. Go with it).

†Metaphor (n) - a figure of speech in which a word or phrase literally denoting one kind of object or idea is used in place of another to suggest a likeness or analogy between them (Merriam-Webster Online Dictionary)

___________

References:

Clark, D. (2009). Evaluation in Instructional Design. [Internet] Available from: http://www.nwlink.com/~donclark/hrd/sat6.html Accessed 12 June 2009

Foxon, M. (1989). Evaluation of training and development programs: A review of the literature. Australian Journal of Educational Technology, 5(2), 89-104. [Internet] Available from: http://www.ascilite.org.au/ajet/ajet5/foxon.html Accessed 12 June 2009

Sims, R., & Jones, D. (2003). Where practice informs theory: Reshaping instructional design for academic communities of practice in online teaching and learning. Information Technology, Education and Society, 4(1), 3-20.

Sims, R. (2008). From three-phase to proactive learning design: Creating effective online teaching and learning environments, In: J. Willis (Ed), Constructivist Instructional Design (C-ID): Foundations, Models, and Practical Examples.

--

Irish Government to tax learners with "failure fee"

The Irish Government is set to tax learners with a "failure fee" in the budget next week.

Today, a source close to the ‘Grandpa Simpson of Irish Politics’ Minister for Education Batt O'Keeffe TD (yes, that really is his name) announced that the Department intends to tax the corporate sector through a levy on incorrectly answered questions in e-learning evaluations in an effort to generate money for the public purse in response to the deepening global economic crisis.

When questioned further the government source - who did not want to be named for national security reasons - elaborated on this plan, by saying that:

The e-learning sector in Ireland is doing really well, despite the downturn. There are lots of people out there using courseware over and over again, basically for free once they pay their initial fee for the content. We are looking for a way to continue to generate revenue from people every time they use a piece of e-learning. After consulting with experts, we have developed a bold and innovative approach to do this: we are going to charge learners a fee when they answer test questions incorrectly.

The government have already cut primary teacher numbers (making the student-to-teacher ratio the highest in Western Europe), and re-introduced third-level fees for the less well-off.

We are working with a number of prominent Irish e-learning providers to mandate that an extra piece of code called Reactive User Scoring Evaluations (RUSE) will be embedded in all e-learning content that will require the learner - or their company - to sign up with PayPal and the Department of Revenue so that they can be automatically charged a "failure fee" of one cent every time they answer a question wrong.

We are also collaborating with organizations like ADL/SCORM to add a new field to IMS manifest files to help us track this levy, so that when a user submits their test, the data is sent to our database. There seems to be a concern at EU level that this affects Irish citizens' privacy and human rights, but in these tough times we have to put aside such selfish ideas and think of the greater good of the nation.

The source continued:

This has been incorrectly called a stealth tax. It's not. We want everyone to know about it. We see this as an opportunity to monetize a previously untouched area of education, and a great way to motivate people to study even harder and answer correctly on training courses.

Frankly, if people are too stupid to get the answers right, they're too thick to be in a job. We think that we're doing companies a favor by letting them know how many eejits they have in their midst, which is keeping people with real skills out of employment. Based on current worker fail rates, this scheme will also net us about €6 million in the first year, which is a real sweet deal.

An opposition party spokesman stated that

this is typical of the gombeen-man ignorance in this government: everyone knows that this will fall flat on its face, much like the e-voting debacle a few years ago. I don't know, sometimes I just despair of this crowd of wasters. They are truly unfit to hold public office in a modern democracy.

Industry insiders speculate that a move by the government to implement this program, called the Finance Act for Knowledge and Education - or FAKE - will lead to an increase in ‘unsupervised education’ and dangerous 'free-form learning' taking place 'off the grid.'

One particularly gloomy respondent considered that Ireland would see a resurgence of the 'hedge-schools' which emerged during the Penal Laws in the 19th Century. What's worse, it’s the kind of "innovation" that governments internationally will see as a legitimate source of tax revenue in tough times ahead.

If you want to help me stop this disaster before it’s too late, please e-mail the Secretary of the Department of Education, a Ms. Avril O'Fol at [email protected] with the e-mail header "Think of the children - stop the madness."

It’s important that you do this today, the first of April 2009, or it will be too late.

Thank you.

______________________

Notes:

A Gombeen Man is a pejorative Hiberno-English term used in Ireland for a shady, small-time "wheeler-dealer" or businessman who is always looking to make a quick profit, often at someone else's expense or through the acceptance of bribes.

An eejit is similarly an offensive term used in Ireland that deliberately insults somebody's intelligence or foresight. An idiot.

--