Phases of the 3PD Approach: Discovering Instructional Design 15

The intent of the Three-Phase Development (3PD) Model was to provide a new focus for the end-to-end learning content and evaluation development process, especially for Web-based teaching and learning. As discussed yesterday, a central tenet of 3PD was that course creation could not be viewed as a short-term development process, but rather as a long-term collaborative process which would

generate and evolve into focused communities of practice with shared understanding and a philosophy of continuous improvement

(Sims & Jones, 2003 , p. 18)

Three-Phase Design is configured to elicit learning content through a three-step process of developing functionality, evaluating, elaborating, and enhancing and maintaining materials, rather than the more traditional systems approach of analyze, design, develop, implement, evaluate. The approach also aims to align the "three essential competency sets" for courseware development - course design, subject matter exposition, and content production - in an integrated fashion rather than as a set of uncoordinated activities.

Rather than process driving development, it is the context of the educational components which determine the members of development teams in a targeted and effective manner. Ideally, these teams would remain for the duration of the project, potentially over a number of semesters.

(Sims, 2008 p.3)

To achieve this goal, 3PD specifies a series of "baselines" (2008 p.4) that align with implementation iterations – the first focusing on building functional and essential course components, the second on enhancement or interactivity, and the third to ongoing maintenance of the courseware (see Figure 1). These three phases of development integrate systems-based methodological approaches to content development, scaffolding of contributors, and quality assurance.

Figure 1: Three-Phase Design & Scaffolding
(after Sims & Jones, 2003)
[Click to enlarge]

According to Sims and Jones, Phase 1 is a predelivery mode, which involves the gathering and preparation of web-based teaching resources, learning channel, specifying assessment-based outcomes, preferred teaching modality, and learning/learner activities designed to attain the prescribed outcomes. Three-phase Design enables a teacher with minimal experience in Web-based training and learning environments to access "functional learning structures" (Sims, 2008 p.4) and in-team expertise from the Developers and the Educational Designers in the group.

Phase 2 (Enhancement) is the delivery stage in 3PD. The asynchronisity of digital network supported learning, and the object-oriented nature of e-learning is such that modifications can be implemented in courseware on an ongoing basis (for example to take account of new learning materials or new knowledge) to enhance the student's ability to achieve the learning objectives. The second phase can be in this way to take place during course delivery, with Kirkpatrick Level 1 and Level 2-style feedback from both instructors and learners being used to modify and/or enhance delivery either continuously. or in a staged manner. For example modifications may be implemented before the beginning of each new semester, based upon the reactions of learners who took the course during the previous semester.

The third stage of 3PD - the maintenance phase - occurs during the "main sequence" (to borrow a term form astronomy) of the course lifecycle. In time, a course will attain a stable state where the teaching strategies and learning activities are working effectively, it’s materials are up-to-date, and the course is taken by sufficient number of learners to make delivery and maintenance cost-effective for the host institution.

Sims (2008) considers that:

The implications of applying the 3PD model is that the original functional system will always be subject to change, and that development environments need to schedule resources for the life-time of that course. The continual process of gathering and incorporating evaluation data caters for the sustainability of the course.

(p.6)

Phase 3 provides an opportunity for a rigorous quality assurance process to be undertaken, and for stakeholders in the course development project to consolidate the instructional design and collaborative skills acquired during the 3PD process: ideally these skills are then applied to the development of a new learning program, where they continue to be refined, with remediation taking place as necessary.
___________

References:

Sims, R. (2006). Beyond instructional design: Making learning design a reality.Journal of Learning Design, 1(2), 1-7. Internet: Available from: http://www.jld.qut.edu.au/ Accessed 3 June 2009.

Sims, R., & Jones, D. (2002). Continuous Improvement Through Shared Understanding: Reconceptualising Instructional Design for Online Learning. Proceedings of the 2002 ascilite conference: winds of change in the sea of learning: charting the course of digital education. Internet: Available from: http://www.ascilite.org.au/conferences/auckland02/proceedings/papers/162.pdf Accessed 3 June 2009

Sims, R., & Jones, D. (2003). Where practice informs theory: Reshaping instructional design for academic communities of practice in online teaching and learning. Information Technology, Education and Society, 4(1), 3-20.

Sims, R. (2008). From three-phase to proactive learning design: Creating effective online teaching and learning environments, in J. Willis (Ed), Constructivist Instructional Design (C-ID): Foundations, Models, and Practical Examples.

Sims, R. Analysis of Three Instructional Design Models. Internet: Available from: http://www.de-research.com/PhDFinalPapers/CT_3IDModels.pdf Accessed 1 June 2009

--

Discovering Instructional Design 14: the Three-Phase Design Model

In the field of education, instructional design has traditionally been applied using established models, typically using a top-down approach, that focus on explicit definitions of audience, environment, strategies, activities and outcomes. However, when different traditions of design are considered, more creative and organic elements are emphasised, which also embrace a ‘bottom-up’ strategy.

Kays, E, & Sims, R. (2006).

The growth of e-learning has motivated educators to re-examine the theory and practical application of instructional design models to the task of courseware design, development, and delivery.

As we have seen, the ICARE model takes a 'traditional' linear systems approach to course design, but has also been used as the basis for migrating content to an online environment. As we know, migration is a directed, systematic movement of a group of objects, organisms, or people. If we step outside the ICT world for a moment, what else can we say about migration? We know that it is time-consuming, resource-intensive, and risky, with no guarantee that the migrant will reach their destination, and if they do, that they will survive and thrive in their new environment. Returning to the world of digital technology, Gregory Muira (2007) asserts that there are a number of other disadvantages associated with migration:

• Migration addresses the possible obsolescence of the data carrier, but does not address the fact that certain technologies which run the data may be abandoned altogether, leaving migration useless.
• It's time consuming - migration is a continual process, which must be repeated every time a media reaches obsolescence, for all data objects stored on a certain media.
• It's costly - an organization must purchase additional data storage media at each a migration.

Migration can be confused and messy

As a result of these disadvantages, technology professionals have begun to develop alternatives to migration, such as emulation. However, emulation per se is not an effective solution for educational technologists - we know the issues and complications inherent in attempting to replicate the classroom or instructor-led environment in an online milieu. So whither instructional design in the Digital Age?

One well-developed approach is Sims and Jones’ Three-Phase Design (3PD) Model. Three-Phase Design is

an enhancement to the traditional design process [that] focuses on the creation of functional course delivery components,with evaluation and improvement activities integrated with scaffolding (support) for the teacher and learners to provide a dynamic teaching and learning environment in which resources and strategies can be developed or modified during the actual delivery stage.

(Sims & Jones, 2002 p.8).

As illustrated in Figure 1, the process is supported by a "team" (p.8) consisting of an academic (A), a developer (D), and an educational designer (ED) who all contribute to each part of a learning program's iterative progression through the model. The authors' consider that the "ultimate goal" of the model is to disintermediate the Developer and the Educational Designer, enabling the Academic to function as an independent Developer and Educational Designer over time.

Figure 1. The 3PD model including ADDIE components (after Sims & Jones, 2002)

I would assert that in its stated goal, and to a certain extent in it's execution, that the 3PD model is a direct-line antecedent of the Rapid E-Learning approach to courseware design, development and delivery. It's important to point out thought that Sims and Jones themselves view online course creation

not ...as a short-term development process, but rather as a long-term collaborative process which would “generate and evolve into focused communities of practice with shared understanding and a philosophy of continuous improvement” the value of 3PD would be realised through a three-step process of develop functionality, evaluate/elaborate/enhance and maintain rather than the more traditional sequence of design, develop, implement, evaluate.

(2003, p. 18)

Three-Phase Design also integrates the three "essential competency sets for unit or course development" (Sims, in press) - design, subject matter exposition, and production, in a cohesive rather than disparate fashion. Here, development is not driven by a an overarching and inflexible process, but rather it is the context of the learning materials which determines the development in a targeted and effective manner. The approach is based upon the assumption that learning takes place in an online an collaborative environment. Sims and Jones state that 3PD "proposes four critical factors:"

1. Instructional design must align with institutional expectation, contemporary pedagogies, and available resources and skills
2. Skills building is facilitated through the scaffolding process to enable those less proficient in design and development to develop the appropriate competencies.
3. A team-based approach is used to develop communication and collaboration among group members. Sims and Jones (2002) point out that the growth in social media reinforces the importance of this factor.
4. Scaffolded support is incorporated into content design-time to enable instructors and staff to confront new and learning paradigms.

Next time: A closer look at 3PD's phases

___________

References:

Kays, E, & Sims, R. (2006). Reinventing and reinvigorating instructional design:A theory for emergent learning. Proceedings of the 23rd annual ascilite conference: Who’s learning? Whose technology? Internet: Available from: http://www.ascilite.org.au/conferences/sydney06/proceeding/pdf_papers/p197.pdf Accessed 3 June 2009

Muira, G. (2007). Pushing the Boundaries of Traditional Heritage Policy: maintaining long-term access to multimedia content. IFLA Journal 33: 323-326.

Sims, R. (2006). Beyond instructional design: Making learning design a reality.Journal of Learning Design, 1(2), 1-7. Internet: Available from: http://www.jld.qut.edu.au/ Accessed 3 June 2009.

Sims, R., & Jones, D. (2002). Continuous Improvement Through Shared Understanding: Reconceptualising Instructional Design for Online Learning. Proceedings of the 2002 ascilite conference: winds of change in the sea of learning: charting the course of digital education. Internet: Available from: http://www.ascilite.org.au/conferences/auckland02/proceedings/papers/162.pdf Accessed 3 June 2009

Sims, R., & Jones, D. (2003). Where practice informs theory: Reshaping instructional design for academic communities of practice in online teaching and learning. Information Technology, Education and Society, 4(1), 3-20.

Sims, R. (in press). From three-phase to proactive learning design: Creating effective online teaching and learning environments, in J. Willis (Ed), Constructivist Instructional Design (C-ID): Foundations, Models, and Practical Examples.

Sims, R. Analysis of Three Instructional Design Models. Internet: Available from: http://www.de-research.com/PhDFinalPapers/CT_3IDModels.pdf Accessed 1 June 2009

--

Discovering Instructional Design 13: ICARE Model – Middlesex University’s experience

We have now reached the “precipice of the crossroads” as Sopranos’ character Little Carmine said.

So far, all the models I've talked about in this E-Learning Curve Blog series have described a linear process in the design of learning events, for the purpose of delivering the content in a classroom environment.

This approach has been variously categorized as "phased," "stepped," and "unit-based." It assumes a pretty sophisticated knowledge of pedagogy, as well as a degree of experience in instructional design. In the classroom context, instructors, trainers, and lecturers do not necessarily have an instructional design background, and typically do not follow a linear method of planning and learning event decision-making. From a development perspective, we find educators undertaking instructional design who rarely work according to theory. More often than not, the design of instruction is not a linear activity - an idea developed in the Morrison-Ross-Kemp Model, where the application of an instructional design model could be initiated at any point during the design process.

Experience tells learning professionals that an idea of what needs to be developed already exists, and this often provides the basis of the design. The instructional designer then reviews the design to ensure all components of the "model" have been accounted for, and refines the design if necessary to "check all the boxes." The process is then written up as if it occurred in a linear and systematic fashion. I would assert that the substantial majority of educators under time-pressure and with high demands on their skills would admit that they have done this - if not publicly, then certainly in the wee small hours, in the darkness of their souls.

As I discussed yesterday, the ICARE approach is a distillation of the Dick and Carey Model for instructional design. Today, we'll look at how Middlesex University in the UK modified the ICARE model for their Global Campus initiative.

The higher education must undergo a paradigm shift from an environment and culture shaped by the brick and mortar facilities and faculty-centred activities, to an environment defined by “learner-centred” processes shaped by information technology and ubiquitous asynchronous access to subject content material, learner support activities and technology-literate resource personnel.

(Dubois,1998)

In response to the Jacques Dubois’ challenge, Middlesex University established a “Global Campus”. The Global Campus offers distance learning education in countries including Egypt, Hong Kong, Singapore and the UK.

In the conventional mode, the [Middlesex] University staff interacts with students in lectures, seminars, and labs. Initially, the elimination of all staff interaction with students was considered; however, it was felt that staff provide necessary support for the student including instructional support, and study direction. Additionally, meeting in seminars allowed the students to meet with each other leading to more peer interaction which is considered desirable. In this every student would be required to manage their own learning, but be able to interact with both tutors and other students.

(Mojab & Huyck, 2001)

In the Global Campus framework, courseware is distributed asynchronously via the on-line Global Campus WebCT LMS (now owned by BlackBoard), or via a CD-ROM which is issued to all the students registered on the distance learning program. The Global Campus’ instructional framework was developed at Middlesex University based on the ICARE Model (see Figure 1).

Figure 1. Original ICARE Model
[Click to enlarge]

The Middlesex ICARE framework (MDX-ICARE) departs from the base model in both pedagogy and implementation. Pedagogically, the second phase ("Connect") has been changed to "Content." According to Mojab and Huyck it was changed for "clarity." So, introduction is followed by the content - in which the main lesson is presented with intermittent activities to engage the student and make the learning an active rather than a passive process. The activities are often linked to the Apply component of the unit (see Figure 2).

Figure 2 ICARE-MDX Framework
[Click to enlarge]

The Global Campus development team implemented a seven-phase process to convert their materials for delivery over digitally mediated channels.

1. A 20-unit breakdown of the module is provided by the module leader; this includes a brief introduction and list of objectives for each unit. This was considered an important and difficult stage in the development of a new module, with little history of previous runs.
2. The plan is considered and approved by the Global Campus academic board chaired by the Distance Learning Curriculum Leader.
3. The introduction unit (unit 01) is written by the module leader to provide the authoring team with an overall picture of the module and its objectives.
4. The remaining units are divided between a team of authors with a schedule of delivery.
5. Authored units are sent to the internal reviewers and back to the authors for final consideration and amendment.
6. The final version is fully piloted to a small cohort of students.
7. If necessary, further revision is made and the final product is sent to the host campus for implementation.

According to the authors, the Global Campus material was also used for the conventional students. These students had a hard copy of the material in the form of a 500-page handbook (requested by the students), a CD version of the web site, and access to the WebCT site. According to Mojab and Huyck "this material was very beneficial for the conventional students. It provided a great deal of information, and pointers to other information for these students.

The authors continue:

Having said that, initially, some students expressed their unhappiness with information overload. The above material along with the conventional lectures and faculty contact was too much for the students. Some felt that they had to read all of the work and do all of the activities. It had to be pointed out that they were responsible for particular issues in each unit that were specified in the introduction. The unit included material to expand and explore these issues. They were relieved. Indeed the feedback from the students has been very positive.

(2001 p.12)

As a result of course conversion to a Global Campus module, both distance and conventional students had access to a comprehensive set of lecture notes produced for the online modules, meaning that a significantly greater amount of lecture time can be spent on interacting with the students rather than on presenting the lecture material.

In this context,

Not having to present the material, which is provided in print, presents a real opportunity to introduce to and extensively discuss with students additional but related topics which would greatly enhance their understanding of the subject. The absence of this opportunity in the distance learning mode suggests that the conventional students appear to be at advantage.

(Mojab & Huyck, 2001 p.12)

The Global Campus team concluded that the process of refactoring their courseware for e-learning had the following results:

• Higher quality content for students
• Inter-discipline and cross-discipline academic collaboration
• Opportunity for research and publication
• Administration and management infrastructure operational efficiency
• More effective use of time
• Student expectations increased

According to Michael Moore’s (1993) theory of ‘transactional distance,’ the greater distance there is between the learner and the instructor, the more responsibility the learner has to take in the instructional process.

This theory suggests that distance learning models should provide a better model for developing autonomy in the learner than the conventional model of learning. The Middlesex University team recommend that a suitable qualitative and quantitative survey of transactional distance theory could be carried out to measure and compare the development of autonomy in distant and near learner by testing for the following four conditions:

• Students are actively involved in all decisions made about their learning.
• Students are able to learn without the continuous involvement of teachers.
• Students are active rather than passive.
• Students are able to take responsibility about their own learning.

Next: We move into the domain of Instructional Design in the context of e-learning proper.

__________

References:

Dubois, J. (1998). Distance Learning: A transformation Model for Higher Education. 4th International Conference on Technology Supported Learning, Berlin, Germany.

Mojab, D. & Huyck, C. (2001). The Global Campus at Middlesex University: A Model for E-Learning. [Internet] Available from: http://www.cwa.mdx.ac.uk/chris/draft6.doc Retrieved 3 June 2009

Moore, M. (1993), Theory of transactional distance. In: Desmond Keegan (Ed.): Theoretical principles of distance education. London, New York: Routledge 1993, p.22-38.

--

Discovering Instructional Design 12: the ICARE Model

In the broadest sense of the term, 'instructional design' has been described as

…an emerging profession, (2) focused on establishing and maintaining efficient and effective human performance, (3) guided by a model of human performance, (4) carried out systematically, (5) based on open systems theory, and (6) oriented to finding and applying the most cost-effective solutions to human performance problems and discovering
quantum leaps in productivity improvement through human ingenuity.

(Smith & Tillman, 2004 p.1)

More prosaically Gustafson & Branch consider instructional design (ID) as

a system of procedures for developing education and training programs in a consistent and reliable fashion. Instructional design is a complex process that is creative, active, and iterative.

(What is Instructional Design? 2002, p. 17)

The latter assert that instructional design is a complex systematic process with the following characteristics;

• interdependent - no elements can be separated from the system
• synergistic - all the elements can achieve more than the individual elements alone
• dynamic - systems can adjust to changing conditions in environments
• cybernetic - elements communicate among them efficiently

According to Gustafson and Branch, adhering to a instructional systems design process and can make instruction more effective and relevant to learners.

With these parameters in place, let's take a look at the ICARE approach to designing instruction. Based on the venerable Dick and Carey Model and pioneered by San Diego State University in 1997, the model has found a place in the higher education sector. According to Vincent Salyers (2006) ICARE has potential "as one possible means for structuring and organizing course content." As we'll see in my next blog post, the Centre for Learning Development at Middlesex University have adapted the ICARE framework, designed templates with built-in guidelines for use by academics with little experience in instructional design, and extended the model as the basic pedagogy for their ‘Global Campus’ instructional framework for distance education (Mojab & Huyck, 2001).

According to the ICARE Model's main proponents Hoffman and Ritchie (1998), the model is distilled from basic instructional design practice (see Table 1), and adapting various systems to what seemed to be particularly useful components for e-learning course design and development.

Table 1. The ICARE Model

Phase	Description
Introduction	This phase consists of the introduction to the unit of instruction including: Context Objectives Prerequisites Required study time Equipment required Essential reading materials
Connect or Content (MDX interation)	Almost all content will reside in this section
Apply All activities	Exercise, thinking questions, etc are implemented in this phase
Reflect	This phase provides an opportunity for learners to reflect on their acquired knowledge and articulate their experience. This section may include: topics for discussion, a learning journal/log, a self test, formative and summative assessment
Extend	An amalgamation of all the previous phases which offers materials and learning opportunities which can be remedial, supplemental, or advanced, depending on learner performance

In this context for example, when refactoring course content into online modules (what the authors term "distance learning units") a conventional 20-credit module is deconstructed into twenty units worth nine hours of study each. The model has the following five distinctive but interrelated components that are applied to individual lesson/lecture 'unit.'

More…

___________

References:

Gustafson, K. L., & Branch, R. M. (2002). What is instructional design? In: R.A. Reiser & J. A. Dempsey (Eds.), Trends and issues in instructional design and technology (pp. 16-25). Saddle River, NJ: Merrill/Prentice-Hall.

Hoffman, B., & Ritchie, D.C (1998). (2005). Teaching and learning online: Tools, templates, and training. In: J. Willis, D. Willis, & J. Price (Eds.), Technology and teacher education annual - 1998. Charlottesville, VA: Association for Advancement of Computing in Education.

Mojab, D. & Huyck, C. (2001). The Global Campus at Middlesex University: A Model for E-Learning. [Internet] Available from: http://www.cwa.mdx.ac.uk/chris/draft6.doc Retrieved 3 June 2009

Salyers, V. (2006, July). Using the ICARE Format for Structuring Online Courses. Impact 2006, WebCT, 8th Annual Users Conference; San Antonio, TX.

Smith, P.L., & Tillman, J.R. (2004) Instructional Design (3rd Ed). Hoboken, NJ: Wiley Books.

--

Discovering Instructional Design 10: the Dick and Carey Model

According to the Encyclopedia of Distance Learning (Boettcher et al, 2005)

Design models can be defined as the visual representations of an instructional design process, displaying the main phases and relationships. Each phase has an outcomes that feeds the subsequent phase. Currently, there are more than one hundred different ISD models.

(p.164)

The most commonly known models include:

• The Dick and Carey Model
• The Kemp Model
• The iCARE Model
• The ASSURE Model

I will be looking at these models over the next few posts, beginning today with the Dick and Carey Model.

In discussing the Dick and Carey Model, Boettcher et al describe Walter Dick and Lou Carey as the "torchbearers" (p.164) of the Systems Approach, as outline in the former's "authoritative" text The Systematic Design of Instruction.

The Encyclopedia asserts that the Dick and Carey model is "very popular" in current ISD development. This model is based upon the Behaviorist assumption that there is a predictable link between a stimulus and the response that it elicits in the learner. It describes the phases of instruction (see Figure 1) as an iterative process that starts by identifying instructional goals and concludes with evaluation.

The Dick and Carey method is to break instruction down into smaller components or granular objectives that collectively constitute the competency to be acquired. Next, the stimulus and strategy for its presentation that builds each sub-skill are defined. The steps in the model are shown in the Figure 1, and described briefly below.

Figure 1 The Dick and Carey Model
[Click to enlarge]

Phases in the Dick and Carey method:

1. Determine the instructional goal
2. Analyze the instructional goal
3. Analyze the learners and contexts
4. Write performance objectives
5. Develop assessment instruments
6. Develop instructional strategy
7. Design and conduct formative evaluation
8. Revise instruction
9. Undertake summative evaluation

Phases 1 and 2 align very closely with Robert Mager’s approach (1988). He suggests a five-step process in the translation of a vague instructional goal to a set of rigorously defined desired performances, which the author elucidates in a very practical manner:

1. Write down the goal using whatever abstract terms express the intent and be sure the statement is written in terms of outcomes rather than process. For example, “Have a favorable attitude to…” rather than “Develop a favorable attitude to…”.
   
2. Think about achieving the goal in terms of people performance. What would people have to do or say or stop doing and saying before you would be willing to say that they represent the goal? List as many performances as you can think of.
   
3. Sort the list. Many of the items will be ‘fuzzy’ and not describe anything about performance per se. As SMEs to undertake steps 1 and 2 again. Continue until there is a list of performances that collectively represent the goal – until it can be said that if someone did these things and did not do these other things that would represent the goal.
   
4. Expand the words and phrases on the list into complete sentences that tell when or how often the performance is expected to occur. This will help to establish limits around the expected performance. It will enable the instructional designer to say “how much" performance is satisfactory to undertake the task successfully. For example, a goal analysis on security consciousness might include the item ‘no unattended documents’. When expanded into a complete sentence it may read “Employee always locks sensitive documents in safe before leaving room.”
   
5. Test for completeness. Review the performances on your list and ask:

If someone did these things would I be willing to say that he or she is _____________”.

(Mager, R. Making Instruction Work. 1988, pp 45-46)

If this is the case, the goal analysis is finished. If not, return to step 2 and add the missing performances.
___________

References:

Boettcher, J. V., Justice, L., Schenk. K., Rogers P. L., & Berg, G. A (2005). Encyclopedia of Distance Learning. Idea Group Reference

Carey, L. and Dick, W. (2004). The Systematic Design of Instruction (6th Ed.). Allyn & Bacon.

Mager, R. (1988). Making Instruction Work. Belmont , CA: Lake Publishing Co.

Discovering Instructional Design 9: Implementation and Improvement

Over the last few weeks we have been discovering Instructional Design (ID) - the practice of maximizing the effectiveness, efficiency and accessibility of instruction and other learning experiences. The ID process can be said to have a number of steps:

1. Determine the current state and needs of the learner
2. Define the end goal of instruction
3. Develop a learning intervention to assist in the acquisition of new skills, knowledge or expertise.

Once these stages have been undertaken, the learning materials need to implemented and improved, or evaluated, and that’s what we’re going to investigate today.

IMPLEMENTATION

The implementation phase of the instructional design process is when the instruction as designed is actually delivered. Materials are collected, the environment is arranged, and the course is prepared for delivery.

For some courses, (particularly in e-learning environments) a pre-test is made available. This can determine whether the learner can already perform some of the objectives (or larger units of instruction). If that is the case, certain modules or part of the curriculum can be skipped. A pre-test can also determine whether a learner is undertakes an individualized learning path - a basic form of adaptive learning design.

When designing a new course or learning program, the design and materials should be tested during a pilot course. The pilot enables the instructional designer and trainers to have an opportunity to review and revise the course before it is fully implemented. The instructor who actually delivers the training must be knowledgeable and skilled in the competency-based training approach used by the instructional designer when designing the course.

IMPROVEMENT PHASE

A common useful way and comprehensive way of looking at improvement is in terms of the Kirkpatrick model:

Donald L Kirkpatrick first published his ideas on evaluating learning in 1959 in a series of articles in the US Training and Development Journal. The articles were subsequently included in his book Evaluating Training Programs (originally published in 1975; I have the 2006 edition).

In this text he outlined and further developed his theories on evaluating culminating in the Four-Level Model, arguably the most widely used and popular approach for the evaluation of training and learning. Kirkpatrick's Four-level model is now considered an industry standard across the HR and training communities (see Table 1).

Table 1 Kirkpatrick's Four-level Model

Kirkpatrick’s Model	Learning Effect
Level 1: Reactions.	Evaluate participants' satisfaction with the learning intervention.
Level 2: Learning and Level 3: Behavior.	What do participants know they didn't know before? How are they using knowledge in their jobs? What is the learning and performance effect of the intervention?
Level 4: Organization-level benefits.	Has the development of higher levels of domain knowledge improved organizational productivity?

According to the model, evaluation should always begin with Level One, and as time and budget allows, should move sequentially through Levels Two, Three, and Four. Information from each prior level serves as a base for the next level's evaluation. Each successive level represents a more precise measure of the effectiveness of the training program, however, each level also requires a more rigorous and time-consuming analysis.

Kirkpatrick's model aside, the course materials, objectives, delivery, test items, audience profile – all of the instructional components in fact – need to be evaluated. Assessing these elements regularly is especially important for repeating courses or asynchronous courseware. As an example, if substantial majority (70% or 80% of the learners) fail a criterion test item, it would be reasonable to look again at the design of the related piece of instruction.

_________

References:

Kirkpatrick, D. & Kirkpatrick, P. (2006) Evaluating Training Programs. 3rd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

--