2013; 35: e867–e898

WEB PAPER

Comparative effectiveness of instructional

design features in simulation-based education:
Systematic review and meta-analysis
DAVID A. COOK1,2, STANLEY J. HAMSTRA3, RYAN BRYDGES4, BENJAMIN ZENDEJAS2,
JASON H. SZOSTEK2, AMY T. WANG2, PATRICIA J. ERWIN2 & ROSE HATALA5
1
Mayo Medical School, USA, 2Mayo Clinic College of Medicine, USA, 3University of Ottawa, Canada, 4University of Toronto,
Canada, 5University of British Columbia, Canada

Abstract
Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13

Background: Although technology-enhanced simulation is increasingly used in health professions education, features of effective
simulation-based instructional design remain uncertain.
Aims: Evaluate the effectiveness of instructional design features through a systematic review of studies comparing different
simulation-based interventions.
Methods: We systematically searched MEDLINE, EMBASE, CINAHL, ERIC, PsycINFO, Scopus, key journals, and previous review
bibliographies through May 2011. We included original research studies that compared one simulation intervention with another
and involved health professions learners. Working in duplicate, we evaluated study quality and abstracted information on learners,
outcomes, and instructional design features. We pooled results using random effects meta-analysis.
Results: From a pool of 10 903 articles we identified 289 eligible studies enrolling 18 971 trainees, including 208 randomized trials.
Inconsistency was usually large (I 2 4 50%). For skills outcomes, pooled effect sizes ( positive numbers favoring the instructional
For personal use only.

design feature) were 0.68 for range of difficulty (20 studies; p 5 0.001), 0.68 for repetitive practice (7 studies; p ¼ 0.06), 0.66 for
distributed practice (6 studies; p ¼ 0.03), 0.65 for interactivity (89 studies; p 5 0.001), 0.62 for multiple learning strategies
(70 studies; p 5 0.001), 0.52 for individualized learning (59 studies; p 5 0.001), 0.45 for mastery learning (3 studies; p ¼ 0.57), 0.44
for feedback (80 studies; p 5 0.001), 0.34 for longer time (23 studies; p ¼ 0.005), 0.20 for clinical variation (16 studies; p ¼ 0.24),
and 0.22 for group training (8 studies; p ¼ 0.09).
Conclusions: These results confirm quantitatively the effectiveness of several instructional design features in simulation-based
education.

Introduction
Practice points
Technology-enhanced simulation permits educators to create
. Evidence supports the following as best practices for
learner experiences that encourage learning in an environment
simulation-based education: range of difficulty, repeti-
that does not compromise patient safety. We define technol-
tive practice, distributed practice, cognitive interactivity,
ogy-enhanced simulation as an educational tool or device with
multiple learning strategies, individualized learning,
which the learner physically interacts to mimic an aspect of
mastery learning, feedback, longer time, and clinical
clinical care for the purpose of teaching or assessment.
variation.
Previous reviews have confirmed that technology-enhanced
. Future research should clarify the mechanisms of
13

simulation, in comparison with no intervention, is associated

effective simulation-based education: what works, for
with large positive effects (Cook et al. 2011; McGaghie et al.
20

whom, in what contexts?

2011). However, the relative merits of different simulation . Direct comparisons of alternate simulation-based educa-
interventions remain unknown. Since the advantages of one tion instructional designs can clarify these mechanisms.
simulator over another are context-specific (i.e. a given
simulator may be more or less effective depending on the
instructional objectives and educational context), it makes One systematic review identified 10 key features based on
sense to focus on the instructional design features that define prevalence in the literature, but did not examine the impact of
effective simulation training—the active ingredients or mech- these features on educational outcomes (Issenberg et al.,
anisms. A comprehensive synthesis of evidence would be 2005). Other reviews have found an association
timely and useful to educators. between longer training time and improved outcomes

Correspondence: David A. Cook, MD, MHPE, Division of General Internal Medicine, Mayo Clinic College of Medicine, Mayo 17, 200 First Street SW,
Rochester, MN 55905, USA. Tel: 507-266-4156; fax: 507-284-5370; email: [email protected]
ISSN 0142–159X print/ISSN 1466–187X online/13/O10867–898 ß 2013 Informa UK Ltd. e867
DOI: 10.3109/0142159X.2012.714886
 D. A. Cook et al.

(McGaghie et al. 2006) and that simulation with deliberate synthesis of evidence regarding specific instructional design
practice has consistently positive effects (McGaghie et al. features would immediately inform educational practice.
2011). In a review of simulation in comparison with no Second, a thematic summary of the comparisons made and
intervention (Cook et al. 2011), subgroup meta-analyses research questions addressed would inform future research by
provided weak evidence suggesting better outcomes when providing a list of common comparisons (indicating themes
learning activities were distributed over 41 day and when felt to be important and likely worthy of further study) and by
learners were required to demonstrate mastery of the task. highlighting evidence gaps. We sought to address both of
When comparing simulation with non-simulation instruction these needs through a systematic review.
(Cook et al. 2012), subgroup meta-analyses suggested better
outcomes when extraneous cognitive load was low, when
learners worked in groups, and when feedback and learning
Methods
time were greater.
However, such subgroup analyses represent an inefficient This review was planned, conducted, and reported in adher-
method of exploring the effectiveness of design features ence to PRISMA standards of quality for reporting meta-
because they evaluate differences between studies, and analyses (Moher et al. 2009).
Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13

between-study variation in learners, contexts, clinical topics,

and outcome measures introduces error and confounds
interpretations. Most of the subgroup interactions evaluated Questions
in these reviews (Cook et al. 2011, 2012) varied from outcome
to outcome and most were statistically non-significant. The We sought to answer: what instructional design features are
direct comparison of two instructional variations in a single associated with improved outcomes in studies directly com-
study offers a less problematic approach, as it capitalizes on paring one technology-enhanced simulation training approach
within-study (rather than between-study) design differences. with another, and what themes have been addressed in such
For example, meta-analysis of head-to-head comparisons has comparisons? To answer the first question we selected eight
been used to identify effective instructional design features in instructional design features identified in the review by
Internet-based instruction. (Cook et al. 2010b) Issenberg et al. (2005) and additional features of cognitive
For personal use only.

A comprehensive review of head-to-head comparisons of interactivity, distributing training across multiple sessions,
different simulation-based instructional interventions (i.e. group vs independent practice, and time spent learning (see
comparative effectiveness studies) would fulfill two important Box 1 for definitions). We hypothesized that outcomes would
needs in health professions education. First, a quantitative be higher with more of each feature.

Box 1. Definitions of terms*.

Participants
Health professions learner: a student, postgraduate trainee, or practitioner in a profession directly related to human or animal health, including physicians,
dentists, nurses, veterinarians, physical, occupational, and respiratory therapists, and emergency medical technicians and other first responders.

Instructional design key features

Clinical variation: Variation in the clinical context, for example multiple different patient scenarios (absent if no clinically relevant context was stated).
Cognitive interactivity: Training that promotes learners’ cognitive engagement using strategies such as multiple repetitions, feedback, task variation, or intentional
task sequencing.
Curricular integration: Incorporation of the simulation intervention as an integral part (required or formal element) of the curriculum or training program.
Distributed practice: Training spread over a period of time. For this review, we counted this as present for interventions that involved 41 day of simulation
training.
Feedback: Information on performance provided to the learner by the instructor, a peer, or a computer, either during or after the simulation activity.
Group (vs independent) practice: Training activities involving two or more learners (as compared with training alone).
Individualized learning: Training responsive to individual learner needs (i.e. tailored or adapted depending on performance).
Mastery learning: Training model in which learners must attain a clearly-defined standard of performance before qualifying or advancing to the next task.
Multiple learning strategies: The number of different instructional strategies used to facilitate learning, such as patient case, worked example, discussion,
feedback, intentional sequencing, or task variation.
Range of task difficulty: Variation in the difficulty or complexity of the task (explicitly stated).
Repetitive practice: The opportunity for more than one task performance.

Outcomes
Satisfaction. Learners’ reported satisfaction with the course.
Knowledge: Subjective (e.g. learner self-report) or objective (e.g. multiple-choice question knowledge test) assessments of factual or conceptual understanding.
Skills: Subjective (e.g. learner self-report) or objective (e.g. faculty ratings, or objective tests of clinical skills such as computer-scored technique in a virtual reality
surgery simulator, or number of masses detected when examining a breast model) assessments of learners’ ability to demonstrate a procedure or technique
in an educational setting (typically a simulation task). We further classified skills as measures of time (how long it takes a learner to complete the task), process
(e.g. global rating scales, efficiency, or minor errors), and product (successful completion of the task, evaluation of the finished product, or major errors that
would impact a real patient’s well-being). For purposes of meta-analysis we combined process and product skills into a single outcome, non-time skills.
Behaviors and patient effects: Subjective (e.g. learner or patient self-report) or objective (e.g. chart audit or faculty ratings) assessments of behaviors in practice
(such as test ordering) or effects on patients (such as medical errors). We used a classification system similar to that used for Skills, with time and process
measures being counted as behaviors (e.g. procedure time, test ordering, or interviewing technique with real patients) and products being counted as patient
effects (e.g. complications, patient discomfort, or procedure completion rates).

Note: *Some of these definitions were published as an online appendix to a previous publication (Cook et al. 2011).

e868
 Instructional design features for simulation

Study eligibility question, study hypothesis) using an inductive, iterative

approach. We abstracted information on the training level of
We included studies published in any language that investi-
learners, clinical topic, training location (simulation center or
gated use of technology-enhanced simulation to teach health
clinical environment), study design, method of group assign-
professions learners at any stage in training or practice, in
ment, outcomes, and methodological quality. We planned to
comparison with another technology-enhanced simulation
abstract information on simulation fidelity but dropped this
design or a variation or augmentation of the first, using
variable due to difficulty operationalizing it with acceptable
outcomes (Kirkpatrick 1996) of reaction (satisfaction), learning
reliability. We coded simulation features (see Box 1) of:
(knowledge or skills in a test setting), behaviors (in practice),
or effects on patients (see Box 1). Technology-enhanced . clinical variation ( present/absent; ICC, 0.46),
simulation encompasses diverse products including computer- . cognitive interactivity (high/low; ICC, 0.35),
based virtual reality simulators, high fidelity and static manne- . curriculum integration ( present/absent; ICC, 0.49),
quins, plastic models, live animals, inert animal products, and . distributed practice (training on 1 or 41 day; ICC, 0.73),
human cadavers. Because they have been the subject of recent . feedback (high/low; ICC, 0.46),
reviews, we excluded studies in which the only simulation . group vs independent practice (ICC, 0.71),
activities involved computer-based virtual patients (Cook & . individualized learning ( present/absent; ICC, 0.25, with raw
Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13

Triola 2009; Cook et al. 2010a) and human patient actors agreement 85%),
(standardized patients) (Bokken et al. 2008; May et al. 2009). . mastery learning (Issenberg’s ‘‘defined outcomes,’’ i.e.
training to a predefined level of proficiency, present/
absent; ICC, 0.53),
Study identification
. multiple learning strategies (high/low; ICC, 0.49),
We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, . range of task difficulty ( present/absent; ICC, 0.30, with raw
Web of Science, and Scopus using a search strategy developed agreement 82%),
by an experienced research librarian (PJE). The search . repetitive practice (number of repetitions; ICC, 0.60), and
included terms for the intervention (including simulator, . time spent learning (ICC, 0.72).
simulation, manikin, cadaver, MIST, Harvey, and many
Methodological quality was graded using the Medical
others), topic (surgery, endoscopy, anesthesia, trauma, colo-
For personal use only.

Education Research Study Quality Instrument (Reed et al.

noscopy, etc.), and learners (education medical, education
2007) and an adaptation of the Newcastle-Ottawa scale for
nursing, education professional, internship and residency,
cohort studies (Wells et al. 2007; Cook et al. 2008b) that
etc.). We used no beginning date cutoff, and the last date of
evaluates representativeness of the intervention group (ICC,
search was May 11, 2011. In addition, we added all articles
published in two journals devoted to health professions 0.68), selection of the comparison group (ICC, 0.26 with raw
simulation (Simulation in Healthcare and Clinical agreement 86%), comparability of cohorts (statistical adjust-
Simulation in Nursing) since their inception, and the entire ment for baseline characteristics in nonrandomized studies
reference list from several published reviews of health [ICC, 0.88], or randomization [ICC, 0.84] and allocation
professions simulation. Finally, we searched for additional concealment for randomized studies [ICC, 0.63]), blinding of
studies in the reference lists of all included articles published outcome assessment (ICC, 0.58), and completeness of follow-
before 1990 and a random sample of 160 included articles up (ICC, 0.36 with raw agreement 80%).
published in or after 1990. Our complete search strategy has Since the results associated with simulation training may
been published previously (Cook et al. 2011). vary for different outcomes, we distinguished outcomes using
Kirkpatrick’s classification (Kirkpatrick 1996) and abstracted
information separately for satisfaction, learning (knowledge
Study selection and skills, with skills further classified as time to complete the
Working independently and in duplicate, we screened all titles task, process, and product [see Box 1 for definitions]),
and abstracts for inclusion. In the event of disagreement or behaviors with patients (time and process), and results
insufficient information in the abstract we reviewed the full ( patient effects). Authors frequently reported multiple mea-
text of potential articles, again independently and in duplicate, sures of a single outcome (e.g. multiple measures of process
resolving conflicts by consensus. Chance-adjusted interrater skill), in which case we selected, in order of priority, (1) the
agreement for study inclusion, determined using intraclass author-defined primary outcome, (2) a global or summary
correlation coefficient (ICC), was 0.69. Non-English articles measure of effect, (3) the most clinically relevant measure, or
were translated in full. (4) the average of the measures reported. We also prioritized
skill outcomes assessed in a different setting (e.g. different
simulator or clinical setting) over those assessed in the
Data extraction simulator used for training.
Using a data abstraction form we abstracted data indepen-
dently and in duplicate for all variables where reviewer
Data synthesis
judgment was required, resolving conflicts by consensus.
Interrater agreement was fair (ICC 0.2–0.4) or moderate For each reported outcome we calculated the standardized
(0.4–0.6) for most variables (Landis & Koch 1977). We mean difference (Hedges’ g effect size) between each group
identified the main theme of each comparison (research using standard techniques (Borenstein 2009; Morris &
e869
 D. A. Cook et al.

DeShon 2002; Curtin et al. 2002; Hunter & Schmidt 2004) as we For studies with 42 groups (for example, three different
have detailed previously. (Cook et al. 2011) For studies simulation instructional designs), we selected for the main
reporting neither p values nor any measure of variance, we analysis the designs with the greatest between-group differ-
used the average standard deviation from all other studies ence, and then performed sensitivity analyses substituting the
reporting that outcome. If we could not calculate an effect size other design(s). We also performed sensitivity analyses
using reported data we requested additional information from excluding low-quality studies (those with NOS and MERSQI
authors via e-mail. scores below the median) and studies with imprecise effect
We used the I2 statistic (Higgins et al. 2003) to quantify size estimation ( p value upper limits or imputed standard
inconsistency (heterogeneity) across studies. I2 estimates the deviations).
percentage of variability across studies not due to chance, and We used SAS 9.2 (SAS Institute, Cary, NC) for all analyses.
values 450% indicate large inconsistency. Large inconsistency Statistical significance was defined by a two-sided alpha of
weakens the inferences that can be drawn, but does not pre- 0.05. Determinations of educational significance emphasized
clude the pooling of studies sharing a common conceptual link. Cohen’s effect size classifications (50.2 ¼ negligible; 0.2–
We planned meta-analyses to evaluate the effectiveness of 0.49 ¼ small; 0.5–0.8 ¼ moderate) (Cohen, 1988).
each instructional design feature, pooling the results of all
Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13

studies for which that feature varied between two simulation-

based interventions. For example, if one study group received Results
high feedback and the other low feedback, this study would be
included in the ‘‘Feedback’’ meta-analysis. If feedback were Trial flow
equal in both arms, it would be excluded from that analysis. To We identified 10 297 articles using our search strategy and 606
increase the power of these analyses, we merged process and from our review of reference lists and journal indices. From
product skills into a single outcome of ‘‘non-time skills,’’ and these we identified 295 studies comparing two or more
we also combined behaviors and patient effects. Because we simulation training interventions (Figure 1) of which 290
found large inconsistency in most analyses, we used random reported eligible outcomes. Two articles reported the same
effects models to pool weighted effect sizes. Many studies data and we selected the most detailed one for inclusion. We
appear in 41 analysis (i.e. both feedback and repetitive obtained additional outcomes data for 1 study from study
For personal use only.

practice) but no study appeared more than once per analysis. authors. Ultimately, we included 289 studies enrolling

Potentially relevant studies identified and screened for

retrieval (n=10,903)
• 10,297 from database search
• 606 from article reference lists and journal tables of
contents

Studies excluded, with reasons (n=8320)

• Not original research (1314)
• Did not use technology-enhanced simulation (5343)
• No health professions learners (488)
• No comparison group or time point (1175)

Studies retrieved for more detailed evaluation (n=2583)

Studies excluded, with reasons (n=2288)

• Not original research (150)
• Did not use technology-enhanced simulation (486)
• No health professions learners (98)
• No comparison group (864)
• No comparison with other simulation (690)

Studies potentially appropriate for inclusion in this

review (n=295)

Studies excluded from meta-analysis, with reasons (n=6)

• No relevant outcomes (5)
• Duplicate publication (1)

Studies eligible for meta-analysis (n=289)

Figure 1. Trial flow.

e870
 Instructional design features for simulation

18 971 trainees. Twenty-six of these 289 were multi-arm the articles (N ¼ 139) were published in or after 2008, and five
studies that included a comparison with no-intervention, and were published in a language other than English. Learners
the no-intervention results were reported previously (Cook included student and practicing physicians, nurses, emergency
et al. 2011). Table 1 summarizes key study characteristics and medicine technicians, dentists, chiropractors, and veterinar-
Appendix 1 provides a complete listing of articles with ians, among others. Table 1 summarizes the prevalence of
additional information. instructional design features such as feedback (75 studies),
repetitive practice (233 studies), and distributed practice (98
studies). Most studies reported learner skills, including 100
Study characteristics time, 197 process, and 56 product skill outcomes. Fifty-six
Studies in our sample used technology-enhanced simulations studies reported satisfaction, 34 reported knowledge out-
to teach topics such as minimally invasive surgery, dentistry, comes, 1 reported time behavior, 9 reported process behavior,
intubation, physical examination, and teamwork. Nearly half and 8 reported patient effects.

Table 1. Description of included studies.

Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13

Study Characteristic Level No. studies No. participants*

All studies 289 18 971
Group allocation Randomized 208 12 227
Study design Posttest-only 162 12 167
Pretest-posttest 127 6804
Location Simulation center 269 16 785
Clinical environment 13 1421
Both simulation center and clinical 7 765
Participantsy Medical students 111 4894
Physicians postgraduate training 87 2408
Physicians in practice 39 1684
Nurses and nursing students 42 4177
Emergency medical technicians and students 18 824
Dentists and dental students 22 1348
For personal use only.

Veterinarians and veterinary students 6 276

Chiropractors and chiropractic students 2 104
Other/ambiguous/mixed 43 3256
Clinical topicsz Minimally invasive surgery 79 3000
Resuscitation/trauma training 51 6630
Other surgery 34 1530
Dental 21 1416
Intubation 20 1922
Physical examination 18 933
Endoscopy and ureteroscopy 18 827
Vascular access 15 605
Communication and team skills 11 485
Anesthesia 8 260
Obstetrics 6 641
Key features Clinical variation§, present 64 4826
Cognitive interactivity, high 193 13 339
Curriculum integration§, present 52 5395
Distributed practice, 41 day 98 4745
Feedback§, high 75 6245
Group practice 72 6317
Individualized learning§, high 18 1545
Mastery learning, present§ 23 1050
Multiple learning strategies§, high 34 3102
Range of task difficulty§, present 40 1593
Repetitive practice§, present 233 15 521
Outcomesy Satisfaction 56 3042
Knowledge 34 2687
Skill: Time 100 4198
Skill: Process 197 9983
Skill: Product 56 2976
Behavior: Time 1 28
Behavior: Process 9 275
Patient effects 8 422
Quality Newcastle-Ottawa 4 points 160 10 888
MERSQI 12.5 points 173 11 459

Notes: See Appendix 1 for details on individual studies.

*Numbers reflect the number enrolled, except for Outcomes which reflect the number of participants who provided observations for analysis.
y
The number of studies and learners in some subgroups may sum to more than the number for all studies, and percentages may total more than
100%, because several studies included 41 learner group, fit within 41 clinical topic, or reported multiple outcomes.
z
Selected listing of the most frequently addressed topics (numerous other topics were addressed, with lower frequency). Some interventions
addressed 41 clinical topic.
§
Features identified by Issenberg et al. (Issenberg et al., 2005) Mastery learning is similar to Issenberg’s ‘‘defined outcomes.’’

e871
 D. A. Cook et al.

Study quality behaviors and patient effects. Figure 2 shows the pooled effect
size for each instructional design feature, organized by
Table 2 summarizes the methodological quality of included
outcome ( panels A–E). For non-time skills we confirmed
studies. The number of participants providing outcomes
small to moderate positive effects favoring the presence of
ranged from 4 to 817 with a median of 30 (interquartile
each proposed feature of effective simulation except group
range 20–53). Groups were randomly assigned in 208 studies
training, and most (7 of 11) effects were statistically significant.
(72%). Studies lost more than 25% of participants from time of
enrollment or failed to report follow-up for 13 of 56 satisfaction Results for other outcomes nearly always (35 of 38) favored the
outcomes (23%), 5 of 34 knowledge (15%), 31 of 100 time skill proposed feature, but results were usually not statistically
(31%), 62 of 197 process skill (31%), 18 of 56 product skill significant.
(32%), and 1 of 9 process behavior (11%) (time behavior and For example, for the non-time skill outcomes (Figure 2,
patient effect outcomes had complete follow-up). Assessors panel D), 20 studies reported a comparison in which one
were blinded to group assignment for 309 of 461 outcome simulation design included tasks reflecting a range of difficulty
measures (67%). Most outcomes reflect objective measures and the other did not. Among these studies, designs offering a
(e.g. computer scoring, objective key, or human rater). All range of difficulty were associated with better outcomes than
knowledge and time behavior outcomes were determined those of uniform difficulty, with pooled effect size (ES) 0.68
Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13

objectively, while trainee self-assessments comprised five (95% confidence interval [CI], 0.30–1.06, p 5 0.001). This
process skill outcomes and one each of time skill, product difference is statistically significant, and moderate in magni-
skill, process behavior, and patient effect outcomes. The mean tude using Cohen’s classification. Further differences of small
(SD) quality scores averaged 3.5 (1.3) for the Newcastle- to moderate magnitude were found for instructional designs
Ottawa Scale (6 points indicating highest quality) and 12.3 incorporating clinical variation (0.20), more interactivity (0.65),
(1.8) for the Medical Education Research Study Quality training over 41 day (0.66), more feedback (0.44), individu-
Instrument (maximum 18 points). alization (0.52), mastery learning (0.45), more learning strat-
egies (0.62), repetition (ES 0.68), and longer time (0.34).
Findings for knowledge, time, and behavior-patient effect
Meta-analysis
outcomes were similarly favorable, but with smaller and
For meta-analysis we merged process and product skills into a usually statistically non-significant effects (see Figure 2).
For personal use only.

single outcome of ‘‘non-time skills,’’ and we likewise merged Inconsistency was large (I2 4 50%) in most analyses.

Table 2. Quality of included studies.

No. (%) present;

Scale Item Subscale ( points if present) N ¼ 289
Medical Education Research Study Quality Instrument (MERSQI)*
Study design (maximum 3) Observational 2-group (2) 81 (28)
Randomized 2-group (3) 208 (72)
Sampling: No. institutions (maximum 1.5) 1 (0.5) 255 (88)
2 (1) 10 (4)
42 (1.5) 24 (8)
Sampling: Follow-up (maximum 1.5) 550% or not reported (0.5) 81 (28)
50–74% (1) 9 (3)
75% (1.5) 199 (69)
Type of data: Outcome assessment (maximum 3) Subjective (1) 38 (13)
Objective (3) 251 (87)
Validity evidence (maximum 3) Content (1) 83 (29)
Internal structure (1) 78 (27)
Relations to other variables (1) 27 (9)
Data analysis: appropriate (maximum 1) Appropriate (1) 260 (90)
Data analysis: sophistication (maximum 2) Descriptive (1) 13 (5)
Beyond descriptive analysis (2) 276 (95)
Highest outcome type (maximum 3) Satisfaction (1) 25 (9)
Knowledge, skills (1.5) 249 (86)
Behaviors (2) 7 (2)
Patient/health care outcomes (3) 8 (3)

Newcastle-Ottawa Scale (modified)y

Representativeness of sample Present (1) 53 (18)
Comparison group from same community Present (1) 268 (93)
Comparability of comparison cohort, criterion Az Present (1) 213 (74)
Comparability of comparison cohort, criterion Bz Present (1) 106 (37)
Blinded outcome assessment Present (1) 175 (61)
Follow-up highz Present (1) 208 (72)

Notes: *MERSQI total score (maximum 18): mean 12.3 (SD 1.8), median 12.5 (range 6.5–16).
y
NOS total score (maximum 6): mean 3.5 (SD 1.3), median 4 (range 1–6).
z
Comparability of cohorts criterion A was present if the study (a) was randomized, or (b) controlled for a baseline learning outcome; criterion B was present if (a) a
randomized study concealed allocation, or (b) an observational study controlled for another baseline learner characteristic. Follow-up was high if 75% of those
enrolled provided outcome data, or if authors described those lost to follow-up.

e872
 Instructional design features for simulation
Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Figure 2. Random effects meta-analysis

Comparisons of simulation interventions varying in the specified key feature; positive numbers favor the intervention with more of
that feature. If a feature is not present in the analysis for a given outcome, it is because no studies reporting that outcome had
interventions that varied for that particular feature. Some feature comparisons had only 1 relevant study; these are included in the
figures but we note that the effect size reflects only that 1 study (ie, not pooled).

The exception to the predicted pattern was group training, non-time skills: pooled effect sizes varied by 50.08 in all
which showed a small negative association with non-time analyses, and statistical significance changed in only one
skills (ES 0.22 [95% CI, 0.48 to 0.03], p ¼ 0.09). Knowledge instance (the group practice analysis was now statistically
and time outcomes (one study each) showed similar results. significant, p ¼ 0.02). For behavior–patient effects the pooled
Several studies reporting non-time skills and behavior- ES for feedback dropped to 0.18.
patient effects had three simulation arms. Since we could only Additional sensitivity analyses excluded low-quality studies.
compare two groups at once, we first included the groups The direction of effect reversed only rarely (5 of 153 analyses),
with the greatest between-design difference and then per- namely: mastery learning (non-time skill outcomes) when
formed sensitivity analyses substituting the third group excluding imprecise effect size estimation or low NOS score
(see Appendix 2). Results changed almost imperceptibly for (N ¼ 4 studies remaining for each analysis); feedback
e873
 D. A. Cook et al.
Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Figure 2. Continued.

(satisfaction outcomes) when excluding low MERSQI or NOS Discussion

score (N ¼ 2 for each); and interactivity (behavior–patient
effects) when excluding low MERSQI scores (N ¼ 2). In 2005, Issenberg et al. proposed 10 features of effective
simulation based on prevalence in the literature. Our
synthesis of research provides empiric support for nearly
all of these features and several others. Although the
Research themes pooled effect sizes were often small and not statistically
Through an iterative process we identified six main research significant, and between-study inconsistency was high, the
themes, with a much larger number of sub-themes (Table 3). consistent direction of effect across outcomes suggests that
Thirty-eight studies had two or three arms each, resulting in a the benefits are real. To put these findings in perspective,
total of 337 comparisons. The most prevalent main theme the effect size of 0.68 observed for ‘‘range of difficulty’’
involved comparisons of different instructional design features using non-time skill outcomes would translate to a 5%
such as the amount or method of feedback, the sequence of improvement (out of 100%) on a typical skill assessment.
training activities, task variability, or repetition. Second most Effect sizes were generally larger for skills than for
common were studies that compared two technology- knowledge, a tendency we also observed among studies
enhanced simulation modalities, such as mannequin vs part- comparing simulation with non-simulation instruction (Cook
task model, mannequin vs virtual reality, or two different et al. 2012). Of the twelve features evaluated, only group
mannequins. Several studies evaluated the addition of another instruction failed to demonstrate consistently positive effects.
instructional modality (e.g. a lecture, computer-assisted Interestingly, in the previous meta-analysis (Cook et al.
instruction, or another simulation modality) to the standard 2012) of studies comparing simulation with non-simulation
simulation training. The remaining themes focused on the role instruction, group instruction was associated with improved
of the instructor, sensory augmentation including haptics, and outcomes; this incongruity merits further study.
group composition. We initially identified a main theme of We also classified the research themes for 337 simulation–
‘‘fidelity,’’ but upon further reflection realized that all of the simulation comparisons. The most prevalent theme involved
studies thus classified could be more appropriately classified evaluating key features of instructional design. These stud-
using another theme, most often that of ‘‘modality ies, along with those exploring instructor roles and group
comparison.’’ composition, typically allowed generalizable conclusions.
e874
 Instructional design features for simulation

Table 3. Research themes (comparisons) addressed by studies.

No. studies
Theme, No. (%)* Theme definition Sub-theme (% of theme)*
Group composition, N ¼ 2 Compare different approaches to grouping Interdisciplinary vs single-discipline group 1 (50%)
(1%) learners.
Independent vs group 1 (50%)
Instructional design, Compare different design features to Feedback 47 (35.3%)
N ¼ 133 (39%) enhance instructional effectiveness.
Sequence 24 (18%)
Teach cognitive or mental imagery techniques 12 (9%)
Repetition 10 (7.5%)
Task variability 9 (6.8%)
Clinical scenario 5 (3.8%)
Hands on practice 5 (3.8%)
Timing 4 (3%)
Instructions 3 (2.3%)
Stress 3 (2.3%)
Additional practice 2 (1.5%)
Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13

Blending simulation and non-simulation 2 (1.5%)

instruction
Interactivity 2 (1.5%)
Mastery 2 (1.5%)
Testing effect 2 (1.5%)
Add reminder 1 (0.8%)
Instructor, N ¼ 27 (8%) Compare different levels of instructor train- Self-instruction 17 (63%)
ing or presence.
Instructor intensity 5 (18.5%)
Instructor training/experience 3 (11.1%)
Distance supervision 2 (7.4%)
Modality added, N ¼ 37 Evaluate the addition of one or more other CAI, add or compare 9 (24.3%)
(11%) modalities (eg, lecture, CAI, or other
simulation) to baseline simulation training
For personal use only.

Simulator, add 9 (24.3%)

Patient/SP experience, add or compare 7 (18.9%)
Discussion (excluding debriefing), add or 3 (8.1%)
compare
External support, add 2 (5.4%)
Robot assistance, add 2 (5.4%)
Team training, add 2 (5.4%)
Lecture, add or compare 2 (5.4%)
Tutor, add 1 (2.7%)
Sensory augmentation, Evaluate the addition of a feature or effect to Tactile (includes haptics) 11 (57.9%)
N ¼ 19 (6%) enhance sensory experience.
Visual 7 (36.8%)
Olfactory 1 (5.3%)
Simulation modalities com- Compare two technology-enhanced simu- VR vs model 40 (33.6%)
pared, N ¼ 119 (35%) lation modalities
Mannequin vs mannequin 16 (13.4%)
Model vs model 11 (9.2%)
Mannequin vs model 9 (7.6%)
Model vs animal 9 (7.6%)
VR vs VR 8 (6.7%)
Mannequin vs cadaver 5 (4.2%)
Model vs cadaver 4 (3.4%)
VR vs animal 4 (3.4%)
Mannequin vs animal 3 (2.5%)
Synthetic vs human/animal products 3 (2.5%)
VR vs cadaver 2 (1.7%)
VR vs mannequin 2 (1.7%)
Animal living vs dead 1 (0.8%)
Animal products vs animal products 1 (0.8%)
Cadaver vs animal 1 (0.8%)

Notes: *The 289 studies addressed a total of 337 research themes (38 studies had 2 or 3 arms). Percentages are for sub-themes within a theme.
Abbreviations: CAI ¼ computer-assisted instruction; SP ¼ standardized patient; VR ¼ virtual reality.

By contrast, another one-third of the themes focused on As a result, we suspect the findings from modality comparisons
comparing different simulation modalities. While modality will have limited generalizability.
comparisons initially appear useful, we noted that the results One design feature from Issenberg et al.’s review that we
varied widely as technologies changed and evolved, the did not code was fidelity. We found fidelity difficult to code,
educational context varied, and different implementations of during both the quantitative data abstraction and the the-
the same technology employed different instructional designs. matic analysis. We found that ‘‘fidelity’’ encompasses a number
e875
 D. A. Cook et al.

of different facets related to the simulation activity, including Comparison with previous reviews
the characteristics of the simulator that mediate sensory
The present review complements our recent meta-analysis
impressions (visual, auditory, olfactory, and tactile/haptic),
showing that simulation training is associated with large
the nature of the learning objectives and task demands, the
positive effects in comparison with no intervention (Cook
environment, and other factors that might affect learner
et al. 2011). Having established that simulation can be
engagement and suspension of disbelief. Labeling a simulation
effective, the next step is to understand what makes it
as ‘‘high fidelity’’ conveys such diverse potential meanings
effective. Although several other reviews have addressed
that the term loses nearly all usefulness. Based on our simulation in general (Issenberg et al. 2005; McGaghie et al.
experiences during this review, we suggest that researchers 2010) or in comparison with no intervention, (Gurusamy et al.
and educators employ more specific terminology when 2008; McGaghie et al. 2011), we are not aware of previous
discussing the physical and contextual attributes of simulation reviews focused on comparisons of different technology-
training. enhanced simulation interventions or instructional designs. By
confirming the effectiveness of the design features proposed
by Issenberg et al. (2005), our comprehensive and quantitative
Limitations and strengths synthesis represents a novel and important contribution to
Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13

In order to present a comprehensive thematic overview of the the field.

field and achieve adequate statistical power for meta-analyses, Our findings of small to moderate effects favoring theory-
predicted instructional design features parallel the findings of a
we used intentionally broad inclusion criteria. However, in so
review of Internet-based instruction (Cook et al. 2008b). An
doing we included studies reflecting diverse training topics,
association between longer time in practice and improved
instructional designs, and outcome measures. These differ-
outcomes was also reported in a previous review of
ences likely contributed to the large between-study inconsis-
simulation-based education (McGaghie et al. 2006).
tency. This inconsistency tempers our inferences, but does not
preclude meta-analytic pooling (Montori et al. 2003; Cook
2012b). Future original research and research syntheses might Implications
further clarify the importance of these instructional design
The features proposed by Issenberg et al. (2005) as central to
features for specific topics, such as technical and non-
For personal use only.

effective simulation appear to work, as do the additional

technical tasks.
features we identified. We recommend that these be consid-
Literature reviews are necessarily constrained by the
ered the current ‘‘best practices’’ for the field. In order of
quantity and quality of available evidence. Among the
pooled effect size, these are: range of difficulty, repetitive
included studies, sample sizes were relatively small, sample
practice, distributed practice, cognitive interactivity, multiple
representativeness was rarely addressed, outcome validity learning strategies, individualized learning, mastery learning,
evidence was infrequently presented, and many reports feedback, longer time, and clinical variation.
failed to clearly describe key features of the context, However, we simultaneously highlight the need for further
instructional design, or outcomes. Although we found research elucidating what works, for whom, under what
numerous studies reporting skill outcomes and several circumstances. The large inconsistency observed in nearly all
reporting satisfaction and knowledge, we found few studies analyses indicates that the effect varies from study to study,
reporting higher-order outcomes of behavior and patient and the relative contribution of multiple potentially influential
effects. However, more than 70% of the studies used variables (learners, environment, operational definition of
randomization, and MERSQI scores were substantially higher interventions, outcomes, and other study methods) remains
than those found in previous reviews of medical education unclear. Besides meta-analysis, other synthesis methods such
research (Reed et al. 2007, 2008). as realist review (Pawson et al. 2005) will help interpret
Coding reproducibility was suboptimal for some instruc- existing evidence.
tional design features, likely due to both poor reporting and Going forward, we believe that a fundamental change in
difficulty operationalizing coding criteria. However, we the conception and design of new research is required. To-
reached consensus on all codes prior to meta-analysis. date, the number of studies attempting to clarify the use of
To increase statistical power and to reduce the number of simulation by directly comparing different simulation-based
independent meta-analyses, we combined process and prod- interventions is small (N ¼ 289) relative to the number of
uct outcomes for assessments in both an education setting studies comparing simulation with no intervention or non-
(skills) and with real patients (behaviors and patient effects). simulation instruction (N ¼ 690 [see Figure 1]) and studies
Analyzing these separately might have led to slightly different without any comparison (N ¼ 864). Studies comparing simu-
conclusions. lation with simulation will do far more to advance the field
Our review has several additional strengths, including an than comparisons of simulation with non-simulation
extensive literature search led by a skilled librarian; no approaches. (Cook 2010) Yet not all simulation–simulation
restriction based on time or language of publication; explicit comparisons are equally useful, and studies evaluating modal-
inclusion criteria encompassing a broad range of learners, ities or instructional designs without a conceptual or theoret-
outcomes, and study designs; duplicate, independent, and ical rationale have limited generalizability.
reproducible data abstraction; rigorous coding of methodo- The field thus needs research that goes beyond simple com-
logical quality; and hypothesis-driven analyses. parisons of presence/absence of key features (Weinger, 2010).
e876
 Instructional design features for simulation

For example, it appears that feedback improves outcomes— JASON H. SZOSTEK is Assistant Professor of Medicine, Department of
Medicine, College of Medicine, Mayo Clinic, Rochester, MN, USA.
but we expect that much could yet be learned about the basis,
AMY T. WANG is Assistant Professor of Medicine, Department of Medicine,
timing, and delivery of feedback. This research will require
College of Medicine, Mayo Clinic, Rochester, MN, USA.
progressively refined theories and conceptual frameworks that
PATRICIA J. ERWIN is Assistant Professor of Medical Education, College of
programmatically study carefully constructed questions
Medicine, Mayo Clinic, Rochester, MN, USA.
(Bordage 2009; McGaghie et al. 2010). The themes identified
ROSE HATALA is Associate Professor, Department of Medicine, University
in this review (see Table 3) provide a starting point for such of British Columbia, Vancouver, BC, Canada
research programs. It will also be important to systematically
account the costs of alternate instructional approaches (Levin
2001), and explore how costs can inform design decisions References
(Zendejas 2012).
Bokken L, Rethans JJ, Scherpbier AJ, Van Der Vleuten CP. 2008. Strengths
Of course, such research will not be possible without
and weaknesses of simulated and real patients in the teaching of skills
adequate funding. Health professions education research is to medical students: A review. Simul Healthc 3:161–169.
underfunded (Reed et al. 2005), even though funding is Bordage G. 2009. Conceptual frameworks to illuminate and magnify. Med
associated with higher quality work (Reed et al. 2007). Those Educ 43:312–319.
Borenstein M. 2009. Effect sizes for continuous data. In: Cooper H, Hedges
Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13

responsible for funding decisions must recognize the impor-

LV, Valentine JC, editors. The Handbook of Research Synthesis. 2nd ed.
tance of theory-building research that clarifies (Cook et al. New York: Russell Sage Foundation. pp 221–235.
2008a) the modalities and features of simulation-based edu- Cohen J. 1988. Statistical Power Analysis for the Behavioral Sciences.
cation that improve learner and patient outcomes with greatest Hillsdale, NJ: Lawrence Erlbaum.
effectiveness and at lowest cost. Cook DA. 2012a. If you teach them, they will learn: Why medical education
needs comparative effectiveness research, Advances in Health Sciences
Finally, we note that the effect sizes for these comparisons
Education, 17:305–310.
are much lower than those observed for comparisons with no Cook DA. 2012b. Randomized controlled trials and meta-analysis
intervention. This is not unexpected, as comparing training vs in medical education: What role do they play?, Medical Teacher,
no training ought to result in greater improvement than In press.
comparing two active instructional interventions (Cook 2012a). Cook DA. 2010c. One drop at a time: Research to advance the science of
simulation. Simul Healthc 5:1–4.
However, we caution investigators that the small samples that
Cook DA, Bordage G, Schmidt HG. 2008a. Description, justification, and
For personal use only.

proved sufficient to identify statistically significant differences clarification: A framework for classifying the purposes of research in
in no-intervention-comparison studies will be inadequate for medical education. Medl Educ 42:128–133.
simulation–simulation research. Advance calculation of sample Cook DA, Brydges R, Hamstra S, Zendejas B, Szostek JH, Wang AT, Erwin
size, clear justification of educational significance, and use of P, Hatala R. 2012. Comparative effectiveness of technology-enhanced
simulation vs other instructional methods: A systematic review and
confidence intervals when interpreting results will be essential.
meta-analysis. Simul Healthc, In press.
These, together with other research methods that minimize Cook DA, Erwin PJ, Triola MM. 2010a. Computerized virtual patients in
confounding, will facilitate studies that truly advance our health professions education: A systematic review and meta-analysis.
understanding of how to improve healthcare through simula- Acad Med 85:1589–1602.
tion-based education. Cook DA, Hatala R, Brydges R, Zendejas B, Szostek JH, Wang AT, Erwin P,
Hamstra S. 2011. Technology-Enhanced simulation for health profes-
sions education: A systematic review and meta-analysis. J Am Med
Assoc 306:978–988.
Acknowledgements Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM.
2008b. Internet-Based Learning in the Health Professions: A Meta-
Portions of this work were presented at the 2012 International Analysis. JAMA 300:1181–1196.
Meeting on Simulation in Healthcare, in San Diego, California. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM.
2010b. instructional design variations in internet-based learning for
Declaration of interest: This work was supported by health professions education: A systematic review and meta-analysis.
intramural funds, including an award from the Division of Acad Med 85:909–922.
General Internal Medicine, Mayo Clinic. The authors report no Cook DA, Triola MM. 2009. Virtual patients: A critical literature review and
proposed next steps. Medical Education 43:303–311.
conflicts of interest. The authors alone are responsible for the
Curtin F, Altman DG, Elbourne D. 2002. Meta-analysis combining parallel
content and writing of the article. and cross-over clinical trials. I: Continuous outcomes. Stat Med
21:2131–2144.
Gurusamy K, Aggarwal R, Palanivelu L, Davidson BR. 2008. Systematic
review of randomized controlled trials on the effectiveness of virtual
Notes on contributors reality training for laparoscopic surgery. Br J Surg 95:1088–1097.
DAVID A. COOK is Professor of Medicine and Medical Education, Higgins JPT, Thompson SG, Deeks JJ, Altman DG. 2003. Measuring
Department of Medicine, College of Medicine, Mayo Clinic, and Director, inconsistency in meta-analyses. Br Med J 327:557–560.
Office of Education Research, Mayo Medical School, Rochester, MN, USA. Hunter JE, Schmidt FL. 2004. Methods of Meta-Analysis: Correcting Error
and Bias in Research Findings. Thousand Oaks, CA: Sage.
STANLEY J. HAMSTRA is Acting Assistant Dean, Academy for Innovation in
Issenberg SB, Mcgaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ. 2005.
Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, ON,
Features and uses of high-fidelity medical simulations that lead to
Canada.
effective learning: A BEME systematic review. Med Teac 27:10–28.
RYAN BRYDGES is Assistant Professor, Department of Medicine, University Kirkpatrick D. 1996. Revisiting Kirkpatrick’s four-level model. Training Dev
of Toronto, Toronto, ON. 50(1):54–59.
BENJAMIN ZENDEJAS is a Resident in the Department of Surgery, Mayo Landis JR, Koch GG. 1977. The measurement of observer agreement for
Clinic College of Medicine, Rochester, MN, USA. categorical data. Biometrics 33:159–174.
e877
 D. A. Cook et al.

Levin HM. 2001. Waiting for Godot: Cost-effectiveness analysis in Pawson R, Greenhalgh T, Harvey G, Walshe K. 2005. Realist review – a
education. New Directions for Evaluation 90:55–68. new method of systematic review designed for complex policy
May W, Park JH, Lee JP. 2009. A ten-year review of the literature on the use interventions. J Health Serv Res Policy 10(Suppl 1):21–34.
of standardized patients in teaching and learning: 1996–2005. Med Reed DA, Beckman TJ, Wright SM, Levine RB, Kern DE, Cook DA. 2008.
Teach 31:487–492. Predictive validity evidence for medical education research study
Mcgaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. 2011. Does quality instrument scores: Quality of submissions to JGIM’s Medical
simulation-based medical education with deliberate practice yield Education Special Issue. J Gen Intern Med 23:903–907.
better results than traditional clinical education? A meta-analytic Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. 2007.
comparative review of the evidence. Acad Med 86:706–711. Association between funding and quality of published medical
Mcgaghie WC, Issenberg SB, Petrusa ER, Scalese RJ. 2006. Effect of practice education research. J Am Med Assoc 298:1002–1009.
on standardised learning outcomes in simulation-based medical Reed DA, Kern DE, Levine RB, Wright SM. 2005. Costs and funding for
education. Med Educ 40:792–797. published medical education research. J Am Med Assoc 294:1052–1057.
Mcgaghie WC, Issenberg SB, Petrusa ER, Scalese RJ. 2010. A critical review Weinger MB. 2010. The pharmacology of simulation: A conceptual
of simulation-based medical education research: 2003–2009. Med Educ framework to inform progress in simulation research. Simul Healthc
44:50–63. 5:8–15.
Moher D, Liberati A, Tetzlaff J, Altman DG. 2009. Preferred reporting items Wells GA, Shea B, O’connell D, Peterson J, Welch V, Losos M, Tugwell P.
for systematic reviews and meta-analyses: The PRISMA statement. Ann The Newcastle-Ottawa Scale (NOS) for assessing the quality of
Intern Med 151:264–269. nonrandomised studies in meta-analyses. [Accessed 29 February,
Montori VM, Swiontkowski MF, Cook DJ. 2003. Methodologic issues 2012] Available from http://www.ohri.ca/programs/clinical_epidemiol-
Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13

in systematic reviews and meta-analyses. Clin Orthop Relat Res ogy/oxford.htm

413:43–54. Zendejas B, Wang AT, Brydges R, Hamstra SJ, Cook DA. 2012. Cost:
Morris SB, Deshon RP. 2002. Combining effect size estimates in meta- The missing outcome in simulation-based medical education research:
analysis with repeated measures and independent-groups designs. A systematic review. Surgery; Online early doi 10.1016/j.surg.
Psychol Meth 7:105–125. 2012.06.025.
For personal use only.

e878
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Appendix
Table A1. List of all included studies.

Participants Research comparisons Quality

Citation (sorted by year then author) Trainee N RCT Topic Features Theme Outcomes MERSQI NOS

Wittrock JW, et al. Use of different sized models in teaching D 86 Dentistry SA:Visual SPd 10.5 2
restorative dentistry. J Dent Educ. 1970; 34:76–82.
Johnson GH, et al. Teaching pelvic examination to second-year MS 140 Physical exam ClinV, CogI, FB, MA:Patient/SP Sa 9 2
medical students using programmed patients. Am J Obstet Indiv, MLS experience, add
Gynecol. 1975; 121:714–717.
Salvendy G, et al. A second generation training simulator for D 30 RCT Dentistry CogI, FB, GrpP, CM:VR vs model ST, SPc 13.5 3
acquisition of psychomotor skills in cavity preparation. J Dent Indiv
Educ. 1975; 39:466–471.
Holzman GB, et al. Initial pelvic examination instruction: The MS 28 Physical exam Time MA:Tutor, add K, SPc 12.5 3
effectiveness of three contemporary approaches. Am J Obstet
Gynecol. 1977; 129:124–129.
Lefcoe DL, et al. Simulated models: a mode for instruction in root O 12 RCT Dental cleaning SA:Visual P 13.5 5
planing procedures. Educ Dir Dent Aux. 1979; 3:20–4.
Herrin TJ, et al. Modular approach to CPR training. South Med J. MS, RN 303 Resuscitation FB, Indiv In:Self-instruction K 11.5 3
1980; 73:742–744. (BLS,ACLS,ATLS)
Salvendy G, et al. The development and validation of an analytical MS 18 RCT Open surgery/suturing ID:Teach cognition ST, SPc 13.5 4
training program for medical suturing. Hum Factors. 1980;
22:153–170.
LaTurno SA, et al. An evaluation of a teaching aid in endodontics. J D 150 Dentistry MLS SA:Visual SPd 10.5 2
Endod. 1984; 10:507–511.
Stewart RD, et al. Effect of varied training techniques on field EMT 94 Intubation ClinV, CogI, Indiv, MA:Patient/SP P 12 4
endotracheal intubation success rates. Ann Emerg Med. 1984; Mast, MLS, experience, add
13:1032–1036. RangeD
Weiner S, et al. An evaluation of sequential models in the preclinical D 142 Dentistry ID:Sequence SPd 10.5 2
laboratory. J Dent Educ. 1985; 49:109–110.
Champagne MT, et al. Use of a heart sound simulator in teaching RN 37 RCT Physical exam SA:Tactile SPc 15.5 3
cardiac auscultation. Focus Crit Care. 1989; 16:448–456.
Carpenter LG, et al. A comparison of surgical training with live V 12 RCT Veterinary surgery CM:Animal living vs ST, SPc, SPd 12.5 6
anesthetized dogs and cadavers. Veterinary Surgery. 1991; dead
20:373–378.
Stratton SJ, et al. Prospective study of manikin-only versus manikin EMT 60 RCT Intubation ClinV, Mast, MA:Simulator, add P 11 3
and human subject endotracheal intubation training of para- RangeD
medics. Ann Emerg Med. 1991; 20:1314–1318.
Trooskin SZ, et al. Teaching endotracheal intubation using animals EMT 19 RCT Intubation CogI, Indiv CM:Manikin vs SPd, P 14.5 4
and cadavers. Prehospital Disaster Med. 1992; 7:179–182. animal
Bhat BV, et al. Undergraduate training in neonatal resuscitation – a MS 110 Resuscitation DistP ID:Timing K 11.5 4
modified approach. Indian J Matern Child Health. 1993; 4:87–88. (BLS,ACLS,ATLS)
Holmberg DL, et al. Use of a dog abdominal surrogate for teaching V 116X Veterinary surgery ClinV CM:Model vs Sa 7.5 2
surgery. Journal of Veterinary Medical Education. 1993; 20:61– animal
62.
Mazzuca SA, et al. Improved training of house officers in a PG 11 Physical exam MLS ID:Add reminder BP 12 2
rheumatology consult service. Arthritis Care Res. 1993; 6:59–63.
Instructional design features for simulation

e879
(continued)
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Table A1. Continued.

e880
Participants Research comparisons Quality

Citation (sorted by year then author) Trainee N RCT Topic Features Theme Outcomes MERSQI NOS
D. A. Cook et al.

Campbell HS, et al. Teaching medical students how to perform a MS 54 RCT Physical exam CogI, FB, Indiv, In:Instructor train- SPd 11.5 3
clinical breast examination. Acad Med. 1994; 69:993–995. MLS, Time ing/experience
From RP, et al. Assessment of an interactive learning system with MS 97 RCT Intubation MLS In:Self-instruction K, BP 13 6
‘‘sensorized’’ manikin head for airway management instruction.
Anesth Analg. 1994; 79:136–142.
Reader CM, et al. Anatomical artificial teeth for teaching preclinical D 29X Dentistry CM:Model vs Sa, SPd 7.5 2
endodontics. Journal Dental Education. 1994; 58:229–232. cadaver
Greenfield CL, et al. Comparison of surgical skills of veterinary V 36 RCT Open surgery/suturing ClinV CM:Model vs SPc 11.5 4
students trained using models or live animals. J Am Vet Med animal
Assoc. 1995; 206:1840–1845.
van Stralen DW, et al. Retrograde intubation training using a RN, EMT, O 88 Intubation FB, Indiv In:Self-instruction ST 11.5 2
mannequin. Am J Emerg Med. 1995; 13:50–52.
Olsen D, et al. Evaluation of a hemostasis model for teaching basic V 40 RCT Open surgery/suturing RepP CM:Model vs SPc, SPd 13.5 5
surgical skills. Vet Surg. 1996; 25:49–58. animal
Thomas WE, et al. A preliminary evaluation of an innovative synthetic PG 9X Open surgery/suturing CM:Synthetic vs Sa 7 2
soft tissue simulation module (‘Skilltray’) for use in basic surgical human/animal
skills workshops. Ann R Coll Surg Engl. 1996; 78:268–271. products
Long NK, et al. A comparison of two teaching simulations in D 87 RCT Dentistry CM:Synthetic vs ST, SPd 14.5 6
preclinical operative dentistry. Oper Dent. 1997; 22:133–137. human/animal
products
Noordergraaf GJ, et al. Learning cardiopulmonary resuscitation MS 104 Resuscitation FB, GrpP CM:Manikin vs SPc, SPd 12.5 4
skills: does the type of mannequin make a difference? Eur J (BLS,ACLS,ATLS) manikin
Emerg Med. 1997; 4:204–209.
Roberts I, et al. Airway management training using the laryngeal RN 52 RCT Intubation ClinV, CogI, MLS, MA:Patient/SP SPd 12.5 3
mask airway: a comparison of two different training programmes. RangeD experience,
Resuscitation. 1997; 33:211–214. compare
Ali J, et al. Effect of the Advanced Trauma Life Support program on MS 44 Resuscitation CogI, FB, Indiv, ID:Hands on SPc 10.5 2
medical students’ performance in simulated trauma patient (BLS,ACLS,ATLS) MLS, RepP
management. Journal of Trauma: Injury Infection & Critical Care.
1998; 44:588–591.
Christenson J, et al. A comparison of multimedia and standard MS 113 Resuscitation ClinV, CogI, FB, MA:CAI, compare K, SPc 13.5 4
advanced cardiac life support learning. Acad Emerg Med. 1998; (BLS,ACLS,ATLS) Indiv
5:702–708.
Kaczorowski J, et al. Retention of neonatal resuscitation skills and PG 27 RCT Resuscitation CogI, FB, Indiv, In:Self-instruction K, SPc 12 6
knowledge: a randomized controlled trial. Fam Med. 1998; (BLS,ACLS,ATLS) MLS, Time
30:705–711.
Rogers DA, et al. Computer-assisted learning versus a lecture and MS 82 RCT Open surgery/suturing CogI, FB, MLS MA:CAI, compare ST, SPc, SPd 13.5 5
feedback seminar for teaching a basic surgical technical skill. Am
J Surg. 1998; 175:508–510.
Todd KH, et al. Randomized, controlled trial of video self-instruction MS 89 RCT Resuscitation CogI, FB, GrpP, In:Self-instruction K, SPc 13.5 6
versus traditional CPR training. Ann Emerg Med. 1998; 31:364– (BLS,ACLS,ATLS) Indiv, Time
369.
Ali J, et al. Comparison of performance of interns completing the old PG 32 RCT Resuscitation CogI, MLS ID:Interactivity Sa, K, SPc 12.5 4
(1993) and new interactive (1997) Advanced Trauma Life Support (BLS,ACLS,ATLS)
courses. Journal of Trauma-Injury Infection & Critical Care. 1999;
46:80–86.
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Anastakis DJ, et al. Assessment of technical skills transfer from the PG 23X RCT Open surgery/suturing CM:Model vs SPc 12.5 5
bench training model to the human model. Am J Surg. 1999; cadaver
177:167–170.
Burdea G, et al. Virtual reality-based training for the diagnosis of PG 8 Physical exam CogI, FB CM:VR vs model SPd 6.5 1
prostate cancer. IEEE Trans Biomed Eng. 1999; 46:1253–1260.
Gallagher AG, et al. Virtual reality training in laparoscopic surgery: a O 16 RCT Min. invasive surg. CogI, DistP, FB, CM:VR vs model SPc 12.5 3
preliminary assessment of minimally invasive surgical trainer Indiv, RangeD
virtual reality (MIST VR). Endoscopy. 1999; 31:310–313.
Griffon DJ, et al. Evaluation of a hemostasis model for teaching V 40 RCT Open surgery/suturing RepP CM:Model vs K, ST, SPc 13.5 6
ovariohysterectomy in veterinary surgery. Vet Surg. 2000; cadaver
29:309–316.
Jordan J-A, et al. A comparison between randomly alternating O 16 RCT Min. invasive surg. CogI, FB, Indiv, CM:VR vs model SPc 10.5 3
imaging, normal laparoscopic imaging, and virtual reality training RangeD
in laparoscopic psychomotor skill acquisition. Am J Surg. 2000;
180:208–211.
Keyser EJ, et al. A simplified simulator for the training and evaluation PG 22X RCT Min. invasive surg. CM:Model vs model SPc 12.5 4
of laparoscopic skills. Surg Endosc. 2000; 14:149–153.
Kovacs G, et al. A randomized controlled trial on the effect of MS, D, O 53 RCT Intubation CogI, FB, Indiv ID:Repetition SPc 14.5 5
educational interventions in promoting airway management skill
maintenance. Ann Emerg Med. 2000; 36:301–309.
Jordan JA, et al. Virtual reality training leads to faster adaptation to MS, PG, O 24 RCT Min. invasive surg. CogI, FB, Indiv, CM:VR vs model SPc 13.5 4
the novel psychomotor restrictions encountered by laparoscopic RangeD
surgeons. Surg Endosc. 2001; 15:1080–1084.
Pugh CM, et al. The effect of simulator use on learning and self- MS 59 RCT Physical exam CogI ID:Feedback SPc 11.5 3
assessment: The case of Stanford University’s E-Pelvis simulator.
Stud Health Technol Inform. 2001; 81:396–400.
Risucci D, et al. The Effects of Practice and Instruction on Speed PG 14 RCT Min. invasive surg. CogI, Indiv, MLS In:Instructor ST, SPc 11.5 3
and Accuracy during Resident Acquisition of Simulated intensity
Laparoscopic Skills. Curr Surg. 2001; 58:230–235.
Torkington J, et al. Skill transfer from virtual reality to a real MS 20 RCT Min. invasive surg. CogI, FB, Indiv CM:VR vs model ST, SPc 12.5 4
laparoscopic task. Surg Endosc. 2001; 15:1076–1079.
Agazio JB, et al. Evaluation of a virtual reality simulator in MD, RN, EMT 51 RCT Venous access ClinV, CogI, FB, CM:VR vs model Sa, ST, SPc 11.5 3
sustainment training. Mil Med. 2002; 167:893–897. Indiv
Block EF, et al. Use of a human patient simulator for the advanced O 14X Resuscitation CM:Manikin vs Sa 7 2
trauma life support course. Am Surg. 2002; 68:648–651. (BLS,ACLS,ATLS) animal
Byrne AJ, et al. Effect of videotape feedback on anaesthetists’ PG 32 RCT Anesthesia CogI, Indiv, MLS ID:Feedback ST, SPc 11.5 4
performance while managing simulated anaesthetic crises: a
multicentre study. Anaesthesia. 2002; 57:176–179.
Chang KK, et al. Learning intravenous cannulation: a comparison of RN, RN 28 RCT Venous access FB CM:VR vs model BP, P 16 4
the conventional method and the CathSim Intravenous Training
System. J Clin Nurs. 2002; 11:73–78.
Clancy JM, et al. A comparison of student performance in a D 186X Dentistry CM:Manikin vs SPd 12.5 4
simulation clinic and a traditional laboratory environment: three- model
year results. J Dent Educ. 2002; 66:1331–1337.
Hamilton EC, et al. Comparison of video trainer and virtual reality PG 19 RCT Min. invasive surg. CogI, FB, Indiv, CM:VR vs model Sa, ST, SPc, BP 14 5
training systems on acquisition of laparoscopic skills. Surg RangeD
Endosc. 2002; 16:406–411.
Harold KL, et al. Prospective randomized evaluation of surgical PG 17 RCT Min. invasive surg. CogI, FB, MLS In:Self-instruction ST, SPc 11.5 3
resident proficiency with laparoscopic suturing after course
instruction. Surg Endosc. 2002; 16:1729–1731.

(continued)
Instructional design features for simulation

e881
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Table A1. Continued.

e882
Participants Research comparisons Quality

Citation (sorted by year then author) Trainee N RCT Topic Features Theme Outcomes MERSQI NOS
D. A. Cook et al.

Kothari SN, et al. Training in laparoscopic suturing skills using a new MS 24 RCT Min. invasive surg. CogI, FB, Indiv, CM:VR vs model ST 12.5 4
computer-based virtual reality simulator (MIST-VR) provides RangeD
results comparable to those with an established pelvic trainer
system. J Laparoendosc Adv Surg Tech. 2002; 12:167–173.
Matsumoto ED, et al. The effect of bench model fidelity on MS 33 RCT Endoscopy CM:Manikin vs ST, SPc, SPd 14.5 5
endourological skills: a randomized controlled study. J Urol. (GI,Urology,Bronch.) model
2002; 167:1243–1247.
Scaringe JG, et al. The effects of augmented sensory feedback C 71 RCT Physical exam FB ID:Feedback SPc 11.5 4
precision on the acquisition and retention of a simulated
chiropractic task. J Manipulative Physiol Ther. 2002; 25:34–41.
Allen J, et al. A teaching tool in spinal anesthesia. AANA J. 2003; RN 26X RCT Percutaneous proc. CogI, FB ID:Feedback ST, SPd 12.5 3
71:29–36.
Ameur S, et al. Learning bronchoscopy in simulator improved MS 20 Endoscopy CogI ID:Clinical scenario Sa, SPc 10.5 1
dexterity rather than judgement [Swedish]. Lakartidningen. 2003; (GI,Urology,Bronch.)
100:2694–2699.
Engum SA, et al. Intravenous catheter training system: computer- MS, RN 163 Venous access CogI, FB CM:VR vs model Sa, K, SPc 11.5 4
based education versus traditional learning methods. Am J Surg.
2003; 186:67–74.
Gerling GJ, et al. Effect of augmented visual performance feedback MS 6 Physical exam CogI, FB ID:Feedback SPc 11 2
on the effectiveness of clinical breast examination training with a
dynamically configurable breast model. Conf Proc IEEE Int Conf
Syst Man Cybern. 2003; 3:2095–2100.
Gerling GJ, et al. Effectiveness of a dynamic breast examination MS 48 RCT Physical exam ClinV, CogI, MLS, ID:Task variability SPd 12.5 4
training model to improve clinical breast examination (CBE) skills. RangeD
Cancer Detect Prev. 2003; 27:451–456.
Gormley GJ, et al. A randomised study of two training programmes MD 38 RCT Percutaneous proc. ClinV, CogI, MLS, MA:Patient/SP SPc, BP 12 4
for general practitioners in the techniques of shoulder injection. RangeD, Time experience, add
Ann Rheum Dis. 2003; 62:1006–1009.
Katz R, et al. Cadaveric versus porcine models in urological PG, MD 16 Min. invasive surg. CM:Cadaver vs Sa 7 1
laparoscopic training. Urulogia Internationalis. 2003; 71:310– animal
315.
Lee SK, et al. Trauma assessment training with a patient simulator: a PG 60 RCT Resuscitation MA:Patient/SP SPc 14 3
prospective, randomized study. Journal of Trauma: Injury (BLS,ACLS,ATLS) experience,
Infection & Critical Care. 2003; 55:651–657. compare
Quinn F, et al. A pilot study comparing the effectiveness of D 32 RCT Dentistry ClinV, MLS CM:VR vs model SPd 14.5 5
conventional training and virtual reality simulation in the skills
acquisition of junior dental students. Eur J Dent Educ. 2003;
7:13–19.
Quinn F, et al. A study comparing the effectiveness of conventional D 20 RCT Dentistry FB In:Self-instruction SPd 13.5 3
training and virtual reality simulation in the skills acquisition of
junior dental students. Eur J Dent Educ. 2003; 7:164–9.
Buchanan JA. Experience with virtual reality-based technology in D 44 RCT Dentistry FB, Time CM:VR vs model SPd 12.5 5
teaching restorative dental procedures. J Dent Educ. 2004;
68:1258–1265.
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Grober ED, et al. The educational impact of bench model fidelity on PG 40 RCT Microsurgery/ CM:Synthetic vs ST, SPc, SPd 13.5 5
the acquisition of technical skill: the use of clinically relevant Ophthalmology human/animal
outcome measures. Ann Surg. 2004; 240:374–381. products
Hochberger J, et al. The compact Erlangen Active Simulator for PG, MD, RN, O 207 Endoscopy CM:Model vs model Sa 10 2
Interventional Endoscopy: a prospective comparison in struc- (GI,Urology,Bronch.)
tured team-training courses on ‘endoscopic hemostasis’ for
doctors and nurses. Scand J Gastroenterol. 2004; 39:895–902.
Jasinevicius TR, et al. An evaluation of two dental simulation D 28 RCT Dentistry ClinV, FB CM:VR vs model SPd 13.5 4
systems: virtual reality versus contemporary non-computer-
assisted. J Dent Educ. 2004; 68:1151–1162.
Kim HK, et al. Virtual-reality-based laparoscopic surgical training: O 16 Min. invasive surg. SA:Tactile SPc 11.5 2
The role of simulation fidelity in haptic feedback. Comput Aided
Surg. 2004; 9:227–234.
Martin KM, et al. Effective nonanatomical endoscopy training PG, MD, O 40 RCT Endoscopy CM:Model vs model ST, SPc 14.5 5
produces clinical airway endoscopy proficiency. Anesth Analg. (GI,Urology,Bronch.)
2004; 99:938–44.
Munz Y, et al. Laparoscopic virtual reality and box trainers: is one MS 16 RCT Min. invasive surg. CM:VR vs model ST, SPc 12.5 5
superior to the other? Surg Endosc. 2004; 18:485–494.
Nishida M, et al. Training in tooth preparation utilizing a support D 10 RCT Dentistry MA:External sup- SPd 10.5 4
system. J Oral Rehabil. 2004; 31:149–154. port, add
Rumball C, et al. Endotracheal intubation and esophageal tracheal EMT 61 Intubation DistP, Indiv, Time ID:Repetition P 14 2
Combitube insertion by regular ambulance attendants: A com-
parative trial. Prehosp Emerg Care. 2004; 8:15–22.
St. Pierre M, et al. Simulator-based modular human factor training in MD 34 RCT Critical thinking CogI, MLS MA:Discussion, Sa, SPc 12.5 3
anesthesiology. Concept and results of the module add
‘‘Communication and Team Cooperation’’ [German].
Anaesthesist. 2004; 53:144–152.
Backstein D, et al. Effectiveness of repeated video feedback in the PG 26 RCT Open surgery/suturing CogI, FB, MLS ID:Feedback SPc 15.5 3
acquisition of a surgical technical skill. Can J Surg. 2005;
48:195–200.
Bathalon S, et al. Cognitive skills analysis, kinesiology, and mental MS 31 RCT Resuscitation CogI, FB, MLS, ID:Teach cognition SPc 12.5 4
imagery in the acquisition of surgical skills. J Otolaryngol. 2005; (BLS,ACLS,ATLS),Per- RepP
34:328–332. cutaneous proc.
Bowyer CM, et al. Validation of SimPL – a simulator for diagnostic MS 40 RCT Resuscitation CM:VR vs animal K, SPc 12.5 4
peritoneal lavage training. Stud Health Technol Inform. 2005; (BLS,ACLS,ATLS)
111:64–67.
Bowyer MW, et al. Teaching intravenous cannulation to medical MS 14 RCT Venous access ClinV, RangeD CM:VR vs VR SPc 13.5 4
students: comparative analysis of two simulators and two
traditional educational approaches. Stud Health Technol Inform.
2005; 111:57–63.
Donnon T, et al. Impact of cognitive imaging and sex differences on MS 42 RCT Min. invasive surg. CogI, MLS ID:Teach cognition ST 12.5 3
the development of laparoscopic suturing skills. Can J Surg.
2005; 48:387–393.
Dubrowski A, et al. The influence of practice schedules in the MS 19 RCT Open surgery/suturing ID:Sequence SPc, SPd 12.5 3
learning of a complex bone-plating surgical task. Am J Surg.
2005; 190:359–363.
Enebo B, et al. Experience and practice organization in learning a C 33X Spinal manipulation ID:Sequence SPc 12.5 2
simulated high-velocity low-amplitude task. J Manipulative
Physiol Ther. 2005; 28:33–43.
Gerling GJ, et al. Augmented, pulsating tactile feedback facilitates MS 48 RCT Physical exam Indiv, MLS, RangeD ID:Task variability SPd 13.5 3
simulator training of clinical breast examinations. Hum Factors.
2005; 47:670–681.
Instructional design features for simulation

e883
(continued)
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Table A1. Continued.

e884
Participants Research comparisons Quality

Citation (sorted by year then author) Trainee N RCT Topic Features Theme Outcomes MERSQI NOS
D. A. Cook et al.

Kiesslich R, et al. Combined simulation training: a new concept and MD 100X Endoscopy CogI, FB, GrpP, CM:VR vs animal Sa 11 3
workshop is useful for crisis management in gastrointestinal (GI,Urology,Bronch.),T- MLS, RepP
endoscopy [German]. Z Gastroenterol. 2005; 43:1031–1039. eam training
Kirlum HJ, et al. Advanced paediatric laparoscopic surgery: repet- PG 12 RCT Min. invasive surg. CM:Model vs ST 9.5 2
itive training in a rabbit model provides superior skills for live animal
operations. Eur J Pediatr Surg. 2005; 15:149–152.
Lehmann KS, et al. A prospective randomized study to test the MS, MD 32 RCT Min. invasive surg. CM:VR vs model ST, SPc 12.5 2
transfer of basic psychomotor skills from virtual reality to physical
reality in a comparable training setting. Ann Surg. 2005;
241:442–449.
Lentz GM, et al. A six-year study of surgical teaching and skills PG 27 Min. invasive ID:Timing K, SPc 14.5 4
evaluation for obstetric/gynecologic residents in porcine and surg.,Open surgery/
inanimate surgical models. Am J Obstet Gynecol. 2005; suturing
193:2056–2061.
Madan AK, et al. Participants’ opinions of laparoscopic training MS 18X Min. invasive surg. CM:VR vs model Sa 8 1
devices after a basic laparoscopic training course. Am J Surg.
2005; 189:758–61.
Monsieurs KG, et al. Improved basic life support performance by RN 152 RCT Resuscitation CogI, FB MA:External sup- SPc 12.5 5
ward nurses using the CAREventÕ Public Access Resuscitator (BLS,ACLS,ATLS) port, add
(PAR) in a simulated setting. Resuscitation. 2005; 67:45–50.
Tani Botticelli A, et al. The effectiveness of video support in the O 84 RCT Dentistry CogI MA:CAI, compare SPc 12.5 3
teaching of manual skills related to initial periodontal therapy
tested on phantoms. Int J Comput Dent. 2005; 8:117–127.
Uchal M, et al. Validation of a six-task simulation model in minimally PG 17 RCT Min. invasive surg. ID:Sequence SPc 15.5 6
invasive surgery. Surg Endosc. 2005; 19:109–16.
Wierinck E, et al. Effect of augmented visual feedback from a virtual D 24 RCT Dentistry FB, Indiv ID:Feedback ST, SPd 12.5 4
reality simulation system on manual dexterity training. Eur J Dent
Educ. 2005; 9:10–16.
Youngblood PL, et al. Comparison of training on two laparoscopic MS 33 RCT Min. invasive surg. CogI, FB CM:VR vs model ST, SPc 13.5 4
simulators and assessment of skills transfer to surgical perfor-
mance. J Am Coll Surg. 2005; 200:546–551.
Aggarwal R, et al. A competency-based virtual reality training MS 20 RCT Min. invasive surg. RangeD, Time ID:Sequence ST 12.5 5
curriculum for the acquisition of laparoscopic psychomotor skill.
Am J Surg. 2006; 191:128–133.
Berkenstadt H, et al. Evaluation of the Trauma-ManÕ simulator for PG 42X RCT Resuscitation CM:Manikin vs Sa 8 4
training in chest drain insertion. European Journal of Trauma. (BLS,ACLS,ATLS) animal
2006; 32:523–526.
Bond WF, et al. Cognitive versus technical debriefing after simulation PG 62 RCT Critical thinking ID:Teach cognition Sa 10 4
training. Acad Emerg Med. 2006; 13:276–283.
Chandrasekera SK, et al. Basic laparoscopic surgical training: MS 36 RCT Min. invasive surg. CM:Model vs model ST, SPc, SPd 12.5 5
examination of a low-cost alternative. Eur Urol. 2006; 50:1285–
1291.
Chou DS, et al. Comparison of results of virtual-reality simulator and MS 16 RCT Endoscopy RangeD CM:VR vs model SPc 12.5 4
training model for basic ureteroscopy training. J Endourol. 2006; (GI,Urology,Bronch.)
20:266–271.
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Crofts JF, et al. Training for shoulder dystocia: a trial of simulation PG, MD, O 110 RCT Obstetrics CogI, FB, Indiv, CM:Manikin vs ST, SPc, SPd 13.5 5
using low-fidelity and high-fidelity mannequins. Obstet Gynecol. MLS manikin
2006; 108:1477–1485.
Cummings AJ, et al. Evaluation of a novel animal model for teaching MS, PG, MD, RN, EMT, O 42X RCT Intubation CM:Model vs Sa, ST 8.5 2
intubation. Teach Learn Med. 2006; 18:316–319. animal
Dubrowski A, et al. Randomised, controlled study investigating the MS 72 RCT Open surgery/suturing CogI, FB In:Instructor ST, SPc 12.5 4
optimal instructor: student ratios for teaching suturing skills. Med intensity
Educ. 2006; 40:59–63.
Halvorsen FH, et al. Virtual reality simulator training equals MS 26 RCT Robotic surg. CM:VR vs model Sa, SPc 11.5 2
mechanical robotic training in improving robot-assisted basic
suturing skills. Surg Endosc. 2006; 20:1565–1569.
Heinrich M, et al. Comparison of different training models for PG 12 RCT Min. invasive surg. CM:Model vs SPc 12.5 2
laparoscopic surgery in neonates and small infants. Surg animal
Endosc. 2006; 20:641–644.
Jamison RJ, et al. A pilot study assessing simulation using two RN 18 RCT Venous access CM:VR vs model K 14.5 3
simulation methods for teaching intravenous cannulation. Clinical
Simulation in Nursing Education. 2006; 2:e9–e12.
Kimura T, et al. Usefulness of a virtual reality simulator or training box MS 12 Min. invasive surg. CM:VR vs model ST, SPc 11.5 2
for endoscopic surgery training. Surg Endosc. 2006; 20:656–
659.
Mazilu D, et al. Synthetic torso for training in and evaluation of MS, PG, MD, O 25 RCT Min. invasive surg. CM:Model vs model Sa 8 3
urologic laparoscopic skills. J Endourol. 2006; 20:340–345.
Ocel JJ, et al. Formal procedural skills training using a fresh frozen MS 7X Venous CM:Manikin vs Sa 7 2
cadaver model: a pilot study. Clin Anat. 2006; 19:142–146. access,Percutaneous cadaver
proc.,Natural orifice
proc.
Owen H, et al. Comparison of three simulation-based training PG 50 RCT Resuscitation CogI, Indiv, MLS CM:Manikin vs SPc 13 5
methods for management of medical emergencies. (BLS,ACLS,ATLS) manikin
Resuscitation. 2006; 71:204–211.
Panait L, et al. Telementoring versus on-site mentoring in virtual MS 20 RCT Min. invasive surg. In:Distance ST, SPc 13 4
reality-based surgical training. Surg Endosc. 2006; 20:113–118. supervision
Rosenthal ME, et al. Achieving housestaff competence in emer- PG 49 RCT Resuscitation In:Instructor train- SPc 12.5 4
gency airway management using scenario based simulation (BLS,ACLS,ATLS) ing/experience
training: comparison of attending vs housestaff trainers. Chest.
2006; 129:1453–1458.
Rosser JC, Jr., et al. The use of a ‘‘hybrid’’ trainer in an established PG, MD 817 Min. invasive surg. FB, Indiv, MLS ID:Feedback ST 10.5 1
laparoscopic skills program. J Soc Laparoendosc Surg. 2006;
10:4–10.
Savoldelli GL, et al. Value of debriefing during simulated crisis PG 28 RCT Anesthesia,Team CogI, FB, Indiv, ID:Feedback SPc 14.5 5
management: oral versus video-assisted oral feedback. training MLS
Anesthesiology. 2006; 105:279–285.
Scerbo MW, et al. Comparison of a virtual reality simulator and MS 16 Venous access ClinV, RangeD, CM:VR vs model SPc, BP 12 2
simulated limbs for phlebotomy training. J Infus Nurs. 2006; Time
29:214–224.
Scerbo MW, et al. The efficacy of a medical virtual reality simulator MS 20 RCT Venous access ClinV, RangeD CM:VR vs model SPc 12.5 3
for training phlebotomy. Hum Factors. 2006; 48:72–84.
Stefanidis D, et al. Proficiency maintenance: impact of ongoing MS 18 RCT Min. invasive surg. CogI, Time ID:Repetition SPc 11.5 3
simulator training on laparoscopic skill retention. J Am Coll Surg.
2006; 202:599–603.

(continued)
Instructional design features for simulation

e885
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Table A1. Continued.

e886
Participants Research comparisons Quality

Citation (sorted by year then author) Trainee N RCT Topic Features Theme Outcomes MERSQI NOS
D. A. Cook et al.

Strom P, et al. Early exposure to haptic feedback enhances PG 19X RCT Min. invasive surg. SA:Tactile SPc 11.5 3
performance in surgical simulator training: a prospective ran-
domized crossover study in surgical residents. Surg Endosc.
2006; 20:1383–1388.
Ti L-K, et al. The impact of experiential learning on NUS medical MS 54 Anesthesia CogI ID:Hands on SPc 11.5 3
students: our experience with task trainers and human-patient
simulation. Ann Acad Med Singapore. 2006; 35:619–623.
Verdaasdonk EGG, et al. Validation of a new basic virtual reality MS 16 RCT Min. invasive surg. CM:VR vs model ST, SPc 13.5 4
simulator for training of basic endoscopic skills: The SIMENDO.
Surg Endosc. 2006; 20:511–518.
Wierinck E, et al. Effect of reducing frequency of augmented D 24 RCT Dentistry FB, MLS ID:Feedback ST, SPd 12.5 4
feedback on manual dexterity training and its retention. J Dent.
2006; 34:641–647.
Wierinck E, et al. Effect of tutorial input in addition to augmented D 24 RCT Dentistry FB, MLS In:Self-instruction ST, SPd 12.5 4
feedback on manual dexterity training and its retention. Eur J
Dent Educ. 2006; 10:24–31.
Adermann J, et al. The impact of force feedback on training of MS, MD 60 RCT Endoscopy FB SA:Tactile ST, SPc 11.5 3
surgical skills in virtual neuroendoscopy. Int J Comput Assist (GI,Urology,Bronch.)
Radiol Surg. 2007; 2(Suppl 1):S198–S200.
Berry M, et al. Porcine transfer study: virtual reality simulator training MD 12X RCT Endovascular proc. CM:VR vs animal Sa, SPc 12.5 2
compared with porcine training in endovascular novices.
Cardiovasc Intervent Radiol. 2007; 30:455–461.
Birch L, et al. Obstetric skills drills: evaluation of teaching methods. PG, O 4 RCT Obstetrics MLS ID:Blending SPc 13.5 5
Nurse Educ Today. 2007; 27:915–922.
Bruynzeel H, et al. Desktop simulator: key to universal training? Surg MS 20 RCT Min. invasive surg. CM:Model vs model ST, SPd 12.5 4
Endosc. 2007; 21:1637–1640.
Chang J-Y, et al. Effectiveness of two forms of feedback on training O 24 RCT Physical therapy CogI, FB, Indiv, ID:Feedback SPc 12.5 5
of a joint mobilization skill by using a joint translation simulator. MLS
Phys Ther. 2007; 87:418–430.
Cherry RA, et al. The effectiveness of a human patient simulator in PG 44 RCT Resuscitation CM:Manikin vs K, SPc 13.5 4
the ATLS shock skills station. J Surg Res. 2007; 139:229–235. (BLS,ACLS,ATLS) model
Crofts JF, et al. Change in knowledge of midwives and obstetricians PG, MD, O 133 RCT Obstetrics CogI, FB, Indiv, CM:Manikin vs K 14.5 5
following obstetric emergency training: a randomised controlled MLS manikin
trial of local hospital, simulation centre and teamwork training.
BJOG. 2007; 114:1534–1541.
Davis DP, et al. The effectiveness of a novel, algorithm-based RN, EMT 120X Intubation ClinV, MLS, Time MA:Simulator, add SPc, P 14 2
difficult airway curriculum for air medical crews using human
patient simulators. Prehosp Emerg Care. 2007; 11:72–79.
Dubrowski A, et al. A comparison of single- and multiple-stage PG 24 RCT Min. invasive surg. MLS, RangeD ID:Task variability ST, SPc, SPd 13.5 3
approaches to teaching laparoscopic suturing. Am J Surg. 2007;
193:269–273.
Gutierrez F, et al. The effect of degree of immersion upon learning MS 25 RCT Resuscitation CM:VR vs VR K 13.5 5
performance in virtual reality simulations for medical education. (BLS,ACLS,ATLS),Phy-
Stud Health Technol Inform. 2007; 125:155–160. sical exam
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Heinrichs WL, et al. Criterion-based training with surgical simulators: MD 17X RCT Min. invasive surg. CM:VR vs VR Sa 8 4
proficiency of experienced surgeons. J Soc Laparoendosc Surg.
2007; 11:273–302.
Iglesias-Vazquez JA, et al. Cost-efficiency assessment of Advanced MD, RN 250 RCT Resuscitation CM:Manikin vs SPc 12 2
Life Support (ALS) courses based on the comparison of (BLS,ACLS,ATLS) manikin
advanced simulators with conventional manikins. BMC
Emergency Medicine. 2007; 7:18.
Immenroth M, et al. Mental training in surgical education: a MD 66 RCT Min. invasive surg. CogI, MLS ID:Teach cognition SPc 13.5 6
randomized controlled trial. Ann Surg. 2007; 245:385–391.
Kahol K, et al. Augmented virtual reality for laparoscopic surgical tool PG 8 Min. invasive surg. CogI, FB ID:Feedback SPc 11.5 2
training. Lect Notes Comput Sci. 2007; 4553 LNCS:459–467.
Lazarski MP, et al. How do feedback and instructions affect the MS, D 61 Open surgery/suturing MLS ID:Instructions SPd 11.5 3
performance of a simulated surgical task? J Oral Maxillofac Surg.
2007; 65:1155–61.
LeFlore JL, et al. Comparison of self-directed learning versus RN 11 Resuscitation CogI, FB, Indiv In:Self-instruction K, ST, SPc 13.5 5
instructor-modeled learning during a simulated clinical experi- (BLS,ACLS,ATLS),Tea-
ence. Simul Healthc. 2007; 2:170–177. m training
Lee JC, et al. Randomized controlled trial of an instructional DVD for MS, PG, MD, RN 36 RCT Venous access CogI, FB, Indiv MA:Lecture, Sa, SPc 12.5 5
clinical skills teaching. Emerg Med Australas. 2007; 19:241–245. compare
Madan AK, et al. Prospective randomized controlled trial of MS 32 RCT Min. invasive surg. MA:Simulator, add ST, SPc 12.5 5
laparoscopic trainers for basic laparoscopic skills acquisition.
Surg Endosc. 2007; 21:209–213.
Neequaye SK, et al. Identification of skills common to renal and iliac PG 20 RCT Endovascular proc. CogI ID:Task variability ST, SPc 11.5 4
endovascular procedures performed on a virtual reality simulator.
Eur J Vasc Endovasc Surg. 2007; 33:525–532.
Schlosser K, et al. Training of laparoscopic skills with virtual reality MS 14 RCT Min. invasive surg. ID:Task variability ST, SPc 12.5 4
simulator: a critical reappraisal of the learning curve. European
Surgery – Acta Chirurgica Austriaca. 2007; 39:180–184.
Sidhu RS, et al. Laboratory-based vascular anastomosis training: a PG 27 RCT Open surgery/suturing CM:Model vs ST, SPc, SPd 12.5 4
randomized controlled trial evaluating the effects of bench model cadaver
fidelity and level of training on skill acquisition. J Vasc Surg. 2007;
45:343–349.
Spooner BB, et al. An evaluation of objective feedback in basic life O 98 RCT Resuscitation CogI, FB ID:Feedback SPc, SPd 13.5 5
support (BLS) training. Resuscitation. 2007; 73:417–424. (BLS,ACLS,ATLS)
Stefanidis D, et al. Closing the gap in operative performance MS 25 RCT Min. invasive surg. CogI, Time ID:Stress SPc 12.5 4
between novices and experts: does harder mean better for
laparoscopic simulator training? J Am Coll Surg. 2007; 205:307–
313.
Stefanidis D, et al. Construct and face validity and task workload for PG, MD, O 90X Min. invasive surg. CM:VR vs model Sa, SPc 12.5 3
laparoscopic camera navigation: virtual reality versus videotrainer
systems at the SAGES Learning Center. Surg Endosc. 2007;
21:1158–1164.
Stefanidis D, et al. Limited feedback and video tutorials optimize MS 22 Min. invasive surg. FB, Indiv ID:Feedback SPc 11.5 1
learning and resource utilization during laparoscopic simulator
training. Surgery. 2007; 142:202–206.
Thomas EJ, et al. Teaching teamwork during the Neonatal PG 32 RCT Team training MLS MA:Team training, SPc 13.5 5
Resuscitation Program: a randomized trial. J Perinatol. 2007; add
27:409–414.
Torgerson C, et al. Low fidelity simulation of temporal bone drilling MS 24 Open surgery/suturing ID:Task variability SPd 9.5 1
leads to improved but suboptimal outcomes. Stud Health
Technol Inform. 2007; 125:470–472.

(continued)
Instructional design features for simulation

e887
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Table A1. Continued.

e888
Participants Research comparisons Quality

Citation (sorted by year then author) Trainee N RCT Topic Features Theme Outcomes MERSQI NOS
D. A. Cook et al.

Van Sickle KR, et al. The effect of escalating feedback on the MS, O 16 RCT Min. invasive surg. ID:Feedback SPc 13.5 4
acquisition of psychomotor skills for laparoscopy. Surg Endosc.
2007; 21:220–224.
Verdaasdonk EGG, et al. The influence of different training sched- MS, O 20 RCT Min. invasive surg. DistP ID:Timing ST, SPc 11.5 3
ules on the learning of psychomotor skills for endoscopic
surgery. Surg Endosc. 2007; 21:214–219.
Welk A, 21et al. Mental training in dentistry. Quintessence Int. 2007; D 41 RCT Dentistry MLS ID:Teach cognition K, SPc, SPd 13.5 5
38:489–497.
Xeroulis GJ, et al. Teaching suturing and knot-tying skills to medical MS 30 RCT Open surgery/suturing CogI, FB, Indiv, MA:CAI, compare SPc 13.5 4
students: a randomized controlled study comparing computer- MLS
based video instruction and (concurrent and summary) expert
feedback. Surgery. 2007; 141:442–449.
Baranauskas MB, et al. Simulation of ultrasound-guided peripheral PG 6 RCT Anesthesia,Radiology/ Time ID:Repetition ST, SPc 9.5 4
nerve block: learning curve of CET-SMA/HSL Anesthesiology other noninvasive dx
residents. Rev Bras Anestesiol. 2008; 58:106–111.
Bingener J, et al. Randomized double-blinded trial investigating the MS 30 RCT Min. invasive surg. MLS ID:Feedback ST, SPc 15.5 4
impact of a curriculum focused on error recognition on laparo-
scopic suturing training. Am J Surg. 2008; 195:179–182.
Botden SMBI, et al. The importance of haptic feedback in laparo- PG, MD 45X RCT Min. invasive surg. ID:Sequence Sa, SPc, SPd 12 4
scopic suturing training and the additive value of virtual reality
simulation. Surg Endosc. 2008; 22:1214–1222.
Chandra DB, et al. Fiberoptic oral intubation: the effect of model O 28 RCT Endoscopy CM:VR vs model BT, BP, P 16 5
fidelity on training for transfer to patient care. Anesthesiology. (GI,Urology,Bronch.)
2008; 109:1007–1013.
Chang S, et al. Verbal communication improves laparoscopic team MS, PG, O 24 RCT Min. invasive surg. CogI, FB, Indiv ID:Feedback ST, SPc 12.5 2
performance. Surgical Innovation. 2008; 15:143–147.
Chmarra MK, et al. Force feedback and basic laparoscopic skills. PG 19 RCT Min. invasive surg. SA:Tactile ST, SPc 11.5 4
Surg Endosc. 2008; 22:2140–2148.
Cho J, et al. Comparison of manikin versus porcine models in PG, MD, RN, EMT 49X RCT Open surgery/ CM:Model vs Sa 9 2
cricothyrotomy procedure training. Emerg Med J. 2008; 25:732– suturing,Intubation animal
734.
Crofts JF, et al. Patient-actor perception of care: a comparison of MD, O 64 RCT Obstetrics CogI, FB, Indiv CM:Manikin vs SPc 13.5 4
obstetric emergency training using manikins and patient-actors. manikin
Qual Saf Health Care. 2008; 17:20–24.
De Regge M, et al. Basic life support refresher training of nurses: RN 103 RCT Resuscitation FB, GrpP, MLS GC:Solo vs group SPc 12.5 6
individual training and group training are equally effective. (BLS,ACLS,ATLS)
Resuscitation. 2008; 79:283–287.
Dine CJ, et al. Improving cardiopulmonary resuscitation quality and RN 65 RCT Resuscitation CogI, FB, Indiv, ID:Feedback K, SPc 12.5 5
resuscitation training by combining audiovisual feedback and (BLS,ACLS,ATLS) MLS
debriefing. Crit Care Med. 2008; 36:2817–2822.
Ellis D, et al. Hospital, simulation center, and teamwork training for MD, O 24 RCT Obstetrics CM:Manikin vs ST 14.5 5
eclampsia management: a randomized controlled trial. Obstet manikin
Gynecol. 2008; 111:723–731.
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Friedman Z, et al. Teaching lifesaving procedures: the impact of PG 22 RCT Intubation CM:Manikin vs ST, SPc 12.5 4
model fidelity on acquisition and transfer of cricothyrotomy skills model
to performance on cadavers. Anesth Analg. 2008; 107:1663–
1669.
Grady JL, et al. Learning nursing procedures: the influence of RN 39X Natural orifice proc. CM:Manikin vs Sa, SPc 13.5 3
simulator fidelity and student gender on teaching effectiveness. J model
Nurs Educ. 2008; 47:403–408.
Grodin MH, et al. Ophthalmic surgical training: a curriculum to PG, MD 45 RCT Microsurgery/ MA:CAI, compare SPc 13.5 4
enhance surgical simulation. Retina. 2008; 28:1509–1514. Ophthalmology
Isbye DL, et al. Voice advisory manikin versus instructor facilitated MS 43 RCT Resuscitation In:Self-instruction SPc 13.5 5
training in cardiopulmonary resuscitation. Resuscitation. 2008; (BLS,ACLS,ATLS)
79:73–81.
Jiang C, et al. A training model for laparoscopic urethrovesical PG 40 RCT Min. invasive CM:Animal prod- ST, SPc 12.5 4
anastomosis. J Endourol. 2008; 22:1541–1545. surg.,Endoscopy ucts vs animal
(GI,Urology,Bronch.) products
Johnson KB, et al. Part Task and variable priority training in first-year PG 21 RCT Anesthesia CogI, FB, MLS ID:Sequence K, SPc, SPd 14.5 4
anesthesia resident education: a combined didactic and
simulation-based approach to improve management of adverse
airway and respiratory events. Anesthesiology. 2008; 108:831–
840.
Kanumuri P, et al. Virtual reality and computer-enhanced training MS 16 RCT Min. invasive surg. CM:VR vs VR ST, SPc, SPd 13.5 4
devices equally improve laparoscopic surgical skill in novices. J
Soc Laparoendosc Surg. 2008; 12:219–226.
Lammers RL. Learning and retention rates after training in posterior PG 28 RCT Epistaxis management FB, MLS, Time ID:Sequence ST, SPc 14.5 5
epistaxis management. Acad Emerg Med. 2008; 15:1181–9.
LeFlore JL, et al. Effectiveness of 2 methods to teach and evaluate RN, EMT, O 24 Resuscitation CogI, FB, Indiv ID:Hands on Sa, SPc 13.5 2
new content to neonatal transport personnel using high-fidelity (BLS,ACLS,ATLS)
simulation. J Perinat Neonatal Nurs. 2008; 22:319–328.
Low D, et al. The use of the BERCI DCIÕ Video Laryngoscope for MS, EMT 42 RCT Intubation CogI, FB SA:Visual ST, SPd 12.5 5
teaching novices direct laryngoscopy and tracheal intubation.
Anaesthesia. 2008; 63:195–201.
Miotto HC, et al. Advanced Cardiac Life Support Courses: live actors MD, RN, O 225 RCT Resuscitation MLS MA:Patient/SP K 12 5
do not improve training results compared with conventional (BLS,ACLS,ATLS) experience, add
manikins. Resuscitation. 2008; 76:244–248.
Murphy MA, et al. Should we train the trainers? Results of a MS 30 RCT Venous access CogI, Indiv, MLS In:Instructor train- ST, SPc 14.5 5
randomized trial. J Am Coll Surg. 2008; 207:185–190. ing/experience
Nousiainen M, et al. Comparison of expert instruction and MS 16 RCT Open surgery/suturing CogI, FB, Indiv, ID:Interactivity ST, SPc 13.5 4
computer-based video training in teaching fundamental surgical MLS
skills to medical students. Surgery. 2008; 143:539–544.
O’Connor A, et al. How much feedback is necessary for learning to MS 6 RCT Min. invasive surg. CogI, FB, Indiv, ID:Feedback SPd 11.5 2
suture? Surg Endosc. 2008; 22:1614–1619. MLS
Pierce J, et al. Comparative usability studies of full vs. partial MS 25 RCT Resuscitation CM:VR vs VR Sa 10 4
immersive virtual reality simulation for medical education and (BLS,ACLS,ATLS)
training. Stud Health Technol Inform. 2008; 132:372–377.
Rafiq A, et al. Objective assessment of training surgical skills using MS 12 RCT Open surgery/suturing CogI, FB, Indiv, ID:Feedback SPc 11.5 2
simulated tissue interface with real-time feedback. J Surg Educ. MLS
2008; 65:270–274.
Reyes SD, et al. Implementation and evaluation of a virtual simulator RN 28 RCT Venous access CogI, DistP CM:VR vs model Sa, K, SPc 11.5 2
system: teaching intravenous skills. Clinical Simulation in Nursing
Education. 2008; 4:e43–e49.

(continued)
Instructional design features for simulation

e889
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Table A1. Continued.

e890
Participants Research comparisons Quality

Citation (sorted by year then author) Trainee N RCT Topic Features Theme Outcomes MERSQI NOS
D. A. Cook et al.

Rissanen MJ, et al. Asynchronous teaching of psychomotor skills MS 8 Physical exam CogI, FB, Indiv, ID:Feedback SPc 10.5 2
through VR annotations: evaluation in digital rectal examination. MLS
Stud Health Technol Inform. 2008; 132:411–416.
Roberson DW, et al. Improving wound care simulation with the RN 99 Physical exam,wound SA:Olfactory Sa 9 2
addition of odor: A descriptive, quasi-experimental study. care
Ostomy Wound Management. 2008; 54:36–43.
Sanders CW, et al. Learning basic surgical skills with mental MS 63 RCT Open surgery/suturing CogI ID:Teach cognition SPc 14.5 3
imagery: using the simulation centre in the mind. Med Educ.
2008; 42:607–612.
Tanoue K, et al. Effectiveness of endoscopic surgery training for MS 40 RCT Min. invasive surg. RangeD CM:VR vs model ST 12.5 4
medical students using a virtual reality simulator versus a box
trainer: a randomized controlled trial. Surg Endosc. 2008;
22:985–990.
Tzafestas CS, et al. Pilot evaluation study of a virtual paracentesis MD, RN 20 RCT Venous access CogI, Indiv SA:Tactile ST, SPc 12.5 3
simulator for skill training and assessment: the beneficial effect of
haptic display. Presence: Teleoperators and Virtual
Environments. 2008; 17:212–229.
Van Herzeele I, et al. Cognitive training improves clinically relevant PG 20 Endovascular proc. MLS, Time MA:Lecture, add ST, SPd 10.5 2
outcomes during simulated endovascular procedures. J Vasc
Surg. 2008; 48:1223–1230.
Vankipuram M, et al. Virtual reality based training to resolve visio- PG 10 Min. invasive surg. RangeD SA:Tactile ST, SPc, SPd 11.5 2
motor conflicts in surgical environments. HAVE 2008 – IEEE
International Workshop on Haptic Audio Visual Environments and
Games Proceedings. 2008; Article number 4685290:7–12.
Wang XP, et al. Effect of emergency care simulator combined with MS 42 RCT Resuscitation ClinV, CogI MA:Simulator, add SPc 12.5 4
problem-based learning in teaching of cardiopulmonary resus- (BLS,ACLS,ATLS),Tea-
citation [Chinese]. Chung Hua I Hsueh Tsa Chih. 2008; 88:1651– m training
1653.
Wheeler DW, et al. Retention of drug administration skills after O 72 Physiology: MLS MA:CAI, add SPc 11.5 4
intensive teaching. Anaesthesia. 2008; 63:379–384. pharmacology
Youngquist ST, et al. Paramedic self-efficacy and skill retention in EMT 135 Intubation CogI, FB In:Self-instruction SPd 13.5 3
pediatric airway management. Acad Emerg Med. 2008;
15:1295–303.
de Vries W, et al. Self–training in the use of automated external RN 30 RCT Resuscitation CogI, FB, GrpP, In:Self-instruction SPc 10.5 2
defibrillators: the same results for less money. Resuscitation. (BLS,ACLS,ATLS) Indiv, MLS
2008; 76:76–82.
Butler KW, et al. Implementation of active learning pedagogy RN 30 RCT Physiology:fluid and CM:Manikin vs Sa 12 3
comparing low-fidelity simulation versus high-fidelity simulation in electrolyte manikin
pediatric nursing education. Clinical Simulation in Nursing. 2009;
5:e129–e136.
Campbell DM, et al. High-fidelity simulation in neonatal resuscitation. PG 15 RCT Resuscitation Indiv CM:Manikin vs Sa, K, ST 13.5 4
Paediatrics and Child Health. 2009; 14:19–23. (BLS,ACLS,ATLS) manikin
Carter YM, et al. Open lobectomy simulator is an effective tool for MS 18 Open surgery/suturing CogI, DistP, FB, ID:Repetition K, ST, SPc 11.5 2
teaching thoracic surgical skills. Ann Thorac Surg. 2009; Indiv, MLS, RepP,
87:1546–1550. Time
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Day T, et al. Effect of performance feedback on tracheal suctioning RN, EMT 38 RCT Natural orifice proc.,tra- ID:Feedback K, SPc 15 4
knowledge and skills: randomized controlled trial. J Adv Nurs. cheal suctioning
2009; 65:1423–1431.
Deladisma AM, et al. A pilot study to integrate an immersive virtual MS 21 RCT Physical exam DistP, MLS, Time MA:CAI, add SPc 10 4
patient with a breast complaint and breast examination simulator
into a surgery clerkship. Am J Surg. 2009; 197:102–106.
Deutsch ES, et al. Multimodality education for airway endoscopy skill PG 36X Endoscopy CM:VR vs manikin SPc 9.5 1
development. Ann Otol Rhinol Laryngol. 2009; 118:81–86. (GI,Urology,Bronch.)
Domuracki KJ, et al. Learning on a simulator does transfer to clinical MS, RN, RN 101 RCT Intubation FB, Indiv, Mast, ID:Feedback SPc 13.5 5
practice. Resuscitation. 2009; 80:346–349. MLS
Donoghue AJ, et al. Effect of high-fidelity simulation on Pediatric PG 51 RCT Resuscitation CM:Manikin vs SPc 14.5 4
Advanced Life Support training in pediatric house staff: a (BLS,ACLS,ATLS) manikin
randomized trial. Pediatr Emerg Care. 2009; 25:139–144.
Friedman Z, et al. Clinical impact of epidural anesthesia simulation PG 24 RCT Anesthesia,Percutane- ClinV CM:VR vs model BP 15 5
on short- and long-term learning curve: High- versus low-fidelity ous proc.,epidural
model training. Reg Anesth Pain Med. 2009; 34:229–232.
Gerling GJ, et al. The design and evaluation of a computerized and MS, RN 26 RCT Physical exam ClinV, FB, Indiv, CM:Model vs model SPc, SPd 11.5 2
physical simulator for training clinical prostate exams. IEEE RangeD
Transactions on Systems, Man, and Cybernetics Part A, Systems
and Humans. 2009; 39:388–403.
Girzadas DV, Jr., et al. Hybrid simulation combining a high fidelity PG 45 RCT Critical ID:Sequence Sa, ST 9 4
scenario with a pelvic ultrasound task trainer enhances the thinking,Obstetrics
training and evaluation of endovaginal ultrasound skills. Acad
Emerg Med. 2009; 16:429–435.
Helmy S, et al. Development of laparoscopic skills using a new PG 12X RCT Min. invasive surg. CM:Model vs model ST 13 3
inexpensive webcam trainer. Journal of Biological Sciences.
2009; 9:766–771.
Hoadley TA. Learning advanced cardiac life support: a comparison MD, RN, EMT, O 53 RCT Resuscitation CM:Manikin vs Sa, K, SPc 14 2
study of the effects of low- and high-fidelity simulation. Nurs (BLS,ACLS,ATLS) manikin
Educ Perspect. 2009; 30:91–95.
Jayaraman S, et al. Novel hands-free pointer improves instruction MS, PG, MD 10X Min. invasive surg. CogI, FB ID:Instructions ST 11.5 2
efficiency in laparoscopic surgery. Surgical Innovation. 2009;
16:73–77.
Jensen AR, et al. Acquiring basic surgical skills: Is a faculty mentor PG 44 RCT Open surgery/suturing CogI, FB In:Self-instruction ST, SPc, SPd 12.5 6
really needed? Am J Surg. 2009; 197:82–88.
Kahol K, et al. Cognitive simulators for medical education and PG 10 Min. invasive surg. CogI, RangeD ID:Additional SPc 12.5 2
training. Journal of Biomedical Informatics. 2009; 42:593–604. practice
Kardong-Edgren S, et al. VitalSimÕ versus SimManÕ : a comparison RN 76 RCT Resuscitation CM:Manikin vs Sa, K 14.5 4
of BSN student test scores, knowledge retention, and satisfac- (BLS,ACLS,ATLS) manikin
tion. Clinical Simulation in Nursing. 2009; 5:e105–e111.
Kishore TA, et al. Task deconstruction facilitates acquisition of MS 18 RCT Endoscopy RangeD ID:Sequence Sa, SPc 11.5 3
transurethral resection of prostate skills on a virtual reality trainer. (GI,Urology,Bronch.)
J Endourol. 2009; 23:665–668.
Kromann CB, et al. The effect of testing on skills learning. Med Educ. MS 81 RCT Resuscitation Indiv ID:Testing effect SPc 13.5 5
2009; 43:21–27. (BLS,ACLS,ATLS)
LeFlore JL, et al. Alternative educational models for interdisciplinary RN, EMT 13 RCT Resuscitation CogI, FB, Indiv, ID:Hands on Sa, ST, SPc 14.5 4
student teams. Simul Healthc. 2009; 4:135–142. (BLS,ACLS,ATLS),Tea- RepP
m training
McDougall EM, et al. Preliminary study of virtual reality and model MS 20 RCT Min. invasive surg. CM:VR vs model Sa, ST, SPc 15.5 4
simulation for learning laparoscopic suturing skills. J Urol. 2009;
182:1018–1025.

(continued)
Instructional design features for simulation

e891
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Table A1. Continued.

e892
Participants Research comparisons Quality

Citation (sorted by year then author) Trainee N RCT Topic Features Theme Outcomes MERSQI NOS
D. A. Cook et al.

Moulton C-A, et al. Teaching communication skills using the MS, PG 30 RCT Open surgery/ CogI, FB ID:Feedback SPc 13.5 4
integrated procedural performance instrument (IPPI): a random- suturing,Communicati-
ized controlled trial. Am J Surg. 2009; 197:113–118. on skill
Muller MP, et al. Excellence in performance and stress reduction MD 29 RCT Resuscitation ID:Teach cognition SPc 14.5 4
during two different full scale simulator training courses: a pilot (BLS,ACLS,ATLS),Tea-
study. Resuscitation. 2009; 80:919–924. m training
Panait L, et al. The role of haptic feedback in laparoscopic simulation MS 10X Min. invasive surg. CogI, MLS SA:Tactile ST, SPc 11.5 3
training. J Surg Res. 2009; 156:312–316.
Rodgers DL, et al. The effect of high-fidelity simulation on educa- RN 34 Resuscitation CM:Manikin vs K, SPc 13.5 3
tional outcomes in an Advanced Cardiovascular Life Support (BLS,ACLS,ATLS) manikin
course. Simul Healthc. 2009; 4:200–206.
Rodriguez Garcia JI, et al. Does the incorporation of a virtual PG 17 RCT Min. invasive surg. CogI, FB, Time MA:Simulator, add SPc 10.5 2
simulator improve abilities in endoscopic surgery acquired with
an inanimate simulator? Cir Esp. 2009; 86:167–70.
Rosenthal ME, et al. Pretraining on Southwestern stations decreases MS 20 Min. invasive surg. Time MA:Simulator, add ST, SPc 11.5 1
training time and cost for proficiency-based fundamentals of
laparoscopic surgery training. J Am Coll Surg. 2009; 209:626–
631.
Sullivan-Mann J, et al. The effects of simulation on nursing students’ RN 53 RCT Critical thinking ClinV ID:Repetition K 14.5 4
critical thinking scores: a quantitative study. Newborn and Infant
Nursing Reviews. 2009; 9:111–116.
Szafranski C, et al. Distractions and surgical proficiency: an PG 14 Min. invasive surg. Indiv ID:Stress ST, SPc 12.5 2
educational perspective. Am J Surg. 2009; 198:804–810.
Ti LK, et al. Experiential learning improves the learning and retention MS 210 RCT Intubation CogI, FB, Indiv In:Instructor SPc, SPd 12.5 6
of endotracheal intubation. Med Educ. 2009; 43:654–660. intensity
Walsh CM, et al. Concurrent versus terminal feedback: it may be MS 30 RCT Endoscopy ID:Feedback ST, SPc 13.5 4
better to wait. Acad Med. 2009; 84 (10 Suppl):S54–S57. (GI,Urology,Bronch.)
Welke TM, et al. Personalized oral debriefing versus standardized PG 30 RCT Resuscitation CogI, FB, Indiv ID:Feedback SPc 14.5 5
multimedia instruction after patient crisis simulation. Anesth (BLS,ACLS,ATLS),Tea-
Analg. 2009; 109:183–189. m training
Yasukawa Y. The effectiveness of cavity preparation training using a D 39 RCT Dentistry CogI, FB, Indiv, ID:Feedback ST, SPc 12.5 5
virtual reality simulation system with or without feedback. MLS
[Japanese]. Kokubyo Gakkai Zasshi. 2009; 76:73–80.
Zausig YA, et al. Inefficacy of simulator-based training on anaes- MD 42 RCT Anesthesia FB, Time ID:Teach cognition SPc 15.5 6
thesiologists’ non-technical skills. Acta Anaesthesiol Scand.
2009; 53:611–619.
Acton RD, et al. Synthesis versus imitation: evaluation of a medical MS 189 Open surgery/suturing CogI, MLS, Time ID:Sequence ST, SPc 12.5 1
student simulation curriculum via Objective Structured
Assessment of Technical Skill. J Surg Educ. 2010; 67:173–178.
Ahmad I, et al. Evaluation of Real-time Visio-haptic deformable V 20 Physical exam SA:Visual SPc 11.5 2
Bovine Rectal Palpation Simulator. Proceedings 2010
International Symposium on Information Technology – Visual
Informatics, ITSim’10 2010; 1:Art. No.: 5561364.
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Blum CA, et al. High-fidelity nursing simulation: impact on student RN 53 Communication CM:Manikin vs SPc 14.5 3
self-confidence and clinical competence. International Journal of skill,Critical model
Nursing Education Scholarship. 2010; 7:Article 18. thinking,Nursing health
assessment
Brydges R, et al. Comparing self-guided learning and educator- RN 30 RCT Venous access Indiv, Mast, Time ID:Sequence SPc 14.5 5
guided learning formats for simulation-based clinical training. J
Adv Nurs. 2010; 66:1832–1844.
Brydges R, et al. Coordinating progressive levels of simulation fidelity MS 30 RCT Venous access ClinV, Indiv ID:Sequence Sa, SPc 14.5 5
to maximize educational benefit. Acad Med. 2010; 85:806–812.
Buzink SN, et al. Do basic psychomotor skills transfer between MS 29X RCT Min. invasive ClinV CM:VR vs VR ST 11.5 3
different image-based procedures? World J Surg. 2010; 34:933– surg.,Endoscopy
940. (GI,Urology,Bronch.)
Cason CL, et al. Improving learning of airway management with RN 76 RCT Critical thinking MA:CAI, compare K 11.5 5
case-based computer microsimulations. Clinical Simulation in
Nursing. 2010; 6:e15–e23.
Chandra V, et al. A comparison of laparoscopic and robotic assisted PG 20 RCT Min. invasive MA:Robot assis- ST, SPc 13.5 4
suturing performance by experts and novices. Surgery. 2010; surg.,Robotic surg. tance, add
147:830–839.
Da Cruz JAS, et al. Does training laparoscopic skills in a virtual reality MS 10 RCT Min. invasive surg. MLS ID:Sequence ST, SPc 11.5 4
simulator improve surgical performance? J Endourol. 2010;
24:1845–1849.
Dantas AK, et al. Assessment of preclinical learning on oral surgery D 20 Dentistry CogI, FB, Indiv, MA:Discussion, SPc 10.5 2
using three instructional strategies. J Dent Educ. 2010; 74:1230– MLS compare
1236.
Davoudi M, et al. Comparative effectiveness of low- and high-fidelity MD 44X RCT Endoscopy CM:VR vs model Sa 9 3
bronchoscopy simulation for training in conventional transbron- (GI,Urology,Bronch.)
chial needle aspiration and user preferences. Respiration. 2010;
80:327–334.
DeMaria Jr S, et al. Adding emotional stressors to training in MS 25 RCT Resuscitation CogI, MLS ID:Stress K, SPc 13.5 4
simulated cardiopulmonary arrest enhances participant perfor- (BLS,ACLS,ATLS)
mance. Med Educ. 2010; 44:1006–1015.
Debes AJ, et al. A tale of two trainers: virtual reality versus a video MS 38X RCT Min. invasive surg. RangeD CM:VR vs model ST, SPc 11.5 5
trainer for acquisition of basic laparoscopic skills. Am J Surg.
2010; 199:840–845.
Donoghue AJ, et al. Perception of realism during mock resuscita- PG 51 RCT Resuscitation CM:Manikin vs Sa 11 3
tions by pediatric housestaff: the impact of simulated physical (BLS,ACLS,ATLS) manikin
features. Simul Healthc. 2010; 5:16–20.
Dunnican WJ, et al. Reverse alignment ‘‘Mirror Image’’ visualization MS, PG 21 RCT Min. invasive surg. ID:Clinical scenario SPc 12 4
as a laparoscopic training tool improves task performance.
Surgical Innovation. 2010; 17:108–113.
Gauger PG, et al. Laparoscopic simulation training with proficiency PG 14 RCT Min. invasive surg. Mast, MLS ID:Mastery SPc, BP 13 3
targets improves practice and performance of novice surgeons.
Am J Surg. 2010; 199:72–80.
Grant JS, et al. Using video-facilitated feedback to improve student RN 40 RCT Communication ID:Feedback SPc 14.5 5
performance following high-fidelity simulation. Clinical Simulation skill,Team training
in Nursing. 2010; 6:e177–e184.
Instructional design features for simulation

e893
(continued)
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Table A1. Continued.

e894
Participants Research comparisons Quality

Citation (sorted by year then author) Trainee N RCT Topic Features Theme Outcomes MERSQI NOS
D. A. Cook et al.

Hein C, et al. A training program for novice paramedics provides EMT 50 RCT Intubation ID:Repetition ST, SPc 13.5 4
initial laryngeal mask airway insertion skill and improves skill
retention at 6 months. Simul Healthc. 2010; 5:33–39.
Hunziker S, et al. Brief leadership instructions improve cardiopul- MS 63 RCT Resuscitation ID:Teach cognition ST, SPc 13.5 6
monary resuscitation in a high-fidelity simulation: a randomized (BLS,ACLS,ATLS)
controlled trial.[Erratum appears in Crit Care Med. 2010
Jun;38(6):1510]. Crit Care Med. 2010; 38:1086–1091.
Kardong-Edgren SE, et al. Comparison of two instructional modal- RN 604 Resuscitation GrpP, Indiv, MLS In:Self-instruction SPc 12.5 2
ities for nursing student CPR skill acquisition. Resuscitation. (BLS,ACLS,ATLS)
2010; 81:1019–1024.
Kent DJ. Effects of a just-in-time educational intervention placed on RN 139 RCT Dressing change CogI ID:Instructions Sa, SPd 14.5 4
wound dressing packages: a multicenter randomized controlled
trial. Journal of Wound, Ostomy, & Continence Nursing. 2010;
37:609–614.
Kromann CB, et al. The testing effect on skills learning might last 6 MS 89 RCT Resuscitation MLS ID:Testing effect SPc 13.5 6
months. Adv Health Sci Educ Theory Pract. 2010; 15:395–401. (BLS,ACLS,ATLS)
Kruglikova I, et al. The impact of constructive feedback on training in PG 21 RCT Endoscopy CogI, FB, Indiv, ID:Feedback ST, SPc 12.5 5
gastrointestinal endoscopy using high-fidelity virtual-reality sim- (GI,Urology,Bronch.) MLS
ulation: a randomised controlled trial. Gut. 2010; 59:181–185.
Lauscher JC, et al. A new surgical trainer (BOPT) improves skill PG 35 RCT Open surgery/suturing CM:Manikin vs ST, SPc 12.5 4
transfer for anastomotic techniques in gastrointestinal surgery model
into the operating room: a prospective randomized trial. World J
Surg. 2010; 34:2017–2025.
Leblanc F, et al. A comparison of human cadaver and augmented MD 31 Min. invasive surg. CM:VR vs cadaver Sa, SPc, SPd 12.5 1
reality simulator models for straight laparoscopic colorectal skills
acquisition training. J Am Coll Surg. 2010; 211:250–255.
Leblanc F, et al. Hand-assisted laparoscopic sigmoid colectomy MD 26 Min. invasive surg. Indiv CM:VR vs cadaver Sa, SPc, SPd 11.5 1
skills acquisition: augmented reality simulator versus human
cadaver training models. J Surg Educ. 2010; 67:200–204.
Lee CC, et al. Comparison of traditional advanced cardiac life EMT 30 RCT Resuscitation FB, Indiv, MLS ID:Blending ST, SPc 10.5 4
support (ACLS) course instruction vs. a scenario-based, perfor- (BLS,ACLS,ATLS)
mance oriented team instruction (SPOTI) method for Korean
paramedic students. J Emerg Med. 2010; 38:89–92.
Lerner MA, et al. Does training on a virtual reality robotic simulator MS, PG 22 Robotic surg. Curr, FB CM:VR vs model ST 11.5 2
improve performance on the da VinciÕ surgical system? J
Endourol. 2010; 24:467–472.
Martı́nez AM, et al. Adaptation to a dynamic visual perspective in PG 26X Min. invasive surg. SA:Visual ST, SPc 10.5 2
laparoscopy through training in the cutting task. Surg Endosc.
2010; 24:1341–1346.
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

McCormick MJ, et al. Case scenarios and simulations as techniques RN, O 34 Physiology:Asthma GC:Interdiscplinary Sa 9 1
to facilitate asthma education and team building among health vs single disc.
care students. Journal of Asthma and Allergy Educators. 2010;
1:18–22.
Mishra S, et al. Percutaneous renal access training: Content MD 24X Endovascular proc. CM:VR vs animal Sa 7 2
validation comparison between a live porcine and a virtual reality
(VR) simulation model. BJU International. 2010; 106:1753–1756.
Mohammadi Y, et al. Comparison of laparoscopy training using the MS 43 Min. invasive surg. CM:VR vs model ST, SPc 11.5 3
box trainer versus the virtual trainer. J Soc Laparoendosc Surg.
2010; 14:205–212.
Molinas CR, et al. Defining a structured training program for PG, MD 40 RCT Min. invasive surg. RangeD ID:Sequence SPc 11.5 3
acquiring basic and advanced laparoscopic psychomotor skills in
a simulator. Gynecological Surgery. 2010; 7:427–435.
Morandeira Rivas A, et al. Low cost simulator for acquiring basic MS 16 RCT Min. invasive surg. CM:Model vs model ST 13.5 4
laparoscopic skills. Cir Esp. 2010; 87:26–32.
Muresan C, III, et al. Transfer of training in the development of MS 20 RCT Min. invasive surg. ID:Additional ST, SPc 12.5 3
intracorporeal suturing skill in medical student novices: a practice
prospective randomized trial. Am J Surg. 2010; 200:537–541.
Oermann MH, et al. HeartCode BLS with voice assisted manikin for RN 603 RCT Resuscitation GrpP, Indiv, MLS ID:Feedback SPc 12.5 4
teaching nursing students: preliminary results. Nurs Educ (BLS,ACLS,ATLS)
Perspect. 2010; 31:303–308.
Okrainec A, et al. Telesimulation: an effective method for teaching PG, MD 16 Min. invasive surg. CogI, FB, Indiv, In:Distance SPc 12 1
the fundamentals of laparoscopic surgery in resource-restricted MLS supervision
countries. Surg Endosc. 2010; 24:417–422.
Orde S, et al. A randomised trial comparing a 4-stage to 2-stage MS, RN, RN 120 RCT Intubation MLS ID:Sequence ST, SPc, SPd 13.5 5
teaching technique for laryngeal mask insertion. Resuscitation.
2010; 81:1687–1691.
Perkins GD, et al. The effect of pre-course e-learning prior to MD, RN, O 551 RCT Resuscitation FB, MLS MA:CAI, add K, SPc 15.5 5
advanced life support training: A randomised controlled trial. (BLS,ACLS,ATLS)
Resuscitation. 2010; 81:877–881.
Persoon MC, et al. The effect of a low-fidelity model on cystoscopic MS 32 RCT Endoscopy MA:Simulator, add Sa, ST, SPc, SPd 13.5 4
skill training: a single-blinded randomized controlled trial. Simul (GI,Urology,Bronch.)
Healthc. 2010; 5:213–218.
Salkini MW, et al. The role of haptic feedback in laparoscopic training MS 20 Min. invasive surg. FB SA:Tactile SPc 10.5 2
using the LapMentor II. J Endourol. 2010; 24:99–102.
Scavone BM, et al. A randomized controlled trial of the impact of PG 32 RCT Anesthesia ID:Clinical scenario Sa, ST, SPc 14.5 4
simulation-based training on resident performance during a
simulated obstetric anesthesia emergency. Simul Healthc. 2010;
5:320–324.
Stefanidis D, et al. Initial laparoscopic basic skills training shortens MS 18 RCT Min. invasive surg. Time MA:Simulator, add ST, SPc 14.5 5
the learning curve of laparoscopic suturing and is cost-effective.
J Am Coll Surg. 2010; 210:436–440.
Stefanidis D, et al. Robotic assistance improves intracorporeal MS 24X Min. invasive MA:Robot assis- ST, SPc 11.5 1
suturing performance and safety in the operating room while surg.,Robotic surg. tance, add
decreasing operator workload. Surg Endosc. 2010; 24:377–382.
Suebnukarn S, et al. Augmented kinematic feedback from haptic D 16 RCT Dentistry CogI, FB, MLS ID:Feedback ST, SPc 12.5 4
virtual reality for dental skill acquisition. J Dent Educ. 2010;
74:1357–1366.

(continued)
Instructional design features for simulation

e895
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Table A1. Continued.

e896
Participants Research comparisons Quality

Citation (sorted by year then author) Trainee N RCT Topic Features Theme Outcomes MERSQI NOS
D. A. Cook et al.

Thomas EJ, et al. Team training in the neonatal resuscitation PG 67 RCT Resuscitation MA:Team training, ST, SPc 14.5 6
program for interns: teamwork and quality of resuscitations. (BLS,ACLS,ATLS) add
Pediatrics. 2010; 125:539–546.
Van Heukelom JN, et al. Comparison of postsimulation debriefing MS 161 RCT Resuscitation ID:Feedback Sa 11 5
versus in-simulation debriefing in medical simulation. Simul (BLS,ACLS,ATLS)
Healthc. 2010; 5:91–97.
Wandell HF. Using a virtual reality simulator in phlebotomy training. O 25 Venous access ClinV, FB CM:VR vs model SPc 11.5 2
Laboratory Medicine. 2010; 41:463–466.
Wong W, et al. The effect of cross-training with adjustable airway MS, EMT 47 Intubation CogI, MLS, RangeD ID:Task variability SPd 10.5 2
model anatomies on laryngoscopy skill transfer. Anesth Analg.
2010; Available online October 21, 2010; in press.
Yang JH, et al. Comparison of four manikins and fresh frozen MD, RN 56X RCT Intubation CM:Manikin vs Sa 9 3
cadaver models for direct laryngoscopic orotracheal intubation cadaver
training. Emerg Med J. 2010; 27:13–16.
Zendejas B, et al. Teaching first or teaching last: Does the timing MS, PG 49X Various surgical topics ID:Sequence K 12.5 5
matter in simulation-based surgical scenarios? J Surg Educ.
2010; 67:432–438.
Alfes CM. Evaluating the use of simulation with beginning nursing RN 63 Pain management CogI ID:Hands on Sa, SPc 9.5 2
students. J Nurs Educ. 2011; 50:89–93.
Arora S, et al. Mental practice enhances surgical technical skills: A PG 18 RCT Min. invasive surg. CogI, Indiv, MLS ID:Teach cognition SPc 14.5 5
randomized controlled study. Ann Surg. 2011; 253:265–270.
Auerbach M, et al. Repetitive pediatric simulation resuscitation PG 151 Resuscitation CogI, MLS, RepP ID:Repetition Sa 9 1
training. Pediatr Emerg Care. 2011; 27:29–31. (BLS,ACLS,ATLS)
Bath J, et al. Standardization is superior to traditional methods of PG 18 RCT Open surgery/suturing MLS ID:Sequence Sa, K, SPc 12.5 5
teaching open vascular simulation. J Vasc Surg. 2011; 53:229–
223.e2
Fraser K, et al. Simulation training improves diagnostic performance MS 57 RCT Physical exam ID:Clinical scenario SPd 11.5 4
on a real patient with similar clinical findings. Chest. 2011;
139:376–381.
Guhde J. Nursing students’ perceptions of the effect on critical RN 133X Nursing tasks FB, GrpP, MLS ID:Task variability Sa 8 3
thinking, assessment, and learner satisfaction in simple versus
complex high-fidelity simulation scenarios. J Nurs Educ. 2011;
50:73–78.
Maggio MP, et al. The effect of magnification loupes on the D 232 Dentistry MLS SA:Visual ST, SPc 11.5 3
performance of preclinical dental students. Quintessence Int.
2011; 42:45–55.
Naughton PA, et al. Skills training after night shift work enables PG 20 Endovascular proc. ID:Timing ST, SPc 10.5 2
acquisition of endovascular technical skills on a virtual reality
simulator. J Vasc Surg. 2011; 53:858–866.
Oermann MH, et al. Effects of monthly practice on nursing students’ RN 495 RCT Resuscitation CogI, DistP, FB, ID:Repetition SPc 13.5 5
CPR psychomotor skill performance. Resuscitation. 2011; (BLS,ACLS,ATLS) Indiv, MLS, RepP,
82:447–453 Time
 Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

Snyder CW, et al. Effects of Virtual Reality Simulator Training Method MS 32 RCT Min. invasive FB, Indiv, MLS ID:Sequence ST 13.5 4
and Observational Learning on Surgical Performance. World J surg.,Endoscopy
Surg. 2011; 35:245–252. (GI,Urology,Bronch.)
Sutton RM, et al. ‘‘Booster’’ training: Evaluation of instructor-led PG, RN 46 RCT Resuscitation CogI, FB ID:Feedback SPc 12.5 4
bedside cardiopulmonary resuscitation skill training and auto- (BLS,ACLS,ATLS)
mated corrective feedback to improve cardiopulmonary resus-
citation... Pediatr Crit Care Med. 2011; Online 2010 Jul 9; in
press.
Swanson EA, et al. Comparison of selected teaching strategies RN 96 RCT Critical thinking FB In:Instructor SPc 14.5 4
incorporating simulation and student outcomes. Clinical intensity
Simulation in Nursing. 2011; 7(3):e81–e90.
Thompson JR, et al. Limited value of haptics in virtual reality MS, O 8 RCT Min. invasive surg. SA:Tactile ST, SPc 12 3
laparoscopic cholecystectomy training. Surg Endosc. 2011;
25:1107–1114.
Uccelli J, et al. The validity of take-home surgical simulators to PG 14 RCT Min. invasive surg. CogI, FB, Indiv, ID:Sequence ST, SPc 11.5 3
enhance resident technical skill proficiency. Am J Surg. 2011; MLS, Time
201:315–319.
Zhao YC, et al. Can virtual reality simulator be used as a training aid PG 20 RCT Endoscopy FB, GrpP, Indiv, CM:VR vs model SPc, SPd 13.5 4
to improve cadaver temporal bone dissection? Results of a (GI,Urology,Bronch.) MLS
randomized blinded control trial. Laryngoscope. 2011; 121:831–
837.

Trainees: MS ¼ medical student, PG ¼ postgraduate physician trainee, MD ¼ practicing physician, RN ¼ nurse or nursing student, EMT ¼ emergency medical technician/paramedic/first responder or EMT student, D ¼ dentist or dental
student, V ¼ veterinarian or veterinary student, C ¼ chiropractor or student, O ¼ other/mixed.
N: Number of outcome observations; this is usually the number of trainees, but in some cases reflects the number of teams observed or the number of patient observations. X ¼ Crossover.
RCT ¼ randomized controlled trial.
Feature: Key features that varied between interventions (i.e. included in meta-analyses), see main text for definitions. Features that were coded the same for both groups (eg, both present or both absent) are not listed. ClinV ¼ clinical
variation; CogI ¼ cognitive interactivity; Curr ¼ curriculum integration; DistP ¼ distributed practice; FB ¼ feedback; GrpP ¼ group practice; Indiv ¼ individualization; Mast ¼ mastery learning; MLS ¼ multiple learning strategies; RangeD ¼ range
of difficulty; RepP ¼ repetitive practice; Time ¼ time learning.
Theme: The overall research theme (research question). Main themes are indicated first, with sub-themes following the colon. MA ¼ modality added; GC ¼ group composition; ID ¼ instructional design; In ¼ Instructor; CM ¼ compare sim
modality; SA ¼ sensory augmentation.
Outcomes: Sa ¼ Satisfaction, K ¼ knowledge, ST ¼ skill-time, SPc ¼ skill-process, SPd ¼ skill-product, BT ¼ behavior-time, BP ¼ behavior-process, P ¼ patient effects.
Quality: MERSQI ¼ Medical Education Research Study Quality Instrument (maximum score 18); NOS ¼ modified Newcastle-Ottawa scale (maximum score 6).
Instructional design features for simulation

e897
 D. A. Cook et al.

Table A2. Results of sensitivity analyses.

Feature No. studies Pooled ES (95% CI) p I2 Main analysis pooled ES

Non-time Skills
Clinical variation 16 0.18 (0.14, 0.51) 0.27 83 0.20
Cog. interactivity 89 0.59 (0.44, 0.75) 0.00 88 0.65
Feedback 80 0.36 (0.21, 0.50) 0.00 87 0.44
Group practice 8 0.30 (0.56, 0.05) 0.02 75 0.22
Individualization 59 0.46 (0.28, 0.64) 0.00 88 0.52
Mult. learning strat. 70 0.54 (0.40, 0.68) 0.00 85 0.62
Range of difficulty 20 0.64 (0.26, 1.03) 0.00 87 0.68
Repetitive practice 7 0.68 (0.03, 1.38) 0.06 89 0.68

Behaviors and Patient Effects

Feedback 2 0.18 (0.23, 0.58) 0.39 0 0.32

Notes: Several studies reporting non-time skills had 3 simulation arms. Since we could only compare 2 groups at once, we first included the interventions with the
greatest between-group difference and then performed sensitivity analyses substituting the third intervention. The table above summarizes all results influenced by the
alternate analyses. The main analysis pooled effect size (ES) is provided to aid in comparison.
Med Teach Downloaded from informahealthcare.com by University of Missouri Columbia on 09/21/13
For personal use only.

e898