Medical Teacher, Vol. 25, No. 2, 2003, pp.

171–176

The CIP (comprehensive integrative puzzle)

assessment method

ROSALIE BER
Bruce Rappaport Faculty of Medicine, Technion—Israel Institute of Technology, Haifa, Israel

SUMMARY This paper describes a novel tool for assessment in simulated patients have flourished. Yet, although appropriate
medical education, the comprehensive integrative puzzle (CIP). for testing clinical skills, the OSCE has limited validity in the
The dual scoring system of the puzzle stresses the integrative assessment of diagnostic reasoning.
elements of diagnostic thinking and clinical reasoning, while The objective of this paper is to describe the Comprehen-
Med Teach Downloaded from informahealthcare.com by Duke-Nus Graduate Med School Sgp on 11/03/14

preserving the ability to discern proficiency in various disciplinary sive Integral Puzzle (CIP) assessment method (Ber, 1995,
elements. The CIP has the format of an ‘extended matching’ 1997). This paper-and-pencil test combines an assessment of
crossword puzzle. Its answer sheet is a grid comprising rows and knowledge in various biomedical disciplines (e.g. biochem-
columns. The left-hand column contains diagnoses or brief clinical istry, microbiology, immunology, pharmacology and pathol-
vignettes. To complete the cells of the grid the student is required to ogy) with an assessment of diagnostic reasoning and clinical
match, stepwise, the various ‘disciplinary investigations’ to the problem solving. As such, the CIP may promote integration
diagnoses or clinical vignettes. When the puzzle is completed each between preclinical courses and clinical training, and help
horizontal row reflects a coherent medical case. The completed students adapt to the transition to the clinical clerkship
horizontal rows reflect integrative ability (diagnostic thinking and rotations.
clinical reasoning) and the vertical columns measure the student’s The Bruce Rappaport Faculty of Medicine of the
proficiency in interpreting medical history data, physical examina- Technion utilizes a six-year Baccalaureate MD program,
tion findings, laboratory test results, ECG, imaging, special tests, which consists of three years of preclinical studies (equivalent
For personal use only.

pathology and pharmacology. The CIP has been well accepted by to pre-med and years one and two in most North American
teachers and students during the last seven years at the Bruce medical schools). The first three years consist of discipline-
Rappaport Faculty of Medicine in Haifa, and it has favorably oriented courses. During the first 27 weeks of the fourth year
affected both student assessment and teaching. The reliability of the the students take a preclinical–clinical integrated system-
test and its validity will be reported separately. oriented course (system-integrated course) consisting of 13
modules, and the last two and a half years consist of clerkship
rotations in various clinical departments (equivalent to years
Introduction
three and four in North American medical schools). The CIP
One of the difficulties that students encounter during their was originally designed for the summative assessment of
undergraduate training is the need to restructure their students’ learning in the system-integrated course; however,
knowledge base when they move from a preclinical discipline it may be implemented at every stage in the training of the
or organ-system orientation, to a disease orientation during student physician, resident or as an assessment of continuous
clerkship rotations and finally, to the patient orientation of medical education programs.
clinical practice. To redress this difficulty, medical educators
have attempted to replace traditional curricula by integrative
programs, such as the various problem-based-learning
formats (Moore, 1991; Schmidt et al., 1996; Maudsley, Method
1999; Colliver, 2000; Norman & Schmidt, 2000), and to The CIP is similar to the extended matching assessment
introduce medical students into the precepts of the hypoth- described by Case & Swanson (1993). However, it seems to
esis-generation–verification strategy of diagnostic reasoning appeal more to students and teachers because of the fun in
(Elstein et al., 1978; Ber & Alroy, 1981; Schwartz et al., 1993). building and solving a ‘matching puzzle’ (like a crossword
Briefly, this strategy consists of generation of diagnostic puzzle). The horizontal matching requires an ability to
hypotheses early in the clinical encounter, and a search for integrate among disciplines and the vertical columns depict
additional clues from the history, physical examination, knowledge in the various disciplines. Although it can be used
laboratory and special tests to either confirm or refute each at all stages of medical training, the description that follows
one of them. pertains to the assessment of students in the systems-
The assessment of such integrative programs and of integrated course. The matrix and detailed design of such a
diagnostic reasoning is a challenge to medical educators. CIP at the termination of a systems-block is presented below,
Multiple-choice questions (MCQ), true/false questions, and the detailed instructions to teachers on how to prepare a
patient-management problems, and long and short essay CIP and to students on how to solve a CIP can be obtained
formats are either restricted to tested areas, or cumbersome
and difficult to assess, while oral examinations have poor
Correspondence: Rosalie Ber MD DSc, Medical Education Department, B.
reliability. During the last two decades, objective structured Rappaport Faculty of Medicine, PO Box 9649, Haifa 31096, Israel. Email:
clinical examinations (OSCE) using real, standardized or rosalieb@techunix.technion.ac.il

ISSN 0142–159X print/ISSN 1466–187X online/03/020171-06 ß 2003 Taylor & Francis Ltd 171
DOI: 10.1080/0142159031000092571
 Rosalie Ber

from the corresponding author upon request and/or found at diagnostic entities could be:
the Medical Teacher website [1].
(1) unstable angina;
(2) myocardial infarction;
Test preparation (3) rheumatic mitral stenosis;
(4) acute pericarditis;
The preparation of a CIP is teamwork by the participants in
(5) infective endocarditis;
the teaching of a systems-block: clinician, pathologist, micro-
(6) hypertrophic cardiomyopathy.
biologist, pharmacologist, biochemist, radiologist, etc. Each
team determines the clinical scenarios and prepares a pool of The student is expected to insert into the remaining cells
relevant material, such as ECG and EEG strips or interpreta- matching data, which he/she selects from a multiple-choice
tions, X-ray, CT, MRI and US images or interpretations pool of options indicated by letters (a–f for the most simple
thereof, pathology pictures and/or slides or their descriptions CIP, and a–z for more complicated CIPs).
and endoscopy photographs or descriptions. The team To complete any cell in columns I–VI the student must
members review the assembled CIP, determine the scoring match the (a–f ) options from a pool of distracters in the
criteria (see suggested CIP scoring below) and compose the sections (I–VI) to each one of the diagnoses respectively. The
‘Instructions for students’. The person who coordinates these student is advised to begin with the medical history vignettes,
Med Teach Downloaded from informahealthcare.com by Duke-Nus Graduate Med School Sgp on 11/03/14

team activities is also responsible for pilot testing and timing matching the most suitable letters (a–f ) in section I to the six
of the CIP on colleagues or on senior students. diagnoses and marking them in column I in the puzzle grid.
For example for the above CIP in the cardiovascular system:
Test format Section I: Patient’s presenting stories

A typical CIP format consists of a puzzle grid (Figure 1). (a) A 45-year-old man arrived at the emergency room
The first column of the grid lists the diagnostic entities, for because of chest pressure, which began three hours
example for a CIP in the cardiovascular system the six beforehand while resting. There is no history of former
For personal use only.

Figure 1. CIP answering sheet (grid) with correct answers.

172
 The CIP assessment method

illnesses. He has smoked 20 cigarettes a day for the last And so on, section, by section, matching the letters (a–f ) in
28 years. each section to the six diagnoses/vignettes and marking them
(b) A 28-year-old woman, in her third month of pregnancy, in their respective columns.
arrived at the emergency room because of severe short-
ness of breath (dyspnoea). She complains of exertional Section III: Chest X-ray and ECG
fatigue from the beginning of her pregnancy, and (Here the students receive ECG strips and X-ray films, or the
increasing shortness of breath during the last week. interpretation thereof )
(c) A 25-year-old man complains of shortness of breath and
dizziness on exertion. Both his grandfather and elder (a) An ECG which shows elevation of ST segment in leads
brother died suddenly at the age of 32 years. II, III, AVF and V3R.
(d) A 75-year-old man arrived at the emergency room (b) An ECG which shows atrial fibrillation and enlargement
because of 15 minutes of chest pain and sweating, of the right ventricle, and a chest X-ray which shows an
which began without any prior exertion. The patient enlarged left atrium with elevation of the left bronchus,
had had a coronary by-pass operation nine years before- pulmonary edema.
hand and was asymptomatic for nine years. (c) An ECG which shows sinus rhythm, and a 2 mm
(e) A 32-year-old woman, with known congenital heart depression of the ST segment of the frontal leads, and
Med Teach Downloaded from informahealthcare.com by Duke-Nus Graduate Med School Sgp on 11/03/14

defect, was hospitalized with a three-week history of a chest X-ray which shows metal stitches in sternum and
fever, malaise, night sweats and increasing shortness of venous congestion of lungs.
breath. (d) An ECG which shows sinus rhythm with marked left
(f ) A 40-year-old woman, suffering for the last three weeks ventricular strain and septal Q waves, and a chest X-ray
from flu-like symptoms, complains of continuous ante- which shows a normal sized heart.
rior chest pain during the last week. The pain is aggra- (e) An echocardiogram which shows vegetations on the
vated by inspiration, change of posture or swallowing. aortic leaflets.
(f ) An ECG which shows marked ST segment elevation in
The student proceeds to the physical examination findings, leads II, III, AVF and V3–V6 and a chest X-ray which
matching the most suitable letters (a–f ) in section II to the six shows marked pericardial effusion.
diagnoses/vignettes and marking them in column II in the
puzzle grid. Section IV: Laboratory and other tests
For personal use only.

Section II: Physical examination (a) On echocardiography—reduction in the contractility of

(a) Pale, sweating, apprehensive. BP 80/60 mmHg, pulse the right ventricle and the lower part of the left ventricle.
95/minute, with slight irregularity—premature beats? Cholesterol 5.689 mmol/l (220 mg%), LDL 4.137
Lungs—alveolar breathing, no additional sounds. On mmol/l (160 mg%), HDL 0.905 mmol/l (35 mg%).
palpation—hyperdynamic cardiac point of maximum (b) On echocardiography—marked thickening of the inter-
impact (PMI), and on auscultation—accentuated first ventricular septum, 30 mmHg pulse gradient at the
heart sound. outlet of the left ventricle.
(b) Excellent general condition. BP 130/80 mmHg, pulse (c) On catheterization—complete obstruction of the left
78/minute, regular, abrupt pulse wave. Jugular venous main coronary artery and 95% narrowing of the venous
pulse shows marked ‘a’ wave. Lungs—alveolar breathing, bypass to the anterior descending coronary artery. LDL
no additional sounds. On palpation—presystolic lift of 4.784 mmol/l (185 mg%).
the PMI, on auscultation—heart sounds normal, harsh (d) On echocardiography—enlargement of the left atrium.
systolic murmur increases on sitting and decreases upon On catheterization—high pressure (50/25 mmHg) in the
squatting. pulmonary arteries, and a diastolic pulse gradient of 16
(c) Good general condition. BP 170/100 mmHg. No jugular mmHg between the pulmonary wedge pressure and the
vein congestion. On auscultation of chest—alveolar left ventricle.
breathing with fine crepitations at bases of both lungs, (e) Streptococcus viridans was cultured from two blood
faint heart sounds. samples.
(d) Severe dyspnoea, frothing at mouth. BP 130/80 mmHg, (f ) Here a graph of arterial pressure is demonstrated, showing
completely irregular pulse, roughly 120/minute. Severe how the arterial pressure wave undulates markedly
jugular venous congestion. Pounding PMI. On ausculta- between inspiration and expiration.
tion of chest—alveolar breathing with fine crepitations
Section V: Treatment and follow-up
over both lungs, an additional heart sound follows the
second heart sound and this is followed by a harsh (a) Treatment with beta-blockers improved the dyspnoea
diastolic murmur. complaint. (Here is a possibility for the teachers to frame
(e) Pale sweating, apprehensive. Shallow breathing, BP 100/ questions regarding the mechanism of action of beta-blockers)
70 mmHg, weak pulse 120/minute. Distended jugular (b) Immediate catheterization and ballooning of the valve
veins. resulted in rapid clinical improvement.
(f ) Severe dyspnoea, BP 110/60, mmHg, regular pulse of (c) Infusion of 2000 ml, including infusion of streptokinase,
110/minute. On auscultation of chest—alveolar breathing resulted in a rise in blood pressure. The pain disappeared
with bibasilar rales, an additional ‘gallop like’ sound after the streptokinase treatment. (Here is a possibility for
follows the 2nd heart sound, a short systolic and diastolic the teachers to ask questions regarding streptokinase and TPA
murmur at the upper left sternal border. or alternative treatments)

173
 Rosalie Ber

(d) Treatment with aspirin, heparin and ACE inhibitors abdominal pain, etc. The six (or more) presenting vignettes
resulted in gradual improvement in the patient’s condi- need to be matched to the sets of physical examination
tion. (Here is a possibility for the teachers to ask questions findings, tests, imaging, treatment/pharmacology and pathol-
regarding mechanism of action of ACE inhibitors) ogy. (In the case of a common symptom or sign, this needs to
(e) Pericardial tap yielded 750 ml of clear yellow fluid and be matched in addition to clinical vignettes or to requesting
the patient’s general feeling improved. (Here is a possibility specific anamnestic information). For the more advanced
for the teachers to ask questions about the pathophysiology of CIP, determining the most likely diagnoses comprises the last
pericardial effusion, or the difference between transudate and step. These need to be inserted in the last column, either
exudate) manually, or by letter selected from an extensive list
(f ) Crystalline penicillin G 12–18
106 units injected every (numbered a–z) provided in the last section.
12 hours for four weeks. (Here is a possibility for the teachers Furthermore, the number of options for each section
to frame questions regarding mechanism of action of different may be increased. The instruction that ‘each option may be
types of antibiotics) used once, more than once or not at all’ brings the
CIP all the closer to clinical reality. Yet more complicated
Section VI: Pathology CIP assessments may include more than six vignettes (rows),
more ‘disciplinary’ sections (columns), as well as subdivision
Med Teach Downloaded from informahealthcare.com by Duke-Nus Graduate Med School Sgp on 11/03/14

(a) One or two histopathology slides—one demonstrating of columns (e.g. macro- and micro-pathology, or laboratory
necrosis of myocardium and another demonstrating an tests subdivided into: biochemical, hematological, serological
obstructed coronary artery. etc.). Depending on the skill to be assessed, the CIP may
(b) A histopathology slide demonstrating a damaged valve include ECG and EEG tracing strips, X-ray, CT, MRI and
leaflet, with marked infiltration of polymorphonuclear US images, and cardiac and respiratory tracings, which need
cells and clusters of hematoxylin-stained bacteria. to be interpreted by the student, prior to matching in the
(c) A histopathology slide demonstrating scarred tissue sur- puzzle.
rounded by muscle cells.
(d) Either a picture of or a description of hypertrophic cardiac
muscle and asymmetric hypertrophy of the septum. Test scoring (CIP scoring)
(e) Either a picture of or a description of a thickened mitral The CIP horizontal scoring reflects the student’s integrative
valve, and a histopathology slide demonstrating fibrosis ability (diagnostic thinking, clinical reasoning) and the
For personal use only.

of the chorda tendinae and the subvalvular myocardium. vertical scoring assesses the student’s mastery of the various
(f ) A picture of ‘bread and butter’ pericardium. disciplines. The scoring system can be chosen by each
When the puzzle is completed the horizontal rows should teaching team (or course director) and adapted to the stage of
reflect six coherent medical cases. study of the students and level of difficulty of the test. To pass
The students are instructed to transfer their answers (a) to the test, a student needs to match correctly horizontally and
( f ) to an optical reading sheet, according to the numbers in vertically a predetermined number of cells. The team or
the upper left-hand corner of each square in the puzzle grid. course director decides whether none, all or which of the
They hand in both the optical reading sheet and the puzzle columns are ‘criterion’ columns to obtain a pass or fail grade.
grid. Scoring is through an optical-reader answer sheet, Our proposed scoring system (Ber, 1998) for the above-
providing immediate feedback to instructors and students. described CIP for beginning students in the system-
In this easiest form of CIP, each option (a–f ) may be used integrated course is the following:
once only. Thus, if the student knows the correct matching  6 correct out of 6 ¼ 100% (full) marks;
for five of the six diagnoses, the sixth will be correct by  5 correct out of 6 ¼ 80% marks;
default, and the student will thus receive a 16% bonus to his/  4 correct out of 6 ¼ 60% marks;
her grade. In the case of a four-diagnoses CIP with four  3, 2, 1, or 0 correct out of 6 ¼ 0 marks.
options per section, the student’s bonus would be 25% of the
grade. Tests with higher degrees of difficulty may offer more The passing score in the system-integrated course is 60/100.
than six sections (columns), more than six options/distracters Therefore the student must achieve a minimum average of
per section, and may include instructions that ‘each option 60% points in the horizontal scores, as well as 60% in four
may be used once, more than once, or not at all’. predetermined ‘criterion’ vertical scores, namely history,
physical examination, pathology and pharmacology.
Achieving an average horizontal score as high as 80% may
still result in failure, if the student in the systems-integrated
Adaptation to the level of study
course fails to achieve the passing vertical score of 60% in any
The CIP design can be modified according to the level of the one of the history, physical examination, pathology or
students/physicians to be assessed. Less complicated CIPs pharmacology ‘criterion’ column(s).
present diagnoses in the first column, and require students to
match the additional data to the diagnoses, as described
above. More complicated CIPs can be used for students in
Discussion
clinical clerkships, for interns and residents, and for CME
assessment. Learning occurs best when students are encouraged to
In such cases, the left-hand column of the puzzle grid integrate new information with past knowledge, to relate
comprises six or more clinical vignettes, or a common among various disciplines and to apply theoretical concepts
symptom or sign, such as dyspnoea, edema, diarrhea, to clinical situations (Schmidt, 1983; Schwartz et al., 1993;

174
 The CIP assessment method

Albanese, 2000). Such integration between the biomedical Acknowledgements

disciplines and clinical reasoning requires not only integrative
Prof. G. Alroy was most helpful in the initial discussions
teaching programs but also integrated methods of assess-
on the concept of the CIP, Prof. R. Einat designed the first
ment. The CIP is based on the premise that assessment
pilot CIP in gastroenterology and liver diseases and Prof.
drives learning (‘assessment is the tail that wags the dog’);
G. Brook was the first ‘student’ to try it out. The author is
consequently, its main advantage is that it is more effective in
also indebted to the members of the preclinical–clinical
promoting retention of knowledge than the traditional
integrated system-oriented course (system-integrated course)
compartmentalization into disciplines, which are forgotten
think-tank, and to the coordinators of the systems in the
shortly after examinations.
preclinical–clinical integrative course, who designed the
In the early stages of designing the preclinical–clinical
first systems CIPs. Thanks are due also to Profs P.
integrated system-oriented course, the teachers of pathology
Schwartz and D. Loten for adopting the CIP as a teaching
and pharmacology were apprehensive about the integration
tool, at the University of Otago Medical School in Dunedin,
of their disciplines within the integrative course, and this
Otago, New Zealand, and to Prof. C. Hazlett at the Chinese
unease surfaced especially during the design of the CIP
University of Hong Kong, who helped with statistics and the
assessment method. Although the CIP scoring method was
final transition from a four-case to a six-case CIP. Moreover,
proposed at the initiation of the systems-integrated course, it
Med Teach Downloaded from informahealthcare.com by Duke-Nus Graduate Med School Sgp on 11/03/14

the author is grateful to the two successive deans of the Bruce

was decided to use both conventional scoring (i.e. the
Rappaport Faculty of Medicine in Haifa—Profs P. Lavie and
average of all correct individual matching) and the CIP
R. Beyar—for encouragement and support. Last, but not
scoring method in parallel but to grade the students
least, hearty thanks are due to Prof. J. Benbassat for critical
according to conventional scoring. When we analyzed the
reading of the manuscript.
different methods of scoring CIPs for the first four years of
running the course, we found that conventional scoring
allowed students to gain passing grades, even when they
consistently failed the Pathology disciplinary element of the Notes
course (Ber & Brik, 2000). Therefore, for the last two years [1] Appendix 1: Instructions to Teachers, and Appendix 2:
we have determined that the pathology and treatment/ Instructions to Students, are available either from the
pharmacology columns are ‘criterion’ columns, thus satisfy- author on request, or from the Medical Teacher website:
ing the demands of these two disciplines in the preclinical–
For personal use only.

www.medicalteacher.org
clinical integrated system-oriented course. In addition, we
have added the medical history and physical examination to
the ‘criterion’ columns. Each CIP system team may decide
which additional disciplinary column(s) is (are) ‘criterion’ References
column(s) for their system block, e.g. should the teachers of ALBANESE, M. (2000) Problem-based learning: why curricula are likely to
the cardiovascular system decide that ECG interpretation is a show little effect on knowledge and clinical skills, Medical Education, 34,
critical skill, this column would also become a ‘criterion’ pp. 729–738.
column. BER, R. (1995) Assessing an integrative course, paper presented at AMEE
The CIP has been in use at the B. Rappaport Faculty annual conference on Trends in Medical Education, Zaragoza, Spain,
September.
of Medicine for the last seven years. Although its prepara-
BER, R. (1997) Design of an integrative course and assessment method:
tion is time-consuming and laborious, it has evolved into a the CIP (Comprehensive Integrative Puzzle), in: A.J.J.A. SCHERPBIER,
major component not only in assessment but also in the C.P.M. VAN DER VLEUTEN, J.J. RETHANS & A.F.W. VAN DER STEEG
planning of teaching. Students so far have responded to the (Eds) Advances in Medical Education, pp. 84–86 (Dordrecht, Kluwer
CIP with initial apprehension followed by enthusiasm, with Academic Publishers).
comments like: ‘‘this is the first exam that makes sense’’ and BER, R. (1998) The CIP (Comprehensive Integrative Puzzle) scoring
system, paper presented at the AMEE annual conference, Prague,
‘‘why can’t we have this kind of an exam more often?’’ and a
Czech Republic, September.
demand for formative ‘dummy’ CIPs before the summative
BER, R. & ALROY, G. (1981) The teaching of history-taking and
examination. diagnostic thinking: description of a method, Medical Education, 15,
Because of its versatility, flexibility and appeal to medical pp. 97–99.
students, the CIP, with or without additional MCQs, is today BER, R. &, BRIK, R. (2000) Four years’ experience with the CIP
the main summative assessment tool of the system-integrated assessment method: formative vs summative, paper presented at the
modules, and is also used for formative assessment in some of Ottawa in Africa Conference, Cape Town, South Africa, March.
CASE, S.M. & SWANSON, D.B. (1993) Extended-matching items: a
the elective clerkships (nephrology and cardiology). Some
practical alternative to free-response questions, Teaching and Learning in
faculty members have difficulty in accepting the notion that Medicine, 5, pp. 107–115.
the examination format affects student learning, and that COLLIVER, J.A. (2000) Effectiveness of problem-based learning curricula:
integration is not a mere juxtaposition in teaching or research and theory, Academic Medicine, 75, pp. 259–266.
assessment. The suggestion to system-block team coordina- ELSTEIN, A.S., SHILMAN, L.S. & SHULMAN, L.S. (1978) Medical Problem
tors that the CIP be designed prior to teaching has not always Solving: An Analysis of Clinical Reasoning (Cambridge, MA, Harvard
been accepted. However, the need to prepare a CIP University Press).
MAUDSLEY, G. (1999) Do we all mean the same thing by ‘problem-based
promotes, in and of itself, cooperation, and frequently leads
learning’? A review of the concepts and a formulation of the ground
to improved integration of the course in the following year. rules, Academic Medicine, 74, pp. 178–185.
It remains to be determined whether the CIP currently MOORE, G.T. (1991) Initiating problem-based learning at Harvard
being developed as a web-based tool for formative assessment medical school, in: D. BOUD & G. FELETTI (Eds) The Challenge of
(the webCIPß) will also lead to improvement of learning. Problem Based Learning, pp. 80–87 (New York, St Martin’s Press).

175
 Rosalie Ber

NORMAN, G.R. & SCHMIDT, I.I.G. (2000) Effectiveness of problem-based of diagnostic competence: comparison of a problem-based, an inte-
learning curricula: theory, practice and paper darts, Medical Education, grated and a conventional curriculum, Academic Medicine, 71,
34, pp. 721–728. pp. 654–658.
SCHMIDT, H.G. (1983) Problem-based learning: rationale and descrip- SCHWARTZ, R.W., BURGETT, J.E. & DONNELY, M.B. (1993) An overview
tion, Journal of Medical Education, 17, pp. 11–16. of the clinical reasoning process and its relationship to problem-based
SCHMIDT,H.G.,MACHIELS-BONGAERTS,M.,HERMANS,H., TEN CATE,T.J., learning, Current Surgery, January, pp. 66–69.
VENECAMP, R. & BOSHUIZEN, H.P.A. (1996) The development
Med Teach Downloaded from informahealthcare.com by Duke-Nus Graduate Med School Sgp on 11/03/14
For personal use only.

176