Review

Improving the quality of reports of meta-analyses of randomised

controlled trials: the QUOROM statement

David Moher, Deborah J Cook, Susan Eastwood, Ingram Olkin, Drummond Rennie, Donna F Stroup, for the QUOROM Group*

Summary Introduction
Health-care providers and other decision-makers now
Background The Quality of Reporting of Meta-analyses
have, among their information resources, a form of clinical
(QUOROM) conference was convened to address standards for
report called the meta-analysis,1-4 a review in which bias has
improving the quality of reporting of meta-analyses of clinical been reduced by the systematic identification, appraisal,
randomised controlled trials (RCTs). synthesis, and, if relevant, statistical aggregation of all
Methods The QUOROM group consisted of 30 clinical relevant studies on a specific topic according to a
epidemiologists, clinicians, statisticians, editors, and predetermined and explicit method.3 The number of
researchers. In conference, the group was asked to identify published meta-analyses has increased substantially in the
items they thought should be included in a checklist of past decade.5 These integrative articles can be helpful for
standards. Whenever possible, checklist items were guided by clinical decisions, and they may also serve as the policy
research evidence suggesting that failure to adhere to the foundation for evidence-based practice guidelines,
item proposed could lead to biased results. A modified Delphi economic evaluations, and future research agendas. The
value of meta-analysis is evident in the work of the
technique was used in assessing candidate items.
international Cochrane Collaboration,6,7 the primary
Findings The conference resulted in the QUOROM statement, purpose of which is to generate and disseminate high-
a checklist, and a flow diagram. The checklist describes our quality systematic reviews of health-care interventions.
preferred way to present the abstract, introduction, methods, Like any research enterprise, particularly one that is
results, and discussion sections of a report of a meta- observational, the meta-analysis of evidence can be flawed.
analysis. It is organised into 21 headings and subheadings Accordingly, the process by which meta-analyses are
regarding searches, selection, validity assessment, data carried out has undergone scrutiny. A 1987 survey of 86
abstraction, study characteristics, and quantitative data English-language meta-analyses8 assessed each publication
synthesis, and in the results with trial flow, study on 23 items from six content areas judged important in the
characteristics, and quantitative data synthesis; research conduct and reporting of a meta-analysis of randomised
documentation was identified for eight of the 18 items. The trials: study design, combinability, control of bias,
flow diagram provides information about both the numbers of statistical analysis, sensitivity analysis, and problems of
RCTs identified, included, and excluded and the reasons for applicability. The survey results showed that only 24 (28%)
exclusion of trials. of the 86 meta-analyses reported that all six content areas
had been addressed. The updated survey, which included
Interpretation We hope this report will generate further more recently published meta-analyses, showed little
thought about ways to improve the quality of reports of meta- improvement in the rigour with which they were reported.9
analyses of RCTs and that interested readers, reviewers, Several publications have described the science of
researchers, and editors will use the QUOROM statement and reviewing research,1 differences among narrative reviews,
generate ideas for its improvement. systematic reviews, and meta-analyses,2 and how to carry
out,3,4,10 critically appraise,1115 and apply16 meta-analyses in
Lancet 1999; 354: 1896900 practice. The increase in the number of meta-analyses
See Commentary page ???????? published has highlighted such issues as discordant meta-
analyses on the same topic17 and discordant meta-analyses
and randomised-trial results on the same question.18
An important consideration in interpretation and use of
meta-analyses is to ascertain that the investigators who did
the meta-analysis not only report explicitly the methods
*Other members listed at end of paper
they used to analyse the articles they reviewed, but also
University of Ottawa, Thomas C Chalmers Centre for Systematic report the methods used in the research articles they
Reviews, Ottawa (D Moher MSc); McMaster University, Hamilton analysed. The meta-analytical review methods used may
(D J Cook MD), Ontario, Canada; University of California,
not be provided when a paper is initially submitted: even
San Francisco (S Eastwood ELS(D)); Stanford University, Stanford, CA
(I Olkin PhD); JAMA, Chicago, IL (D Rennie PhD); and Centers for when they are, other factors such as page limitations, peer
Disease Control and Prevention, Atlanta, GA, USA (D F Stroup PhD) review, and editorial decisions may change the content and
format of the report before publication.
Correspondence to: Dr David Moher, Thomas C Chalmers Centre for
Systematic Reviews, Childrens Hospital of Eastern Ontario Research Several investigators have suggested guidelines for
Institute, 401 Smyth Road, Ottawa, Ontario K1H 8L1, Canada reporting of meta-analyses.3,19 However, a consensus across
(e-mail [email protected]) disciplines has not developed. After the initiative to

1896 THE LANCET Vol 354 November 27, 1999

improve the quality of reporting of randomised controlled the UK and North America who are interested in meta-
trials (RCTs),2022 we organised the Quality of Reporting of analysis. These 30 individuals were invited to a conference in
Meta-analyses (QUOROM) conference to address these Chicago on Oct 23, 1996. Participants were surveyed before
issues as they relate to meta-analyses of RCTs. This report the meeting to elicit their views on current reporting standards
of meta-analyses and whether these needed improvement. In
summarises the proceedings of that conference. The issues
addition, they were sent relevant citations for review and were
discussed might also be useful for reporting of systematic asked to indicate in which of the six groups they wished to
reviews (ie, meta-analysis, as defined above, without participate.
statistical aggregation), particularly of RCTs. The conference included small-group and plenary sessions.
Each small group had a facilitator who was a member of the
Methods steering committee and was responsible for ensuring the
The QUOROM steering committee began with a comprehensive discussions of as many as possible of the issues relevant to their
review of publications on the conduct and reporting of meta- specific remit. Each small group also had a recorder, who was
analyses. The databases searched included MEDLINE and the responsible for documenting the main points and the consensus on
Cochrane Library,23 which consists of the Cochrane Database of each issue discussed during that session; the recorder presented
Systematic Reviews, the Cochrane Controlled Trials Register, the the group's consensus during the plenary sessions. During the
York Database of Abstracts of Reviews of Effectiveness, and the plenary sessions, an elected scribe from each small group was
Cochrane Review Methodology Database. We examined reference responsible for recording the principal points relevant to that
lists of the retrieved articles and individual personal files. Articles group's charge that arose during the plenary discussion.
of potential relevance were retrieved and critically appraised by the The participants in each small group were asked to identify
QUOROM steering committee. The committee generated a draft items that they thought should be included in a checklist of
agenda for the conference, which included six domains requiring standards that would be useful for investigators, editors, and peer
discussion and debate. The content areas were slightly modified reviewers. We asked that, whenever possible, items included in the
during preliminary discussions at the conference and are reported checklist be guided by research evidence that suggested that a
as: the search for the evidence; decision-making on which evidence failure to adhere to the particular checklist item proposed could
to include; description of the characteristics of primary studies; lead to biased results. For example, a substantial lack of sensitivity
quantitative data synthesis; reliability and issues related to internal and specificity of MEDLINE searches is evident.24 Therefore, the
validity (or quality); and clinical implications related to external checklist suggests that investigators explicitly describe all search
validity (or generalisability). strategies used to locate articles for inclusion in a meta-analysis. In
In planning the QUOROM conference, the steering committee considering whether candidate items were essential, each subgroup
identified clinical epidemiologists, clinicians, statisticians, and used a modified Delphi technique25 that was replicated in the
researchers who conduct meta-analysis as well as editors from plenary sessions.

Heading Subheading Descriptor Reported? (Y/N) Page number

Title Identify the report as a meta-analysis [or systematic review] of RCTs 26
Abstract Use a structured format27

Describe
Objectives The clinical question explicitly
Data sources The databases (ie, list) and other information sources
Review methods The selection criteria (ie, population, intervention, outcome, and study design);
methods for validity assessment, data abstraction, and study characteristics, and
quantitative data synthesis in sufficient detail to permit replication
Results Characteristics of the RCTs included and excluded; qualitative and quantitative
findings (ie, point estimates and confidence intervals); and subgroup analyses
Conclusion The main results
Describe
Introduction The explicit clinical problem, biological rationale for the intervention, and rationale for review
Methods Searching The information sources, in detail28 (eg, databases, registers, personal files, expert
informants, agencies, hand-searching), and any restrictions (years considered, publication
status, 29 language of publication30,31)
Selection The inclusion and exclusion criteria (defining population, intervention, principal
outcomes, and study design32
Validity assessment The criteria and process used (eg, masked conditions, quality assessment, and their findings3336)
Data abstraction The process or processes used (eg, completed independently, in duplicate)35,36
Study characteristics The type of study design, participants characteristics, details of intervention, outcome
definitions, &c,37 and how clinical heterogeneity was assessed
Quantitative data synthesis The principal measures of effect (eg, relative risk), method of combining results
(statistical testing and confidence intervals), handling of missing data; how statistical
heterogeneity was assessed;38 a rationale for any a-priori sensitivity and subgroup analyses;
and any assessment of publication bias39
Results Trial flow Provide a meta-analysis profile summarising trial flow (see figure)
Study characteristics Present descriptive data for each trial (eg, age, sample size, intervention, dose, duration,
follow-up period)
Quantitative data synthesis Report agreement on the selection and validity assessment; present simple summary
results (for each treatment group in each trial, for each primary outcome); present data
needed to calculate effect sizes and confidence intervals in intention-to-treat analyses
(eg 232 tables of counts, means and SDs, proportions)
Discussion Summarise key findings; discuss clinical inferences based on internal and external validity;
interpret the results in light of the totality of available evidence; describe potential
biases in the review process (eg, publication bias); and suggest a future research agenda

Quality of reporting of meta-analyses

THE LANCET Vol 354 November 27, 1999 1897

Results
The conference resulted in the QUOROM statement: a
checklist (table) and a flow diagram (figure). The checklist
of standards for reporting of meta-analyses describes our
preferred way to present the abstract, introduction,
methods, results, and discussion sections of a report of a
meta-analysis. The checklist is organised into 21 headings
and subheadings to encourage authors to provide readers
with information on searches, selection, validity
assessment, data abstraction, study characteristics,
quantitative data synthesis, and trial flow. Authors are
asked to provide a flow diagram (figure) providing
information about the number of RCTs identified,
included, and excluded and the reasons for excluding
them.10

Pretesting
After development of the checklist and flow diagram, two
members of the steering committee (DM, DJC) undertook
pretesting with epidemiology graduate students studying
meta-analysis, residents in general internal medicine,
participants at a Canadian Cochrane Center workshop,
and faculty members of departments of medicine and of
epidemiology and biostatistics. One group of candidates for
a master's degree in epidemiology used the checklist and
flow diagram to report their meta-analyses as if their work
were being submitted for publication. Feedback from these
four groups was positive, most users stating that the Progress through the stages of a meta-analysis for RCTs
checklist and flow diagram would be likely to improve
reporting standards. Modifications of the checklist (eg, discussion should include comments about whether the
inclusion of a statement about major findings) and changes results obtained may have been influenced by such bias.
to the flow diagram (eg, more detail) were incorporated. Publication bias derives from the selective publishing of
studies with statistically significant or directionally positive
results,4042 and it can lead to inflated estimates of efficacy in
Discussion
meta-analyses. For example, trials of single alkylating
In developing the checklist, we identified supporting agents versus multiple-agent cytotoxic chemotherapy in the
scientific evidence for only eight of 18 items to guide the treatment of ovarian cancer have been analysed.39
reporting of meta-analyses of RCTs.26-39 Some of this Published trials yielded significant results in favour of the
evidence is indirect. For example, we ask authors to use a multiple-agent therapy, but that finding was not supported
structured abstract format. The supporting evidence for when the results of all trialsboth those published and
this item was collected by examining abstracts of original those registered but not publishedwere analysed.
reports of individual studies27 and may not pertain The statement asks authors to be explicit about the
specifically to the reporting of meta-analyses. However, the publication status of reports included in a meta-analysis.
QUOROM group judged this a reasonable approach by Only about a third of published meta-analyses report the
analogy with other types of research reports and pending inclusion of unpublished data.29,43 Although one study
further evidence about the merits of structured abstracts found that there were no substantial differences in the
for meta-analyses. dimensions of study quality between published and
We have asked authors to be explicit in reporting the unpublished clinical research,42 another suggested that
criteria used when assessing the quality of trials included intervention effects reported in journals were 33% greater
in meta-analyses and the outcome of the quality than those reported in doctoral dissertations.44 The role of
assessment. There is direct and compelling evidence to the grey literature (difficult to locate or retrieve) was
support recommendations about reporting on the quality examined in 39 meta-analyses that included 467 RCTs,
of RCTs included in a meta-analysis. A meta-analytic 102 of which were grey literature.29 Meta-analyses limited
database of 255 obstetric RCTs provided evidence that to published trials, compared with those that included both
trials with inadequate reporting of allocation concealment published and grey literature, overestimated the treatment
(ie, keeping the intervention assignments hidden from all effect by an average of 12%. There is still debate between
participants in the trial until the point of allocation) editors and investigators about the importance of including
overestimated the intervention effect by 30% compared unpublished data in a meta-analysis.43
with trials in which this information was adequately We have asked authors to be explicit in reporting
reported.33 Similar results for several disease categories and whether they have used any restrictions on language of
methods of quality assessment have been reported.34 These publication. Roughly a third of published meta-analyses
findings suggest that inclusion of reports of low-quality have some language restrictions as part of the eligibility
RCTs in meta-analyses is likely to alter the summary criteria for including individual trials.30 The reason for such
measures of the intervention effect. restrictions is not clear, since there is no evidence to
We also ask authors to be explicit in reporting support differences in study quality, and there is evidence
assessment of publication bias, and we recommend that the that language restrictions may result in a biased summary.

1898 THE LANCET Vol 354 November 27, 1999

The reports of 127 RCTs written in English, compared the QUOROM group need to survey the literature
with those reported in four other languages, showed little continually to help inform themselves about emerging
or no difference in several important methodological evidence on reporting of meta-analyses. This information
features.45 Similar results have been reported elsewhere.31 needs to be collated and presented annually for two
The role of language restrictions has been studied in 211 purposes. The first is decisions on which checklist items to
RCTs included in 18 meta-analyses in which trials keep, delete, or add; these decisions can be made similarly
published in languages other than English were included in to the selection of the original items. The second purpose is
the quantitative summary.30 Language-restricted meta- so that an up to date summary on the reporting of meta-
analyses overestimated the treatment effect by only 2% on analyses can be prepared. These efforts are being
average compared with language-inclusive meta-analyses. coordinated through a website. This approach is similar to
However, the language-inclusive meta-analyses were more the CONSORT initiative.
precise.30 In summary, our choice of items to include in a meta-
Reports of RCTs with statistically positive results are analysis report was based on evidence whenever possible,
more likely than those with negative results to be published which implies the need to include items that can
in English.31 Likewise, there is emerging evidence to systematically influence estimates of treatment effects.
suggest that reports of RCTs from certain countries mostly Currently, we lack a detailed understanding of all the
have statistically positive results.46 factors leading to bias in the result of a meta-analysis.
We used several methods to generate the checklist and Clearly, research is required to help improve the quality of
flow diagram: a systematic review of the reporting of meta- reporting of meta-analyses. Such evidence may also act as a
analyses; focus groups of the steering committee; and a catalyst for improving the methods by which meta-analyses
modified Delphi approach during the conference. are conducted.
Although we did not involve certain users of meta-analyses The QUOROM checklist and flow diagram are available
(policy-makers or patients), we formally pretested this on The Lancet's website [www.thelancet.com]. We hope
document with representatives of several constituencies that this document will generate further interest in the field
who would use the recommendations and made of meta-analysis and that, like the CONSORT initiative,
modifications accordingly. the QUOROM statement will become available in different
The QUOROM group also discussed the format of a languages and locations as it is disseminated. We invite
meta-analysis report, how best to assess the impact of the interested readers, reviewers, researchers, and editors to
QUOROM statement, and how best to disseminate it. The use the QUOROM statement and generate ideas for
format we recommend includes 15 subheadings that reflect improvement.
the sequential stages in the conduct of the meta-analysis Contributors
within the text of the report of a meta-analysis. The David Moher, Deborah Cook, Susan Eastwood, Ingram Olkin,
Drummond Rennie, and Donna Stroup developed the QUOROM
checklist included in the statement can also be used during statement. They all planned the meeting, participated in regular
the planning, performing, and reporting of a meta-analysis conference calls, identified and secured funding, identified and invited
and during peer review of the report after its submission to participants, and planned the meeting agenda. All of them helped write
a journal. the report, including revisions.
We delayed publication of the QUOROM statement QUOROM participants
until its impact on the editorial process had been assessed. D G Altman (ICRF/NHS Centre for Statistics in Medicine, Oxford, UK);
J A Berlin (University of Pennsylvania, Philadelphia, PA, USA); L Bero
We organised an RCT involving eight medical journals to (University of California, San Francisco, CA, USA); W DuMouchel
assess the impact of use of QUOROM criteria on journal (AT&T Laboratories, New York, NY, USA); K Dickersin (Brown
peer review. Accrual is now complete and we will report University, Providence, RI, USA); J J Deeks (ICRF/NHS Centre for
Statistics in Medicine, Oxford, UK); P Fontanarosa (JAMA, Chicago, IL,
the trial results elsewhere. USA); N Geller (National Heart, Lung, and Blood Institute, Bethesda,
After about 5 weeks of electronic posting we had MD, USA); F Godlee (BMJ, London, UK); S Goodman (Annals of
received five comments from investigators, whom we thank Internal Medicine, Philadelphia, PA, USA); R Horton (The Lancet,
(London, UK); P Huston (University of Ottawa, Ottawa, Canada);
for their thoughtful consideration of the statement. Several A R Jadad (McMaster University, Hamilton, Canada); K Kafadar
issues, in particular in relation to terminology, cannot be (University of Colorado, Denver, CO, USA); T Klassen (University of
addressed in the statement at present. The QUOROM Alberta, Edmonton, Canada); S Morton (RAND, Santa Monica, CA,
USA); C Mulrow (University of Texas, San Antonio, TX, USA); S Pyke
group is agreed on the importance of making changes to (GlaxoWellcome, London, UK); H S Sacks (Mount Sinai School of
the checklist in the light of documented evidence and must Medicine, New York, NY, USA); K F Schulz, (Family Health
resist changes based on opinion or anecdotal evidence International, Research Triangle Park, NC, USA); S G Thompson
(Imperial College School of Medicine, London, UK); M Winker (JAMA,
unless there is a compelling rationale for doing otherwise. Chicago, IL, USA); S Yusuf (McMaster University, Hamilton, Canada).
Nonetheless, the issues raised have been noted for
Acknowledgments
consideration and discussion in future. We thank Iain Chalmers, Ted Colton, Sander Greenland, Brian Haynes,
Several queries addressed the distinction between the Edward J Huth, Alessandro Liberati, Tom Louis, Roy Pitkin, David
meta-analysis and systematic review. As we indicate in the Sackett, Trevor Sheldon, and Chris Silagy, for reviewing earlier drafts of
this paper, and Jacqueline Page for helping with revisions.
introduction, and throughout the statement, the Financial support was provided by Abbott Laboratories, Agency for
QUOROM group agreed to observe the distinction as Health Care Policy & Research, GlaxoWellcome, and Merck & Co.
defined by the Potsdam consultation on meta-analysis.3
We were also asked to clarify the checklist item asking
References
investigators to interpret their results in light of the totality
1 Mulrow CD. The medical review article: state of the science.
of evidence. Increasingly, several meta-analyses on the Ann Intern Med 1987; 106: 48588.
same topic are reported.47-49 If other similar reports are 2 Cook DJ, Mulrow C, Haynes RB. Systematic reviews: synthesis of best
available, authors should discuss their results as they relate evidence for clinical decisions. Ann Intern Med 1997; 126: 37680.
to such evidence. 3 Cook DJ, Sackett DL, Spitzer W. Methodologic guidelines for
systematic reviews of randomized controlled trials in health care from
For the QUOROM statement to continue to be useful, it the Potsdam consultation on meta-analysis. J Clin Epidemiol 1995; 48:
must remain evidence based and up to date. Members of 16771.

THE LANCET Vol 354 November 27, 1999 1899

4 Deeks J, Glanville J, Sheldon T. Undertaking systematic reviews of 28 Tramr M, Reynolds DJM, Moore RA, McQuay HJ. Impact of covert
research on effectiveness CRD guidelines for those carrying out or duplicate publication on meta-analysis: a case study. BMJ 1997; 315:
commissioning reviews. CRD report no 4. York: NHS Centre for 63540.
Reviews and Dissemination, University of York, 1996. 29 McAuley L, Moher D, Tugwell P. The influence of grey literature on
5 Chalmers I, Haynes RB. Reporting, updating, and correcting meta-analysis. MSc Thesis: University of Ottawa, 1999.
systematic reviews of the effects of health care. In: Chalmers I, 30 Moher D, Pham B, Klassen TP, et al. Does the language of
Altman DG, eds. Systematic reviews. London: BMJ Publishing Group publication of reports of randomized trials influence the estimates of
1995: 86-95. intervention effectiveness reported in meta-analyses? 6th Cochrane
6 Bero L, Rennie D. The Cochrane Collaboration: preparing, Colloquium; 1998.
maintaining, and disseminating systematic reviews of the effects of 31 Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C,
health care. JAMA 1995; 274: 193538. Antes G. Language bias in randomised controlled trials published in
7 Huston P. The Cochrane Collaboration helping unravel tangled web English and German. Lancet 1997; 350: 32629.
woven by international research. Can Med Assoc J 1996; 154: 138992. 32 Khan KS, Daya S, Collins JA, Walter S. Empirical evidence of bias in
8 Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC. infertility research: overestimation of treatment effect in crossover
Meta-analyses of randomized controlled trials. N Engl J Med 1987; trials using pregnancy as the outcome measure. Fertil Steril 1996; 65:
316: 45055. 93945.
9 Sacks HS, Reitman D, Pagano D, Kupelnick B. Meta-analysis: an 33 Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of
update. Mt Sinai J Med 1996; 63: 21624. bias: dimensions of methodological quality associated with estimates
10 Mulrow CD, Oxman AD, eds. Cochrane Collaboration Handbook. of treatment effects in controlled trials. JAMA 1995; 273:
In: The Cochrane Library [database on disk and CDROM]. Oxford: 40812.
Cochrane Collaboration. Update Software: 1994, issue 4. 34 Moher D, Pham B, Jones A, et al. Does the quality of reports of
11 Oxman AD, Cook DJ, Guyatt GH, and the Evidence-Based Medicine randomised trials affect estimates of intervention efficacy reported in
Working Group. Users' guides to the medical literature: VI, how to meta-analyses? Lancet 1998; 352: 60913.
use an overview. JAMA 1994; 272: 136771. 35 Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports
12 Klassen TP, Jadad AR, Moher D. Guides for reading and interpreting of randomized clinical trials: is blinding necessary? Control Clin Trials
systematic reviews: 1, getting started. Arch Pediatr Adolesc Med 1998; 1996; 17: 112.
152: 70004. 36 Berlin JA on behalf of the University of Pennsylvania meta-analysis
13 L'Abb KA, Detsky AS, O'Rourke K. Meta-analysis in clinical blinding study group. Does blinding of readers affect the results of
research. Ann Intern Med 1987; 107: 22433. meta-analyses? Lancet 1997; 350: 18586.
14 Olkin I. A critical look at some popular meta-analytic methods. Am J 37 Barnes DE, Bero LA. Why review articles on the health effects of
Epidemiol 1984; 140: 28788. passive smoking reach different conclusions. JAMA 1998; 279:
15 Olkin I. Statistical and theoretical considerations in meta-analysis. 156670.
J Clin Epidemiol 1995; 48: 13346. 38 Thompson SG. Why sources of heterogeneity in meta-analysis should
16 Guyatt GH, Sackett DL, Sinclair J, Hayward R, Cook DJ, Cook RJ. be investigated. BMJ 1994; 309: 135155.
Users' guides to the medical literature: IX, a method for grading health 39 Simes RJ. Publication bias: the case for an international registry of
care recommendations. JAMA 1995; 274: 180004. clinical trials. J Clin Oncol 1986; 4: 152941.
17 Jadad AR, Cook DJ, Browman G. A guide to interpreting discordant 40 Sterling TD, Rosenbaum WL, Weinkam JJ. Publication decisions
systematic reviews. Can Med Assoc J 1997; 156: 141116. revisited: the effect of the outcome of statistical tests on the decision to
18 LeLorier J, Gregroire G, Benhaddad A, Lapierre J, Derderian F. publish and vice versa. Am Statist 1995; 49:10812.
Discrepancies between meta-analyses and subsequent large 41 Dickersin K, Min YI. NIH clinical trials and publication bias. Online J
randomized, controlled trials. N Engl J Med 1997; 337: 53642. Curr Clin Trials 1993: April 28; doc no 50.
19 Shea B, Dub C, Moher D. Assessing the quality of reports of meta- 42 Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias
analyses: a systematic review of scales and checklists. In: Egger M, in clinical research. Lancet 1991; 337: 86772.
Davey Smith G, Altman DG, eds. Systematic reviews, 2nd edn. 43 Cook DJ, Guyatt GH, Ryan G, et al. Should unpublished data be
London: BMJ Publishing Group (in press). included in meta-analyses? Current convictions and controversies.
20 The Standards of Reporting Trials Group. A proposal for structured JAMA 1993; 269: 274953.
reporting of randomized controlled trials. JAMA 1994; 272: 192631. 44 Smith ML. Publication bias and meta-analysis. Eval Educ 1980; 4:
21 The Asilomar Working Group on Recommendations for Reporting of 2224.
Clinical Trials in the Biomedical Literature. Checklist of information 45 Moher D, Fortin P, Jadad AR, et al. Completeness of reporting of
for inclusion in reports of clinical trials. Ann Intern Med 1996; 124: trials published in languages other than English: implications for
74143. conduct and reporting of systematic reviews. Lancet 1996; 347:
22 Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting 36366.
of randomized controlled trials: the CONSORT statement. JAMA 46 Vickers A, Goyal N, Harland R, Rees R. Do certain countries produce
1996; 276: 63739. only positive results? A systematic review of controlled trials. Control
23 The Cochrane Library [database on disk and CDROM]. Oxford: Clin Trials 1998; 19: 15966.
Cochrane Collaboration. Update Software, 1996, issue 3. 47 Kennedy E, Song F, Hunter R, Clark A, Gilbody S. Risperidone
24 Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for versus typical antipsychotic medication for schizophrenia (Cochrane
systematic reviews. BMJ 1994; 309: 128691. Review). In: Cochrane Library, issue 3. Oxford: Update Software,
25 Whitman N. The Delphi technique as an alternative for committee 1999.
meetings. J Nurs Educ 1990; 29: 37779. 48 Davies A, Adena MA, Keks NA, Catts SV, Lambert T, Schweitzer I.
26 Dickersin K, Higgins K, Meinert CL. Identification of meta-analyses: Risperidone versus haloperidol: I, meta-analysis of efficacy and safety.
the need for standard terminology. Control Clin Trials 1990; 11: 5266. Clin Ther 1998; 20: 5871.
27 Taddio A, Pain T, Fassos FF, Boon H, Illersich AL, Einarson TR: 49 Leucht S, Pitschel-Walz G, Abraham D, Kissling W. Efficacy and
Quality of nonstructured and structured abstracts of original research extrapyramidal side-effects of the new antipsychotics olanzapine,
articles in the British Medical Journal, the Canadian Medical quetiapine, risperidone, and sertindole compared to conventional
Association Journal and the Journal of the American Medical antipsychotics and placebo: a meta-analysis of randomized controlled
Association. Can Med Assoc J 1994; 150: 161115. trials. Schizophrenia Res 1999; 35: 5168.

1900 THE LANCET Vol 354 November 27, 1999