Steps for

Conducting
Systematic Reviews
Irena Bond,
Associate Professor/Library Manager
Massachusetts College of Pharmacy and Health Sciences (MCPHS)
[email protected]

June 28, 2023

Goals
• To provide a step-by-step approach for properly conducting
Systematic Reviews (SR) for beginners and junior researchers

• To provide links to accepted and widely used international templates,

checklists, and guidelines used in conducting SR
What is a SR?
• A systematic review is a review that uses a systematic method to
summarize evidence on questions with a detailed and comprehensive
plan of study

• Opposite - Unsystematic narrative review tends to be descriptive, in

which the authors select articles based on their point of view which
leads to its bias and poorer quality
Why SR?
• Abundance of studies – clinical medicine is more complex
• Studies are often heterogeneous in their design, quality, subjects…
• A well-conducted SR is a feasible solution to keep up-to-date of
current Evidence-Based Medicine (EBM)
• Understanding of SR steps is important for its conduction. It is not
easy to be done
• All steps of SR should be done independently by 2–3 reviewers’
discussion, to ensure data quality and accuracy
Steps in Conducting SR (1)
1. Development of research question
2. Preliminary research/Forming inclusion/exclusion criteria
3. Search strategy
4. Searching databases
5. Protocol registration
6. Title
7. Abstract
8. Full-text screening
9. Manual searching
10. Extracting data
11. Quality assessment
12. Data checking
13. Statistical analysis (there are more steps and variety of meta analysis)
14. Double data checking
15. Manuscript writing and submission
 Steps for conducting SR – Flow chart (1)

Fig. 1 Detailed flow diagram guideline for systematic review and meta-analysis steps. Note: Star icon refers to “2–3 reviewers screen independently”
Research Question
• Should be feasible, interesting, novel, ethical, and relevant. It should
be clear, logical, and well-defined
• PICO question framework (usually used for SR of clinical trial study)
– Patient/Population, Intervention, Comparison, Outcome
• PICO for an observational study - use without intervention or
comparator, in many cases it is usually enough to use P (Patient) and
O (outcome) only or PEO-patient, exposure, outcome
Example
We want to know whether the Ebola vaccine is safe and can induce
sufficient immunogenicity to the subjects.
Question: How is the safety and immunogenicity of Ebola vaccine in
human?
P: healthy subjects (human)
I: vaccination
C: placebo
O: safety or adverse effects
Preliminary Research and Idea
Validation (1)
• Start by doing a simple search in PubMed or Google Scholar with
search terms Ebola AND vaccine
• Why? to identify relevant articles, ensure the validity of the proposed
idea, avoid duplication of previously addressed questions, and assure
that we have enough articles for conducting its analysis
• We find a SR of determinant factors influencing antibody response
from vaccination of Ebola vaccine in non-human primate and human,
which is a relevant paper to read to get a deeper insight and identify
gaps for better formulation of our research question or purpose
• We can still conduct systematic review and meta-analysis of Ebola
vaccine because we evaluate safety as a different outcome and
different population (only human)
Inclusion and Exclusion Criteria
• Eligibility criteria - based on the PICO approach, study design, and
date.
• Exclusion criteria - unrelated, duplicated, unavailable full texts, or
abstract-only papers. Exclusions should be stated in advance to refrain
the researcher from bias
• Inclusion criteria - articles with the target patients, investigated
interventions, or the comparison between two studied interventions.
They contain information answering our research question. Most
importantly there should be clear and sufficient information, including
positive or negative, to answer the question
Example – Inclusion & Exclusion
Criteria
Inclusion Criteria:
• any clinical trial evaluating the safety of Ebola vaccine
• no restriction regarding country, patient age, race, gender, publication language,
and date

Exclusion criteria:
• study of Ebola vaccine in non-human subjects or in vitro studies
• Study with data not reliably extracted, duplicate, or overlapping data
• abstract-only papers as preceding papers, conference, editorial, and author
response theses and books
• articles without available full text available;
• case reports, case series, and systematic review studies.
Search Strategy
• A standard search strategy is used in PubMed first
• Later the PubMed search strategy is modified according to each specific
database to get the best relevant results
• The basic search strategy is built based on the research question formulation
(i.e.,PICO). Search strategies are constructed to include free-text terms (e.g.,
in the title and abstract) and any appropriate subject indexing (e.g., MeSH)
expected to retrieve eligible studies, with the help of an expert in the review
topic field or an information specialist
• Outcome terms are usually not searched because outcome is not mentioned
obviously in the articles
• The improvement of the search term is made while doing a trial search and
looking for another relevant term within each concept from retrieved papers
• A detailed search strategy for each database is developed next
Example:
• The final search strategy for PubMed is as follows:

(ebola OR ebola virus OR ebolavirus OR ebola virus disease OR EVD)

AND (vaccine OR vaccination OR vaccinated OR immunization) AND
(“clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH
Terms] OR “clinical trial”[All Fields])

• Because the study for this topic is limited, we do not include outcome
term (safety and immunogenicity) in the search term to capture more
studies.
 PRISMA Flow
Diagram

Fig. 2 PRISMA flow diagram of studies’ screening and selection

Search databases, import all results to
a library, and export to an excel sheet
• According to the AMSTAR guidelines, at least two databases have to be
searched in the SR, but as you increase the number of searched
databases, you get much yield and more accurate and comprehensive
results
• The ordering of the databases depends mostly on the review questions;
being in a study of clinical trials, you will rely mostly on Cochrane, mRCTs,
or International Clinical Trials Registry Platform (ICTRP)
• Proposed databases for our topic: PubMed, Scopus, Web of Science,
EMBASE, GHL, VHL, Cochrane, Google Scholar, Clinical trials.gov, mRCTs,
POPLINE, and SIGLE (grey literature)
• Finally, all records are collected into one Reference Manager (e.g. Zotero,
Mendeley, etc. ) library in order to delete duplicates and then to export
records into an excel sheet with essential information for screening
Google Scholar Advanced
Search Example
1. With all of the words: ebola virus
With at least one of the words: vaccine vaccination vaccinated
immunization
Where my words occur: in the title of the article

2. With all of the words: EVD

With at least one of the words: vaccine vaccination vaccinated
immunization
Where my words occur: in the title of the article
 Protocol Writing and
Registration
• Protocols describe the rationale, hypothesis, and planned methods of the project
before the review has begun and allow readers of completed reviews to identify
deviations from planned methods and determine whether the bias impacts
the interpretation of review results and conclusions (e.g. criteria you will use to
screen literature, how you will search for literature, QA tools you chose etc.)
• Protocol registration is important at an early stage
• It guarantees transparency in the research process, feedback from other
researchers, and protects from duplication problems
• It is considered a documented proof of team plan of action, research question,
eligibility criteria, intervention/exposure, quality assessment, and pre-analysis plan
• It is recommended that researchers send it to the principal investigator (PI) to revise
it, then upload it to registry sites
• There are many registry sites available for SR like by Cochrane and Campbell
collaborations; PROSPERO is recommended as it is easier. There is a PROSPERO (
https://www.crd.york.ac.uk/prospero/) protocol template (1)
Title and Abstract Screening
• Decisions to select retrieved articles for further assessment are based
on eligibility criteria, to minimize the chance of including non-relevant
articles
• When there is a doubt about an article decision, the team should be
inclusive rather than exclusive, until the main leader or PI makes a
decision after discussion and consensus
• All excluded records should be given exclusion reasons
Full text Downloading and
Screening
• Reserach4Life Resources

• Many search engines provide links for free to access full text articles
In case not found, we can search in some research websites as
Research Gate, which offer an option of direct full-text request from
authors.

• Contact your librarian to explore other options

Manual search
• Searching reference lists from included studies/reviews
• Contacting authors and experts
• Looking at cited articles in PubMed and Google Scholar (citation tracking- track
all the articles that cite each one of the included articles)
• Looking at relevant (similar) articles
• All the possible relevant article must undergo further scrutiny against the
inclusion criteria, after following the same records yielded from electronic
databases, i.e., title/ab abstract and full-text screening
• Each reviewer has a “tag” and a distinct method to compile all the results at the
end for comparison of differences and discussion and to maximize the retrieval
and minimize the bias
• The number of included articles has to be stated before addition to the overall
included records
 Data Extraction and Quality
Assessment
• Collect data from included full-texts in a structured extraction excel sheet
• Data presented in graphs is extracted by software tools, e.g. Web plot digitizer (1)
• The sheet is classified into the study and patient characteristics, outcomes, and
quality assessment (QA)
• In the EBOLA example, extract: name of authors, country of patients, year of
publication, study design (case report, cohort study, or clinical trial or RCT), sample
size, the infected point of time after EBOLA infection, follow-up interval after
vaccination time, efficacy, safety, adverse effects after vaccinations, and QA sheet
• To evaluate the risk of bias, reviewers should rate each of the 14 items into
dichotomous variables: yes, no, or not applicable (NA). An overall score is calculated
by adding all the items scores as yes=1, while no and NA = 0. A score will be given
for every paper to classify them as poor, fair, or good, where a score from 0–5 was
considered poor, 6–9 as fair, and 10–14 as good
• QA tools (relevance, reliability, validity, applicability) (4). Variety of tools are
available for the QA, depending on the design (1)
Data Checking
• Every included article is compared with its counterpart in an
extraction sheet, to detect mistakes in data
• 2–3 independent reviewers should do the data checking ideally not
the ones who performed the extraction of those articles. When
resources are limited, each reviewer is assigned a different article
than the one he extracted in the previous stage
Statistical analysis
• Investigators use different methods for combining and summarizing
findings of included studies. Before analysis, there is an important
step called cleaning of data in the extraction sheet, where the analyst
organizes extraction sheet data in a form that can be read by
analytical software

• The analysis consists of 2 types namely qualitative and quantitative

analysis. Qualitative analysis mostly describes data in SR studies,
while quantitative analysis consists of two main types: MA and
network meta-analysis (NMA) (1)
Double Data Checking
• For more assurance on the quality of results, the analyzed data should
be rechecked from full-text data by evidence photos, to allow an
obvious check for the PI of the study.
Manuscript writing, revision,
and submission to a journal
• Writing based on four scientific sections: introduction, methods,
results, and discussion, conclusions.
• Having a characteristic table for study and patient characteristics is a
mandatory step (see template in paper)
• After finishing the manuscript writing, characteristics table, and
PRISMA flow diagram, the team should send it to the PI to revise it
well and reply to his comments
• Finally, choose a suitable journal for the manuscript which fits with
considerable impact factor and fitting field. Pay attention by reading
the author guidelines of journals before submitting the manuscript
References:
1. Tawfik, G.M., Dila, K.A.S., Mohamed, M.Y.F. et al. A step by step guide for
conducting a systematic review and meta-analysis with simulation
data. Trop Med Health 47, 46 (2019). doi.org/10.1186/s41182-0190165-6
2. UNC Libraries. Forming Focused Questions with PICO and other Question
Frameworks. Available at: https://guides.lib.unc.edu/pico/about
3. Morgan RL, Whaley P, Thayer KA, Schünemann HJ. Identifying the PECO: A
framework for formulating good questions to explore the association of
environmental and other exposures with health outcomes. Environ Int.
2018;121(Pt 1):1027-1031. doi:10.1016/j.envint.2018.07.015
4. UNC Libraries. Systematic Reviews: Step 6: Assess Quality of Included Studies. Available at:
https://guides.lib.unc.edu/systematic-reviews/assess-quality
5. Carnegie Melon University. Free Systematic Review and Mata analysis Course.
Available at:
https://oli.cmu.edu/courses/systematic-reviews-and-meta-analysis-o-f/
6. Cochrane. Students 4 Best Evidence. Available at: https://s4be.cochrane.org/