How to Conduct a Systematic Review
This tutorial works best in Internet Explorer, Chrome or Firefox. Recommend Internet Explorer if using accessibility software.
This tutorial introduces systematic reviews and explains the steps to a systematic review
Navigate to each section or step with the menu button in the top corner
This tutorial should take 20 minutes to complete
What is a Systematic Review?
A systematic review is a type of literature review that answers a specific research question
A systematic review involves explicit methods to:
- Perform a comprehensive literature search
- Complete a critical appraisal of the individual studies gathered
- Combine the valid studies using appropriate statistical techniques
Systematic reviews are considered one of the best sources of evidence on a given topic
Performing a Systematic Review:
Systematic reviews involve multiple steps and normally take six to twelve months to complete.
Systematic reviews are performed by a small team of individuals with various skills, including searching experts, statistical experts and subject experts.
Key characteristics of a systematic review:
- A clearly stated set of objectives with pre-defined inclusion and exclusion criteria
- An explicit, reproducible methodology
- A systematic search that attempts to identify all studies that would the eligibility criteria
- An assessment of the validity of the included studies and their findings (i.e. assessing whether a study may be biased)
- A systematic presentation and synthesis of the characteristics and findings of the included studies
Steps to a Systematic Review
- Formulate a question
- Develop protocol
- Conduct search
- Select studies and assess study quality
- Extract data and analyze/summarize and synthesize relevant studies
- Interpret results
Step 1: Formulate a Question
What problem are you trying to address by conducting the review? The research problem should be a structured and unambiguous question. The question should not be changed once the review process has begun, as the review protocol needs to be formed around the question
The PICO Model
The PICO model is a useful framework to break down the research question for a systematic review. A good, answerable question should follow the PICO model.
Population – the group you are trying to study: the test subjects or patients are the population
Intervention – the treatment, exposure or experimental factor to be studied
Comparison – the alternative to compare to the intervention
Outcome – is what the study hopes to measure or achieve
Example question: Does cognitive behavioural therapy prevent depression in individuals who have had a stroke?
Population – post-stroke patients
Intervention – cognitive behavioral therapy
Comparison – no treatment
Outcome – depression rates
There are many alternate models that can be used in place of the PICO model:
Alternate models can work for non-clinical, qualitative or social sciences research questions.
The best model depends on personal preference and the specific question you are trying to answer.
Two examples of alternate models are the ECLIPSE and the SPICE models. Click each for more information.
- Client Group
- Professionals Involved
- E (stands for nothing)
Step 2: Develop Protocol
The protocol is extremely important in systematic reviews. The protocol specifies the methods to be used in the review with the aim of minimizing bias. Transparency is key to a good systematic review, so the protocol needs to be clearly stated.
The protocol needs to include the following elements:
- Inclusion/exclusion criteria
- Searching strategy
- Study selection criteria and process
- Data extraction process
- Assessment of study quality process
- Data synthesis method(s)
- Result dissemination plans
Develop Protocol: Both the objective of the review and the question elements should be included in the protocol
The PICO question elements will help you select search terms and inclusion criteria
The first part of the systematic review protocol is clearly defining the inclusion/exclusion criteria. This is important for submitting to ethics boards and funding bodies. It also helps readers understand the process undertaken and evaluate the quality of your review
Explicitly decide what groups to include in the systematic review.
Decide how you want to limit your criteria before you start your literature search and include this in the protocol.
Inclusion Criteria: Types of Studies
- A systematic review should be based on the best quality evidence available (i.e. the highest level of evidence)
- When possible, use controlled-trials. However, depending on your topic, this might not be possible
- It may be beneficial to perform a scoping review prior to starting the systematic review process to know what quality of studies are available. Include this in your protocol
Inclusion Criteria: Population
- What population do you want to include in your systematic review?
- Decide what population you want to examine before you run your searches
- Inclusion criteria may include: age, gender, nationality
- Example: If you’re studying depression, it can present differently in children and adults
Outline the search strategy that will be used in the systematic review:
Terms – which search terms are you going to use?
Methods – what methods will you use? Are you using a searching expert such as a librarian?
Sources – what databases and sources will you use?
Study Selection Criteria and Process
Determine how studies will be selected for inclusion
How many people are reviewing the studies? What are their qualifications?
How will data be extracted from the selected studies?
Who is performing this? What tools are they using?
Assessment of Study Quality Process
Outline how the quality of included studies will be assessed
What tools or criteria are being used?
Data Synthesis Method(s)
How the results are being synthesized?
- The exact methods will depend on the data available in the included studies
- Include the general plans, such as whether or not a meta-analysis will be performed on the data.
- The protocol should also specify the outcomes of interest and what effect measures will be used
Result Dissemination Plans
How will the findings will be disseminated to reach the appropriate audience?
Step 3: Conduct Search
Conduct a search to find relevant articles for the systematic review. The search strategy will be outlined in the protocol. Once the protocol is in place, the searching process can begin
Use a structured search methodology when conducting a search
Steps to a Search:
- Define your topic/Ask a question (PICO)
State the question you formulated in the first step of the systematic review
- Identify the individual concepts
Pick out the key concepts in your question.
The number of concepts depends on the nature of your question.
Look at your PICO concepts for a place to start
- Search each concept separately
For each concept, consider synonyms, alternate spellings and related terms
Separate these with an OR in your search
Do a different search for each concept
- Combine concept sets
After searching separately, combine your concept searches with AND
- Review results and modify search
After searching review the results. If they are too general, too specific or irrelevant try a modified search
- Add limits/filters if necessary
If needed, limit the search with side-bar filters, such as document type, date range, language etc.
Subject Headings vs Keywords
To ensure you are conducting a comprehensive literature search, search using a combination of keywords and subject headings
Some databases search using subject headings and some search using keywords
Adapt your search strategy to each database
See Western Libraries Video on Subject Headings for more information
It may be beneficial to work with a librarian to help you design comprehensive search strategies across a variety of databases
The search should include scholarly articles as well as grey literature, such as reports published by government or non-profit agencies. For more information, check out our Grey Literature Tutorial
Keep a search log as you work. Keep track of what was searched in which database. This will help you re-run searches or update your systematic review. This is also important in providing transparency to the process
Collect ALL of the retrieved records from each search into a reference manager/citation management tool
Citation management software can:
- Keep track of articles
- Organize and categorize records
- Share articles with other members of the systematic review team
One Important feature of this software is that it can automatically detect duplicate articles, saving time and effort. De-duplicate the results in your tool prior to screening
To learn more, see the library website.
Step 4: Select Studies and Assess Study Quality
Once all results are gathered and de-duplicated, begin screening and assessing the studies
First, screen studies for relevance
Second, assess the quality of the remaining studies
Step 1: Title/abstract screening
Scan just the titles and abstracts to remove studies that are clearly not related to the topic
Step 2: Full-text screening
Use the inclusion/exclusion criteria to screen the full-text of studies
When assessing studies for inclusion, standardized assessment tools help eliminate bias and make for a better systematic review
The exact items on the checklist will depend on the type of studies you are including
There are pre-made checklists you can use such as the Joanna Briggs Institute Critical Appraisal Tools checklist for a randomized controlled trial
Have more than one person review all the collected articles
State in the protocol the number of reviewers and how discrepancies in reviewer selections will be resolved
Use the PRISMA flowchart to keep track of the resources collected or selected in each step
- You will likely have hundreds, if not thousands, of articles to screen
- This will help when writing up the systematic review
Step 5: Extract Data and Analyse, Summarize and Synthesize Relevant Studies
Once the included studies are finalized, use a data extraction form or systematic review software to extract all relevant data from each study
When the data has been extracted, analyze then synthesize the results
Have more than one person extract the data and check for mismatches
Record data exactly as reported
The analysis will depend on the type of data extracted
Meta-analysis involves using of statistical methods to summarize the results of the chosen studies
Whenever possible, perform a meta-analysis of the results. This will help improve the quality of the review and the conclusions
Step 6: Interpret Results
The last step is to interpret the results of the systematic review and disseminate them
This step involves writing up the systematic review
Clearly present your findings
- Include detailed methodology based on your protocol (search strategies used, selection criteria, etc.)
- This ensures that your review can be easily updated in the future with new research findings
Propose future steps
- Provide recommendations for practice and policy-making if sufficient, high quality evidence exists
- Indicate future directions for research to fill existing gaps in knowledge or to strengthen the body of evidence
Characteristics of a Good Systematic Review:
Ask yourself if your systematic review is explicit, comprehensive, reproducible, systematic and if bias has been minimized. If you can answer yes, you have a good systematic review
Best Practices for a Systematic Review
Adopt a Reporting Standard
Identify all sources consulted
Use a search log to keep track of searches
Manage references by a citation management tool
Report the qualifications of searchers
Visual presentation (i.e. Study Identification Flowchart)
Remember: transparency in the process and reproducibility of the searches make for a good systematic review
For access to tools and techniques to help you perform a high quality systematic review visit the Western Libraries Research Guide on Systematic Reviews or reach out to your library
Contact us for more assistance