How to Conduct a Systematic Review

Email us with questions or suggestions on more Videos & How-Tos

This tutorial works best in Internet Explorer, Chrome or Firefox. Recommend Internet Explorer if using accessibility software. 

Script:
How to Conduct a Systematic Review

Introduction 

This tutorial introduces systematic reviews and explains the steps to a systematic review

Navigate to each section or step with the menu button in the top corner

This tutorial should take 20 minutes to complete

What is a Systematic Review?

A systematic review is a type of literature review that answers a specific research question

A systematic review involves explicit methods to:

  • Perform a comprehensive literature search
  • Complete a critical appraisal of the individual studies gathered
  • Combine the valid studies using appropriate statistical techniques

 Systematic reviews are considered one of the best sources of evidence on a given topic

Performing a Systematic Review:

Systematic reviews involve multiple steps and normally take six to twelve months to complete.

Systematic reviews are performed by a small team of individuals with various skills, including searching experts, statistical experts and subject experts.

Key characteristics of a systematic review:

  • A clearly stated set of objectives with pre-defined inclusion and exclusion criteria
  • An explicit, reproducible methodology
  • A systematic search that attempts to identify all studies that would the eligibility criteria
  • An assessment of the validity of the included studies and their findings (i.e. assessing whether a study may be biased)
  • A systematic presentation and synthesis of the characteristics and findings of the included studies

Steps to a Systematic Review

  1. Formulate a question
  2. Develop protocol
  3. Conduct search
  4. Select studies and assess study quality
  5. Extract data and analyze/summarize and synthesize relevant studies
  6. Interpret results

Step 1: Formulate a Question

What problem are you trying to address by conducting the review? The research problem should be a structured and unambiguous question. The question should not be changed once the review process has begun, as the review protocol needs to be formed around the question

The PICO Model

The PICO model is a useful framework to break down the research question for a systematic review. A good, answerable question should follow the PICO model.

Population – the group you are trying to study: the test subjects or patients are the population

Intervention – the treatment, exposure or experimental factor to be studied

Comparison – the alternative to compare to the intervention

Outcome – is what the study hopes to measure or achieve

Example question: Does cognitive behavioural therapy prevent depression in individuals who have had a stroke?

Population – post-stroke patients

Intervention – cognitive behavioral therapy

Comparison – no treatment

Outcome – depression rates

Alternate Models

There are many alternate models that can be used in place of the PICO model:

Alternate models can work for non-clinical, qualitative or social sciences research questions.

The best model depends on personal preference and the specific question you are trying to answer.

Two examples of alternate models are the ECLIPSE and the SPICE models. Click each for more information.

ECLIPSE Model

  • Expectations
  • Client Group
  • Location
  • Impact
  • Professionals Involved
  • Service
  • E (stands for nothing)

 

SPICE Model

  • Setting
  • Perspective
  • Intervention
  • Comparison
  • Evaluation

Step 2: Develop Protocol

The protocol is extremely important in systematic reviews. The protocol specifies the methods to be used in the review with the aim of minimizing bias. Transparency is key to a good systematic review, so the protocol needs to be clearly stated.

The protocol needs to include the following elements:

  1. Inclusion/exclusion criteria
  2. Searching strategy
  3. Study selection criteria and process
  4. Data extraction process
  5. Assessment of study quality process
  6. Data synthesis method(s)
  7. Result dissemination plans

Develop Protocol: Both the objective of the review and the question elements should be included in the protocol

The PICO question elements will help you select search terms and inclusion criteria

  1. Inclusion/Exclusion Criteria

The first part of the systematic review protocol is clearly defining the inclusion/exclusion criteria. This is important for submitting to ethics boards and funding bodies. It also helps readers understand the process undertaken and evaluate the quality of your review

Explicitly decide what groups to include in the systematic review.

Decide how you want to limit your criteria before you start your literature search and include this in the protocol.

Inclusion Criteria: Types of Studies

  • A systematic review should be based on the best quality evidence available (i.e. the highest level of evidence)
  • When possible, use controlled-trials. However, depending on your topic, this might not be possible
  • It may be beneficial to perform a scoping review prior to starting the systematic review process to know what quality of studies are available. Include this in your protocol

Inclusion Criteria: Population

  • What population do you want to include in your systematic review?
  • Decide what population you want to examine before you run your searches
  • Inclusion criteria may include: age, gender, nationality
  • Example: If you’re studying depression, it can present differently in children and adults
  1. Search Strategy

Outline the search strategy that will be used in the systematic review:

Terms – which search terms are you going to use?

Methods – what methods will you use? Are you using a searching expert such as a librarian?

Sources – what databases and sources will you use?

  1. Study Selection Criteria and Process

Determine how studies will be selected for inclusion

How many people are reviewing the studies? What are their qualifications?

  1. Data Extraction

How will data be extracted from the selected studies?

Who is performing this?  What tools are they using?

  1. Assessment of Study Quality Process

Outline how the quality of included studies will be assessed

What tools or criteria are being used?

  1. Data Synthesis Method(s)

How the results are being synthesized?

  • The exact methods will depend on the data available in the included studies
  • Include the general plans, such as whether or not a meta-analysis will be performed on the data.
  • The protocol should also specify the outcomes of interest and what effect measures will be used
  1. Result Dissemination Plans

How will the findings will be disseminated to reach the appropriate audience?

Step 3: Conduct Search

Conduct a search to find relevant articles for the systematic review. The search strategy will be outlined in the protocol. Once the protocol is in place, the searching process can begin

Use a structured search methodology when conducting a search

 Steps to a Search:

  1. Define your topic/Ask a question (PICO)

State the question you formulated in the first step of the systematic review

  1. Identify the individual concepts

Pick out the key concepts in your question.

The number of concepts depends on the nature of your question.

Look at your PICO concepts for a place to start

  1. Search each concept separately

For each concept, consider synonyms, alternate spellings and related terms

Separate these with an OR in your search

Do a different search for each concept

  1. Combine concept sets

After searching separately, combine your concept searches with AND

  • Review results and modify search

After searching review the results. If they are too general, too specific or irrelevant try a modified search

  • Add limits/filters if necessary

If needed, limit the search with side-bar filters, such as document type, date range, language etc.

Subject Headings vs Keywords

To ensure you are conducting a comprehensive literature search, search using a combination of keywords and subject headings

Some databases search using subject headings and some search using keywords

Adapt your search strategy to each database

See Western Libraries Video on Subject Headings for more information

Search Strategy

It may be beneficial to work with a librarian to help you design comprehensive search strategies across a variety of databases

The search should include scholarly articles as well as grey literature, such as reports published by government or non-profit agencies. For more information, check out our Grey Literature Tutorial

Search Log

Keep a search log as you work. Keep track of what was searched in which database. This will help you re-run searches or update your systematic review. This is also important in providing transparency to the process

Citation Management

Collect ALL of the retrieved records from each search into a reference manager/citation management tool

Citation management software can:

  • Keep track of articles
  • Organize and categorize records
  • Share articles with other members of the systematic review team

One Important feature of this software is that it can automatically detect duplicate articles, saving time and effort. De-duplicate the results in your tool prior to screening

 To learn more, see the library website

 Step 4: Select Studies and Assess Study Quality

Once all results are gathered and de-duplicated, begin screening and assessing the studies

First, screen studies for relevance

Second, assess the quality of the remaining studies

Screening Results

Step 1: Title/abstract screening

Scan just the titles and abstracts to remove studies that are clearly not related to the topic

Step 2: Full-text screening

            Use the inclusion/exclusion criteria to screen the full-text of studies

Assessing Results

When assessing studies for inclusion, standardized assessment tools help eliminate bias and make for a better systematic review

The exact items on the checklist will depend on the type of studies you are including

There are pre-made checklists you can use such as the Joanna Briggs Institute Critical Appraisal Tools checklist for a randomized controlled trial

Have more than one person review all the collected articles

State in the protocol the number of reviewers and how discrepancies in reviewer selections will be resolved

PRISMA Flowchart

Use the PRISMA flowchart to keep track of the resources collected or selected in each step

  • You will likely have hundreds, if not thousands, of articles to screen
  • This will help when writing up the systematic review

Step 5: Extract Data and Analyse, Summarize and Synthesize Relevant Studies

Once the included studies are finalized, use a data extraction form or systematic review software to extract all relevant data from each study

When the data has been extracted, analyze then synthesize the results

 Extract data

Have more than one person extract the data and check for mismatches

Record data exactly as reported

 Analyse Data

The analysis will depend on the type of data extracted

Meta-analysis involves using of statistical methods to summarize the results of the chosen studies

Whenever possible, perform a meta-analysis of the results. This will help improve the quality of the review and the conclusions

Step 6: Interpret Results

The last step is to interpret the results of the systematic review and disseminate them

This step involves writing up the systematic review

 

Presenting Results

Clearly present your findings

  • Include detailed methodology based on your protocol (search strategies used, selection criteria, etc.)
  • This ensures that your review can be easily updated in the future with new research findings

Propose future steps

  • Provide recommendations for practice and policy-making if sufficient, high quality evidence exists
  • Indicate future directions for research to fill existing gaps in knowledge or to strengthen the body of evidence

Characteristics of a Good Systematic Review:

Explicit

Comprehensive

Reproducible

Systematic

Bias Minimized

Ask yourself if your systematic review is explicit, comprehensive, reproducible, systematic and if bias has been minimized. If you can answer yes, you have a good systematic review

Best Practices for a Systematic Review

Adopt a Reporting Standard

Identify all sources consulted

Use a search log to keep track of searches

Manage references by a citation management tool

Report the qualifications of searchers

Visual presentation (i.e. Study Identification Flowchart)

 

Remember: transparency in the process and reproducibility of the searches make for a good systematic review

For access to tools and techniques to help you perform a high quality systematic review visit the Western Libraries Research Guide on Systematic Reviews or reach out to your library