This is a summary, written by members of the CITF Secretariat, of:

Bobrovitz N, Noël K, Li Z, Cao C, Deveaux G, Selemon A, Clifton DA, Yanes-Lane M, Yan T, Arora RK. SeroTracker-RoB: A decision rule-based algorithm for reproducible risk of bias assessment of seroprevalence studies. Res Synth Methods. 2023 May;14(3):414-426. doi: https://doi.org/10.1002/jrsm.1620.

The results and/or conclusions contained in the research do not necessarily reflect the views of all CITF members.

Making life easier for researchers doing systematic reviews of seroprevalence studies

Researchers at CITF-funded Serotracker present a new automated tool designed for risk of bias assessment (ROB). Because ROB can be a time consuming and subjective task when performing a systematic review, in this preprint, not yet peer reviewed, the authors offer a set of rules for transparent and reproducible assessments of seroprevalence studies. The SeroTracker ROB decision rules proved useful in classifying all studies in the SeroTracker database and showed good reliability. 

Critical appraisal of studies included in a systematic review is a critical step in insuring the quality of data that will be included. In doing so, researchers can classify the evidence at low, medium or high risk of bias, which allows a more in-depth interpretation of the findings. As a whole this process is known as risk of bias assessment (ROB). The SeroTracker ROB approach for seroprevalence studies uses a modified version of the standardized Joanna Briggs Institute (JBI) Critical Appraisal Checklist for Prevalence Studies together with decision rules that can be applied to the JBI checklist ratings to generate an overall risk of bias assessment.

The decision rules were developed following an evaluation of thousands of seroprevalence studies with published guidance on estimating disease prevalence, reports on the evaluation of prevalence studies, opinions of experts in evidence synthesis and infectious disease epidemiology, and the consensus of researchers at SeroTracker. Seven out of the nine items on the JBI checklist could be automatically completed, however it will still require manual evaluation of the following:

  • Whether the sample frame is representative of the target population; and
  • Whether the characteristics of the sample are representative of the target population in both the main and sub-group analyses.

The decision tree considered different pathways based on the categorical ratings for each item of the JBI in a forward stepwise manner.

The tool was tested using studies contained in the SeroTracker database and by comparing the results from using this approach to the results from a manual evaluation. The SeroTracker approach was able to classify 100% of the SARS-CoV-2 seroprevalence studies (2,070) in the SeroTracker database (as high, medium or low ROB) and reliability compared to manual review was 77%.

The SeroTracker ROB decision rules can support transparent and reproducible evidence synthesis of infection prevalence studies and may have particular value during outbreaks and pandemics.