Advertisement

Transparency Matters: Use Of Digital Software Platform In Replicating Systematic Reviews

      Objectives

      Organizations that submit systematic reviews (SR) and network meta analyses (NMA) to regulatory agencies, health technology assessment bodies, and guideline groups often employ manual methods and technologies that obscure decisions made and limit replicability. This research assessed the transparency of manual methods with comparison to software assisted replication and representation of SRs.

      Methods

      We considered four SRs, three non-governmental and one governmental agencies in the US and Europe, for multiple myeloma (MM), plaque psoriasis (PPs), Type 2 diabetes mellitus (T2DM) and multiple sclerosis (MS). Text descriptions of inclusion criteria (patients, interventions, comparators, outcomes) and meta-analysis methods were used to recreate each SR with software-assisted extraction and analysis tools to record and store decisions made at each step of the analyses. We then compared our results to that of each SR.

      Results

      Seventy-seven studies were included in the original SR: MM=9, MS=33, PPs=28, T2DM=7. The replication SR analysis identified 80 studies: MM=11, MS=34, PPs=28, T2DM=7. Efficacy rankings, point estimates and number of studies differed between the published and recreated SRs. For example, in the PPs NMA, the PASI 75 ranking for infliximab was #3 in the original analysis and #1 in the replication. Other factors that hindered replication included lack of reported search strategy, incorrect inclusion or exclusion of studies (MM and MS), insufficient details on analytical methods (MM), data extraction errors (PPs), and unclear grouping of outcomes (T2DM).

      Conclusions

      Traditional manual methods of performing SRs make it difficult to replicate SRs and analyses due to lack of transparency and traceability of methods; these can impact conclusions drawn. Use of digital platforms with artificial intelligence technology overseen by expert methodologists may improve quality and improve replicability. The use of such technological advancements may also facilitate dialogue with payers and reimbursement agencies by allowing more transparency in submissions.