Transparency Matters: Use Of Digital Software Platform In Replicating Systematic Reviews


      Organizations that submit systematic reviews (SR) and network meta analyses (NMA) to regulatory agencies, health technology assessment bodies, and guideline groups often employ manual methods and technologies that obscure decisions made and limit replicability. This research assessed the transparency of manual methods with comparison to software assisted replication and representation of SRs.


      We considered four SRs, three non-governmental and one governmental agencies in the US and Europe, for multiple myeloma (MM), plaque psoriasis (PPs), Type 2 diabetes mellitus (T2DM) and multiple sclerosis (MS). Text descriptions of inclusion criteria (patients, interventions, comparators, outcomes) and meta-analysis methods were used to recreate each SR with software-assisted extraction and analysis tools to record and store decisions made at each step of the analyses. We then compared our results to that of each SR.


      Seventy-seven studies were included in the original SR: MM=9, MS=33, PPs=28, T2DM=7. The replication SR analysis identified 80 studies: MM=11, MS=34, PPs=28, T2DM=7. Efficacy rankings, point estimates and number of studies differed between the published and recreated SRs. For example, in the PPs NMA, the PASI 75 ranking for infliximab was #3 in the original analysis and #1 in the replication. Other factors that hindered replication included lack of reported search strategy, incorrect inclusion or exclusion of studies (MM and MS), insufficient details on analytical methods (MM), data extraction errors (PPs), and unclear grouping of outcomes (T2DM).


      Traditional manual methods of performing SRs make it difficult to replicate SRs and analyses due to lack of transparency and traceability of methods; these can impact conclusions drawn. Use of digital platforms with artificial intelligence technology overseen by expert methodologists may improve quality and improve replicability. The use of such technological advancements may also facilitate dialogue with payers and reimbursement agencies by allowing more transparency in submissions.