Trials we cannot trust: investigating their impact on systematic reviews and clinical guidelines in spinal pain

Abstract

Copyright © 2023 The Author(s). We previously conducted an exploration of the trustworthiness of a group of clinical trials of cognitive behavioural therapy (CBT) and exercise in spinal pain. We identified multiple concerns in eight trials, judging them untrustworthy. In this study, we systematically explored the impact of these trials (“index trials”) on results, conclusions and recommendations of systematic reviews and clinical practice guidelines (CPGs). We conducted forward citation tracking using Google Scholar and the citationchaser tool, searched the Guidelines International Network (GIN) library and National Institute of Health and Care Excellence (NICE) archive to June 2022 to identify systematic reviews and CPGs. We explored how index trials impacted their findings. Where reviews presented meta-analyses, we extracted or conducted sensitivity analyses for the outcomes pain and disability, to explore how exclusion of index trials affected effect estimates. We developed and applied an ’Impact Index’ to categorise the extent to which index studies impacted their results. We included 32 unique reviews and 10 CPGs. None directly raised concerns regarding the veracity of the trials. Across meta-analyses (55 comparisons), removal of index trials reduced effect sizes by a median 58% (IQR 40 to 74). 85% of comparisons were classified as highly, 3% as moderately, and 11% as minimally impacted. Nine out 10 reviews conducting narrative synthesis drew positive conclusions regarding the intervention tested. Nine out of 10 CPGs made positive recommendations for the intervention(s) evaluated. This cohort of trials, with concerns regarding trustworthiness, has substantially impacted the results of systematic reviews and guideline recommendations.

Perspective: We found that a group of trials of CBT for spinal pain with concerns relating to their trustworthiness have had substantial impacts on the analyses and conclusions of systematic reviews and clinical practice guidelines. This highlights the need for a greater focus on the trustworthiness of studies in evidence appraisal.

Pre-registration: Our protocol was pre-registered on the Open Science Framework: https://osf.io/m92ax/