The Number of Patients Lost to Follow-up May Exceed the Fragility Index of a Randomized Controlled Trial Without Reversing Statistical Significance: A Systematic Review and Statistical Model

Abstract

PurposeTo 1) analyze trends in the publishing of statistical fragility index (FI)-based systematic reviews in the orthopaedic literature, including the prevalence of misleading or inaccurate statements related to the statistical fragility of randomized controlled trials (RCTs) and patients lost to follow-up (LTF) and 2) determine whether RCTs with relatively “low” FIs are truly as sensitive to patients LTF as previously portrayed in the literature. MethodsAll FI-based studies published in the orthopaedic literature were identified using the Cochrane Database of Systematic Reviews, Web of Science Core Collection, PubMed, and MEDLINE databases. All articles involving application of the FI or reverse FI (RFI) to study the statistical fragility of studies in orthopaedics were eligible for inclusion in the study. Study characteristics, median FIs and sample sizes, and misleading or inaccurate statements related to the FI and patients LTF were recorded. Misleading or inaccurate statements were defined as those basing conclusions of trial fragility on the false assumption that adding patients LTF back to a trial has the same statistical effect as existing patients in a trial experiencing the opposite outcome and were determined by two authors. A theoretical RCT with a sample size of 100, p-value of 0.006, and an FI of 4 was used to evaluate the difference in effect on statistical significance between flipping outcome events of patients already included in the trial (the FI) vs. adding patients LTF back to the trial to demonstrate the true sensitivity of RCTs to patients LTF. ResultsOf the 39 FI-based studies, 37 (95%) directly compared the FI to the number of patients lost to follow-up. Of these, 22 (59%) included a statement regarding the FI and patients LTF that was determined to be inaccurate or misleading. In the theoretical RCT, a reversal of significance was not observed until 7 patients LTF (nearly twice the FI) were added to the trial in the distribution of maximal significance reversal. ConclusionsThe claim that any RCT in which the number of patients LTF exceeds the FI could potentially have its significance reversed simply by maintaining study follow-ups is commonly inaccurate and prevalent in orthopaedic studies applying the FI. Patients LTF and the FI are not equivalent. The minimum number of patients LTF required to flip the significance of a typical RCT was demonstrated to be greater than the FI, suggesting RCTs with relatively “low” FIs may not be as sensitive to patients LTF as previously portrayed in the literature; however, only a holistic approach that considers the context in which the trial was conducted, potential biases, and study results can determine the merits of any particular RCT.