Assessing the Robustness of Positive Vascular Surgery Randomized Controlled Trials Using Their Fragility Index

Abstract

The fragility index (FI) measures the robustness of statistically significant findings in randomized controlled trials (RCTs) calculated as the minimum number of event conversions required to alter the result’s statistical significance. In vascular surgery, many clinical guidelines and critical decision-making points are informed by a handful of key RCTs, especially regarding open surgical vs endovascular treatment. The objective of this study is to evaluate the FI of RCTs with statistically significant primary outcomes that compared open vs endovascular surgery. MEDLINE, Embase, and CENTRAL were searched for RCTs comparing open vs endovascular treatments for abdominal aortic aneurysms, carotid artery stenosis, and peripheral arterial disease to December 2022. RCTs with statistically significant primary dichotomous outcomes were included. Data screening and extraction were conducted in duplicate. The FI was calculated by adding an event to the group with the smaller number of events while subtracting a nonevent to the same group until the Fisher exact test produced a non-statistically significant result (P > .05, Table I). The primary outcome measures were FI and proportion of outcomes where loss to follow-up was greater than FI. Secondary analyses were performed to determine whether FI varied by study disease state, presence of commercial funding, and study design. Overall, 5133 articles were captured in the initial search with 21 RCTs reporting 23 primary outcomes being included in the final analysis. The median FI [first quartile, third quartile] was 3 [3, 20] with 16 outcomes (70%) reporting loss to follow-up greater than its FI. The Mann-Whitney U test revealed that commercially funded RCTs and composite outcomes had greater FIs (20 vs 3, P = .035; 21 vs 3, P = .01, respectively). The FI did not vary between disease states (P = .285) or between index trials vs follow-up analyses (P = .147, Table II). There were significant correlations between FI and P value (Pearson r = 0.901, P < .001) and number of events (r = 0.832, P < .001). A small number of event conversions (median 3) are needed to alter the statistical significance of primary outcomes in vascular surgery RCTs evaluating open surgical and endovascular treatments. Most studies had loss to follow-up greater than its FI, which can call into question trial results, and commercially funded studies had a greater FI. The FI and these findings should be considered in future trial design in vascular surgery.Table ITwo-by-two contingency tables highlighting the calculation of the fragility index (FI), with the FI calculated (FI = 70) for the BEST-CLI trial as an exampleTrial 2 × 2 contingency tableCalculation of the FIEventNo eventEventNo eventOpen surgical treatmentABOpen surgical treatmentA + FIB − FIEndovascular treatmentCDEndovascular treatmentCDFisher exact test: P < .05Fisher exact test: P ≥ .05BEST-CLI 2 × 2 contingency tableCalculation of the FI in the BEST-CLI trialMajor adverse limb event or deathNo eventMajor adverse limb event or deathNo eventBypass with single segment great saphenous vein302416Open surgical treatment372 (302 + 70)346 (416 − 70)Endovascular treatment408308Endovascular treatment408308Fisher exact test: P < .05 (P = .00000002)Fisher exact test: P ≥ .05 (P = .05002)BEST-CLI, Best Endovascular vs Best Surgical Therapy in Patients with Critical Limb Ischemia. Open table in a new tab Table IICharacteristics of the fragility index (FI) of primary outcomes of statistically significant RCTs evaluating open vs endovascular treatment in vascular surgeryCategoryOutcomes, nMedian FI [Q1, Q3]Disease state AAA103 [3, 3] CAS910 [3, 20] PAD415 [5, 46]Study design Index trial133 [3, 4] Follow-up study1014 [3, 21]Funding Commerciala920 [5, 25] Noncommercial143 [2, 6]Outcome Single163 [2, 9] Compositea721 [8, 38]Loss to follow-up <FI73 [3, 8] ≥FI1616 [6, 2, 21]AAA, Abdominal aortic aneurysm; CAS, carotid artery stenting; PAD, peripheral arterial disease.aP < .05. Open table in a new tab