External Validity Of Trial Results Research Articles

Impact of Broadening Trial Eligibility Criteria for Patients with Advanced Non-Small Cell Lung Cancer: Real-World Analysis of Select ASCO-Friends Recommendations.

Cancer clinical trials often accrue slowly or miss enrollment targets. Strict eligibility criteria are a major reason. Restrictive criteria also limit opportunities for patient participation while compromising external validity of trial results. We examined the impact of broadening select eligibility criteria on characteristics and number of patients eligible for trials, using recommendations of the American Society of Clinical Oncology (ASCO) and Friends of Cancer Research. A retrospective, observational analysis used electronic health record data from ASCO's CancerLinQ Discovery database. Study cohort included patients with advanced non-small cell lung cancer treated from 2011 to 2018. Patients were grouped by traditional criteria [no brain metastases, no other malignancies, and creatinine clearance (CrCl) ≥ 60 mL/minute] and broadened criteria (including brain metastases, other malignancies, and CrCl ≥ 30 mL/minute). The analysis cohort included 10,500 patients. Median age was 68 years, and 73% of patients were White. Most patients had stage IV disease (65%). A total of 5,005 patients (48%) would be excluded from trial participation using the traditional criteria. The broadened criteria, however, would allow 98% of patients (10,346) to be potential participants. Examination of patients included by traditional criteria (5,495) versus those added (4,851) by broadened criteria showed that the number of women, patients aged 75+ years, and those with stage IV cancer was significantly greater using broadened criteria. This analysis of real-world data demonstrated that broadening three common eligibility criteria has the potential to double the eligible patient population and include trial participants who are more representative of those encountered in practice.See related commentary by Giantonio, p. 2369.

A single center experience of Zilver PTX for femoro-popliteal lesions

Clinical trial data show overall favorable outcomes of paclitaxel-eluting stents for treatment of femoro-popliteal (FP) occlusive disease. However, external validity of trial results may be restricted to less complex FP lesions, and limited data on outcomes of paclitaxel-eluting stents in real world practice have been published. This is a retrospective analysis of data of all patients who received Zilver® PTX® for FP lesion from February 2013 to October 2014 at our center. The primary endpoint was primary patency, defined as peak systolic velocity ratio <2.0 by Doppler ultrasound, or angiographic diameter stenosis <50%, or freedom from clinically driven target lesion revascularization. Seventy-eight patients received Zilver® PTX® for FP lesions in the pre-specified time period. Of them, 63 had follow-up data and were included in this study. Mean patient age was 66.3±9.4years, and 57.1% of the patients were men. Participants had a high prevalence of diabetes (49.2%), hypertension (93.7%), hyperlipidemia (93.7%), previous coronary revascularization (52.4%), or previous peripheral arterial disease (77.8%). Critical limb ischemia was present in 25.4% of the patients, Trans-Atlantic Inter-Society Consensus (TASC) class C or D in 76.2%, in-stent restenosis (ISR) in 36.5%, and total occlusion in 69.8%. Mean lesion length was 218.9±128.3mm, mean number of stents was 2.02±1.0, and total stent length was 189.0±128.5mm. Mean follow-up was 270.4±190.3days. Primary patency rate at 1year was 66.7% by Kaplan-Meier survival curve. When compared with patients with primary patency at follow up, those with an adverse outcome had higher prevalence of TASC II class C or D lesions (100% vs. 68.8%, p=0.013), and were more likely to have ISR (66.7% vs. 27.1%, p=0.012), longer lesion (291.3±138.7 vs. 195.7±117.1, p=0.011), and incomplete coverage of the lesion (full coverage of lesions: 40% vs. 77.1%, p=0.011). Post marketing use of Zilver® PTX® for the treatment of FP lesions is associated with lower patency rates compared with clinical trial data. This may be related to the high prevalence of TASC II class C or D lesions and ISR in real world practice. Future studies should be more representative of contemporary clinical practice.

A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results.

Randomized controlled trials (RCTs) are conducted under idealized and rigorously controlled conditions that may compromise their external validity. A literature review was conducted of published English language articles that reported the findings of studies assessing external validity by a comparison of the patient sample included in RCTs reporting on pharmaceutical interventions with patients from everyday clinical practice. The review focused on publications in the fields of cardiology, mental health, and oncology. A range of databases were interrogated (MEDLINE; EMBASE; Science Citation Index; Cochrane Methodology Register). Double-abstract review and data extraction were performed as per protocol specifications. Out of 5,456 de-duplicated abstracts, 52 studies met the inclusion criteria (cardiology, n = 20; mental health, n = 17; oncology, n = 15). Studies either performed an analysis of the baseline characteristics (demographic, socioeconomic, and clinical parameters) of RCT-enrolled patients compared with a real-world population, or assessed the proportion of real-world patients who would have been eligible for RCT inclusion following the application of RCT inclusion/exclusion criteria. Many of the included studies concluded that RCT samples are highly selected and have a lower risk profile than real-world populations, with the frequent exclusion of elderly patients and patients with co-morbidities. Calculation of ineligibility rates in individual studies showed that a high proportion of the general disease population was often excluded from trials. The majority of studies (n = 37 [71.2 %]) explicitly concluded that RCT samples were not broadly representative of real-world patients and that this may limit the external validity of the RCT. Authors made a number of recommendations to improve external validity. Findings from this review indicate that there is a need to improve the external validity of RCTs such that physicians treating patients in real-world settings have the appropriate evidence on which to base their clinical decisions. This goal could be achieved by trial design modification to include a more representative patient sample and by supplementing RCT evidence with data generated from observational studies. In general, a thoughtful approach to clinical evidence generation is required in which the trade-offs between internal and external validity are considered in a holistic and balanced manner.Electronic supplementary materialThe online version of this article (doi:10.1186/s13063-015-1023-4) contains supplementary material, which is available to authorized users.

Evaluating the Generalisability of Trial Results: Introducing a Centre- and Trial-Level Generalisability Index.

Few randomised controlled trials (RCTs) recruit centres representatively, which may limit the external validity of trial results. The aim of this study was to propose a proof-of-concept method of assessing the generalisability of the clinical and cost-effectiveness findings of a given RCT. We developed a generalisability index (Gix), informed by centre-level characteristics, as a measure of centre and trial representativeness. The centre-level Gix quantifies how representative a centre is in relation to its jurisdiction, e.g. a country or health authority. The trial-level Gix quantifies how representative trial recruitment is in relation to clinical practice in the jurisdiction. Taking a real-world RCT as a case study and assuming trial-wide results to represent 'true jurisdiction values', we used simulation methods to recreate 5000 RCTs and investigate the relationship between trial representativeness, reflected by the standardised trial-Gix, and the deviation of simulated trial results from the 'true values'. The simulation study provides evidence that trial results (odds ratio for the primary outcome and incremental quality-adjusted life-years) were influenced by the representativeness of the sample of recruiting centres. Simulated RCTs with the closest results to the 'true values' were those whose recruitment closely mirrored the jurisdiction-wide context. Results appeared robust to six alternative specifications of the Gix. Our findings suggest that an unrepresentative selection of centres limits the external validity of trial results. The Gix may be a valuable tool to help facilitate rational selection of trial centres and ensure the generalisability of results at the jurisdiction level.

Commentary Regarding Uptake of Carotid Artery Stenting in England and Subsequent Vascular Admissions: An Appropriate Response to Emerging Evidence?

Several randomized controlled trials have shown the superiority of carotid endarterectomy (CEA) over carotid artery stenting (CAS) to prevent stroke or death in the short term in patients with symptomatic carotid artery stenosis. Since there is a gap between science and the real world, administrative data are frequently used to evaluate how evidence from trials is translated into daily clinical practice, to verify the external validity of trial results, and to identify opportunities to improve quality of care. In this issue of the journal, Lee et al. present an interesting study on the uptake of CAS in the UK between 2006 and 2012. 1 Whereas they found an absolute increase in CEA, the proportion of patients treated with CAS remained stable, being <5% of all carotid artery interventions. Patients treated with CAS were younger, lived more often in areas of higher deprivation, had more often amaurosis fugax as qualifying symptom, more co-morbidities, and had a shorter hospital stay. Although the low proportion of CAS does justice to the trial outcomes, the authors can only speculate about the reasons for its slow uptake. CAS was endorsed by National Institute for Health and Care Excellence guidelines during the study period, but maybe physicians were cautious to use CAS in symptomatic patients because of the increased risk of adverse events, or maybe reimbursement issues hampered the diffusion of CAS. The results are comparable to a recent report on variation in clinical practice among countries contributing to the VASCUNET registry. 2 Analysis of UK VASCUNET data showed that a steady proportion of 0.6% of patients had CAS between 2005 and 2010. Wide variation was noted in the utilization of CAS between countries, ranging from 0.1% of cases in Denmark to 17.4% in Italy. There was also wide variation in the indication for a carotid intervention with proportions of asymptomatic patients ranging between 0% (Denmark) to 69% (Italy). Although CAS is increasingly used to treat asymptomatic carotid artery stenosis, especially in the United States, analysis of administrative data indicates that the risk of postoperative stroke or death is higher than after CEA. 3 The study by Lee is an example of the limitations of using administrative data for audit of contemporary practice and to identify opportunities for quality improvement. 4 Although length of stay might be an interesting outcome for policy makers, information on procedure-related complications or prevention of strokes is much more important to clinicians and patients. The Hospital Episodes Statistics (HES) database clearly falls short to address these more meaningful issues for carotid artery disease. HES could not provide reliable data on the indication for the intervention, nor did it allow verification of the indication for readmission, or whether a re-intervention was on the ipsilateral side. In addition, finding associations between the use of CAS and patient characteristics did not allow drawing inference on the relative effectiveness of CAS and CEA in the study population. Despite all inherent limitations to databases, prospectively maintained registries such as VASCUNET or a national audit are better suited for quality control of interventions for carotid artery disease.

Comparison of treatment outcome in metastatic colorectal cancer patients included in a clinical trial versus daily practice in The Netherlands

The external validity of trial results is a matter of debate, and no strong evidence is available to support whether a trial may have a positive or a negative effect on the outcome of patients. Methods. We compared the results of stage IV colorectal cancer patients treated within a large Dutch phase III trial (CAIRO), in which standard chemotherapy and standard safety eligibility criteria were used, to patients treated outside the trial during the trial accrual period in a representative selection of 29 Dutch hospitals. Non-trial patients were identified by the Netherlands Cancer Registry (NCR), and were checked for the trial eligibility criteria. Results. The NCR registered 1946 stage IV colorectal cancer patients who received chemotherapy, of whom 394 patients were included in the CAIRO trial and 30 patients in other trials. Thus, the CAIRO trial participation rate was 20%. In the 29 hospitals, 162 patients received chemotherapy in the trial and 396 patients received chemotherapy outside the trial. Of the non-trial patients, 224 patients fulfilled the trial eligibility criteria. The overall survival of eligible non-trial patients was comparable to trial patients (HR 1.03, p = 0.70). However, non-eligible non-trial patients had a significantly worse outcome (HR 1.70, p < 0.01). Conclusion. These data provide evidence in a common tumor type that trial results have external validity, provided that standard eligibility criteria are being observed. Our finding of a worse outcome for patients not fulfilling these criteria strongly argues against the use of cancer treatments in other patient categories than included in the original trials in which these treatments were investigated.

Influence of clinical trial enrollment on the quality of care and outcomes for patients with non–ST-segment elevation acute coronary syndromes

Clinical trials provide evidence that is formulated into recommendations for practice guidelines, but it remains uncertain whether patients enrolled in trials are similar to those treated in routine practice and whether trial enrollment influences inhospital treatments and outcomes. Using data from the CRUSADE quality improvement initiative, we evaluated predictors of trial enrollment, treatment patterns, and clinical outcomes among high-risk patients with non-ST-segment elevation acute coronary syndromes (NSTE ACS) who were and were not enrolled in a clinical trial during hospitalization. Among 55,172 high-risk patients presenting with NSTE ACS at 443 US hospitals, 1397 (2.5%) patients were enrolled in a clinical trial during index hospitalization. Significant predictors of trial enrollment included male sex, lack of renal insufficiency, and absence of congestive heart failure on presentation. Acute (<24 hours) and discharge secondary prevention interventions recommended by the American College of Cardiology/American Heart Association guidelines for NSTE ACS were used more commonly in patients enrolled in clinical trials. Cardiac catheterization (84.5% vs 65.8%, P < .0001), percutaneous coronary intervention (48.2% vs 36.3%, P < .0001), and bypass surgery (19.1% vs 11.3%, P < .0001) were also performed more frequently in trial patients. The adjusted risk of inhospital mortality was similar in trial versus nontrial patients (odds ratio 0.86, 95% CI 0.60-1.24). Patients with NSTE ACS participating in clinical trials are more likely to receive beneficial therapies and interventions throughout hospitalization, but preferential recruitment of patients with lower-risk features may limit the external validity of trial results.

The Data Protection Act (1998): implications for health researchers.

This paper reports on the methods used in two studies to obtain access to subjects to comply with the common law duty of confidence laid out in the Data Protection Act (1998) and discusses the researchers' problems in interpreting the procedures. The amendments to the United Kingdom (UK) Data Protection Act (1998) are causing confusion within the health service and academic institutions. There is a need to balance patient confidentiality with the requirement to conduct vital, unbiased research in which health service professionals are not subject to ethical dilemmas. This paper examines the recruitment methods used in two studies in which the researchers' attempts to adhere to the requirements lengthened the study costs and may have produced less reliable results. The methodological difficulties in two studies are presented. In Study 1, the difficulties encountered when the Multicentre Research Ethics Committee refused permission for researchers to recruit patients directly to a multicentre randomized controlled trial are discussed. In Study 2, the method used to compile a sampling frame for a national questionnaire survey following the eight principles of the Act are described. Our experience has shown that health care professionals are increasingly required to recruit patients to intervention trials, and that researchers are not allowed access to the names of patients or other subjects to ask them for consent to participate in a study. The requirement for researchers to use "intermediaries" to obtain consent from and recruit subjects to studies increases the risk of selection bias, may expose the practitioner to ethical difficulties and may compromise the external validity of trial results. There is also a danger that research costs will soar when the Data Protection Act (1998) is fully realized. The Data Protection Act (1998) is currently being interpreted in a number of different ways. We conclude there is an urgent need for consensus within the health service and academic communities.

Analysis of randomized and nonrandomized patients in clinical trials using the comprehensive cohort follow-up study design

In clinical research, randomized trials are widely accepted as the definitive method of evaluating the efficacy of therapies. The random assignment of patients to their treatment ensures the internal validity of the comparison of new treatment with controls. An assessment of the external validity of trial results can best be achieved by comparing the study population to the population of patients who met the eligibility criteria but did not consent to randomization. A part of the data of the Coronary Artery Surgery Study (CASS), in which coronary artery bypass surgery is compared to conventional medical therapy in patients with coronary artery disease, is used to illustrate a strategy of multivariate analysis of randomized and nonrandomized patients which allows an investigation of both internal and external validity. The method used Cox's proportional hazards regression model with inclusion of covariates for randomization status and corresponding interactions in addition to the usual covariates for treatment and the important prognostic factors.