Introduction of ICU blood culture protocols and the effect on rates of contamination: A single-center, non-randomized interventional study

BackgroundThere is an increasing number of meta-analyses including data from non-randomized studies for therapeutic evaluation. We aimed to systematically assess the methods used in meta-analyses including non-randomized studies evaluating therapeutic interventions.MethodsFor this methodological review, we searched MEDLINE via PubMed, from January 1, 2013 to December 31, 2013 for meta-analyses including at least one non-randomized study evaluating therapeutic interventions. Etiological assessments and meta-analyses with no comparison group were excluded. Two reviewers independently assessed the general characteristics and key methodological components of the systematic review process and meta-analysis methods.ResultsOne hundred eighty eight meta-analyses were selected: 119 included both randomized controlled trials (RCTs) and non-randomized studies of interventions (NRSI) and 69 only NRSI. Half of the meta-analyses (n = 92, 49 %) evaluated non-pharmacological interventions. “Grey literature” was searched for 72 meta-analyses (38 %). An assessment of methodological quality or risk of bias was reported in 135 meta-analyses (72 %) but this assessment considered the risk of confounding bias in only 33 meta-analyses (18 %). In 130 meta-analyses (69 %), the design of each NRSI was not clearly specified. In 131 (70 %), whether crude or adjusted estimates of treatment effect for NRSI were combined was unclear or not reported. Heterogeneity across studies was assessed in 182 meta-analyses (97 %) and further explored in 157 (84 %). Reporting bias was assessed in 127 (68 %).ConclusionsSome key methodological components of the systematic review process—search for grey literature, description of the type of NRSI included, assessment of risk of confounding bias and reporting of whether crude or adjusted estimates were combined—are not adequately carried out or reported in meta-analyses including NRSI.

BackgroundsSyntheses of non-randomized studies of interventions (NRSIs) and randomized controlled trials (RCTs) are increasingly used in decision-making. This study aimed to summarize when NRSIs are included in evidence syntheses of RCTs, with a particular focus on the methodological issues associated with combining NRSIs and RCTs.MethodsWe searched PubMed to identify clinical systematic reviews published between 9 December 2017 and 9 December 2022, randomly sampling reviews in a 1:1 ratio of Core and non-Core clinical journals. We included systematic reviews with RCTs and NRSIs for the same clinical question. Clinical scenarios for considering the inclusion of NRSIs in eligible studies were classified. We extracted the methodological characteristics of the included studies, assessed the concordance of estimates between RCTs and NRSIs, calculated the ratio of the relative effect estimate from NRSIs to that from RCTs, and evaluated the impact on the estimates of pooled estimates when NRSIs are included.ResultsTwo hundred twenty systematic reviews were included in the analysis. The clinical scenarios for including NRSIs were grouped into four main justifications: adverse outcomes (n = 140, 63.6%), long-term outcomes (n = 36, 16.4%), the applicability of RCT results to broader populations (n = 11, 5.0%), and other (n = 33, 15.0%). When conducting a meta-analysis, none of these reviews assessed the compatibility of the different types of evidence prior, 203 (92.3%) combined estimates from RCTs and NRSIs in the same meta-analysis. Of the 203 studies, 169 (76.8%) used crude estimates of NRSIs, and 28 (13.8%) combined RCTs and multiple types of NRSIs. Seventy-seven studies (35.5%) showed “qualitative disagree” between estimates from RCTs and NRSIs, and 101 studies (46.5%) found “important difference”. The integration of NRSIs changed the qualitative direction of estimates from RCTs in 72 out of 200 studies (36.0%).ConclusionsSystematic reviews typically include NRSIs in the context of assessing adverse or long-term outcomes. The inclusion of NRSIs in a meta-analysis of RCTs has a substantial impact on effect estimates, but discrepancies between RCTs and NRSIs are often ignored. Our proposed recommendations will help researchers to consider carefully when and how to synthesis evidence from RCTs and NRSIs.

Very low-certainty evidence suggested that it is unclear whether gluten intake is associated with all-cause mortality. Our findings also indicate that low-certainty evidence may show little or no association between gluten intake and cardiovascular mortality and non-fatal myocardial infarction. Low-certainty evidence suggested that a lower compared with a higher gluten intake may be associated with a slightly increased risk to develop type 2 diabetes - a major cardiovascular risk factor. For other cardiovascular risk factors it is unclear whether there is a difference between a gluten-free and normal diet. Given the limited findings from this review predominantly based on observational studies, no recommendations for practice can be made.

Editors' Choice December 2022

To consider methods and related evidence for evaluating bias in non-randomised intervention studies. Systematic reviews and methodological papers were identified from a search of electronic databases; handsearches of key medical journals and contact with experts working in the field. New empirical studies were conducted using data from two large randomised clinical trials. Three systematic reviews and new empirical investigations were conducted. The reviews considered, in regard to non-randomised studies, (1) the existing evidence of bias, (2) the content of quality assessment tools, (3) the ways that study quality has been assessed and addressed. (4) The empirical investigations were conducted generating non-randomised studies from two large, multicentre randomised controlled trials (RCTs) and selectively resampling trial participants according to allocated treatment, centre and period. In the systematic reviews, eight studies compared results of randomised and non-randomised studies across multiple interventions using meta-epidemiological techniques. A total of 194 tools were identified that could be or had been used to assess non-randomised studies. Sixty tools covered at least five of six pre-specified internal validity domains. Fourteen tools covered three of four core items of particular importance for non-randomised studies. Six tools were thought suitable for use in systematic reviews. Of 511 systematic reviews that included non-randomised studies, only 169 (33%) assessed study quality. Sixty-nine reviews investigated the impact of quality on study results in a quantitative manner. The new empirical studies estimated the bias associated with non-random allocation and found that the bias could lead to consistent over- or underestimations of treatment effects, also the bias increased variation in results for both historical and concurrent controls, owing to haphazard differences in case-mix between groups. The biases were large enough to lead studies falsely to conclude significant findings of benefit or harm. Four strategies for case-mix adjustment were evaluated: none adequately adjusted for bias in historically and concurrently controlled studies. Logistic regression on average increased bias. Propensity score methods performed better, but were not satisfactory in most situations. Detailed investigation revealed that adequate adjustment can only be achieved in the unrealistic situation when selection depends on a single factor. Results of non-randomised studies sometimes, but not always, differ from results of randomised studies of the same intervention. Non-randomised studies may still give seriously misleading results when treated and control groups appear similar in key prognostic factors. Standard methods of case-mix adjustment do not guarantee removal of bias. Residual confounding may be high even when good prognostic data are available, and in some situations adjusted results may appear more biased than unadjusted results. Although many quality assessment tools exist and have been used for appraising non-randomised studies, most omit key quality domains. Healthcare policies based upon non-randomised studies or systematic reviews of non-randomised studies may need re-evaluation if the uncertainty in the true evidence base was not fully appreciated when policies were made. The inability of case-mix adjustment methods to compensate for selection bias and our inability to identify non-randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical. Recommendations for further research include: applying the resampling methodology in other clinical areas to ascertain whether the biases described are typical; developing or refining existing quality assessment tools for non-randomised studies; investigating how quality assessments of non-randomised studies can be incorporated into reviews and the implications of individual quality features for interpretation of a review's results; examination of the reasons for the apparent failure of case-mix adjustment methods; and further evaluation of the role of the propensity score.

Case studies to explore the optimal use of randomized and nonrandomized studies in evidence syntheses that use GRADE

Including non-randomized studies of interventions in meta-analyses of randomized controlled trials changed the estimates in more than a third of the studies: evidence from an empirical analysis.

IntroductionAlthough interest in including non-randomised studies of interventions (NRSIs) in meta-analysis of randomised controlled trials (RCTs) is growing, estimates of effectiveness obtained from NRSIs are vulnerable to greater bias than...

For preventing relapses in relapsing MS, rituximab as 'first choice' and as 'switching' may compare favourably with a wide range of approved DMTs. A protective effect of rituximab against disability worsening is uncertain. There is limited information to determine the effect of rituximab for progressive MS. The evidence is uncertain about the effect of rituximab on SAEs. They are relatively rare in people with MS, thus difficult to study, and they were not well reported in studies. There is an increased risk of common infections with rituximab, but absolute risk is small. Rituximab is widely used as off-label treatment in people with MS; however, randomised evidence is weak. In the absence of randomised evidence, remaining uncertainties on beneficial and adverse effects of rituximab for MS might be clarified by making real-world data available.

Persons infected with SARS-CoV-2 can develop long-term symptoms known as postCOVID-19 condition (PCC; symptoms ≥3 months after infection) or long COVID (LC; symptoms ≥1 month after infection). Vaccination against COVID-19 might prevent PCC/LC, but the extent of protection is unclear. The aim of this systematic review was to evaluate the vaccine efficacy/effectiveness (VE) of COVID-19 vaccines given prior to SARS-CoV-2 infection in preventing PCC or LC. Studies were identified in Embase, MEDLINE, PreView, COVID-19 L.OVE repository, and Cochrane Library up to August 1, 2024. Randomized controlled trials and nonrandomized studies of interventions (NRSI) that investigated immunization with a COVID-19 vaccine before SARS-CoV-2 infection were eligible, irrespective of participant age and sex. Risk of bias was assessed using the ''Risk Of Bias In Nonrandomized Studies-of Interventions'' tool. Primary outcome was PCC, secondary outcomes were LC, time until reconvalescence, limitations in everyday activity, and quality of life. Meta-analyses were primarily conducted using the random-effects model. A total of 6423 records were screened, and 65 NRSI reporting adjusted estimates were included, comprising >5.7 mio. VE for ≥1 vaccine dose against PCC was 41.0% (95% CI 27.8%; 51.7%; 22 NRSI, certainty of evidence: low). VE after 1, 2, or 3 doses versus unvaccinated was 19.1% (-119.4%; 70.2%, 3 NRSI), 43.2% (4.5%; 66.2%; 4 NRSI), and 70.0% (30.0%; 87.0%; 1 NRSI), respectively. In <18 years old, VE against PCC was 26% for ≥1 dose (-4%; 48%, 1 NRSI) and in >60 years old 41% (17%; 59%, 1 NRSI). VE after pre-Omicron-SARS-CoV-2 infection was 32.1% (-54.3%; 70.1%, 3 NRSI) and 20.9% (-10.1%; 43.3%, 2 NRSI) after Omicron infection. Sensitivity analyses indicated no influence of risk of bias and effect measure. COVID-19 vaccines may be moderately effective in preventing PCC/LC. VE may increase with the number of vaccine doses administered.

ABSTRACTWe performed an update (last search: 24 July 2023) of a systematic review on relative efficacy/effectiveness (rVE) and safety of newer/enhanced seasonal influenza vaccines in comparison with standard influenza vaccine or in head‐to‐head comparison. Eligible studies investigated adults aged ≥ 18 years, analysed the MF59‐adjuvanted or high‐dose or cell‐based or recombinant or mRNA‐based influenza vaccine and reported rVE or safety in randomised controlled trials (RCT) or non‐randomised studies of interventions (NRSI). Of 1561 new entries identified, 17 studies were included. Together with 42 studies identified in the previous primary review they added up to 59 studies, all comparing newer/enhanced with standard seasonal influenza vaccines. Relative VE against laboratory‐confirmed influenza was −30% (95%CI: −146% to 31%) to 88% (51%–100%; 7 NRSI) for the MF59‐adjuvanted vaccine (low certainty of evidence, CoE); 24.2% (9.7%–36.5%; 1 RCT) and −9% (−158% to 54%) to 19% (−27% to 48%; 1 NRSI) for the high‐dose vaccine (moderate CoE); −5.8% (−36.1% to 17.7%) to 21.4% (−7.3% to 42.4%; 2 NRSI) for the cell‐based vaccine (low CoE); 30% (10%–47%; 1 RCT) and 3% (−31% to 28%) to 19% (−27% to 48%; 1 NRSI) for the recombinant vaccine (moderate CoE), respectively. Relative VE against laboratory‐confirmed influenza‐related hospitalisation was 59.2% (14.6%–80.5%; 1 NRSI) for the MF59‐adjuvanted (moderate CoE); 27% (−1 to 48%; 1 NRSI) for the high‐dose (low CoE); 8.5% (−75.9% to 52.3%; 1 NRSI) for the cell‐based (low CoE); −7.3% (−52.1% to 24.4%) to 16.3% (−8.7% to 35.5%; 1 RCT) for the recombinant vaccine. No increased risk of serious adverse events was detected for any vaccine (12 RCT, 7 NRSI; low CoE). While all have a favourable safety profile, evidence on rVE of newer/enhanced vaccines is still limited, warranting further studies.

GRADE Guidance 44: strategies to enhance the utilization of randomized and non-randomized studies in evidence syntheses of health interventions.

Introduction: The Risk Of Bias in Non-randomized Studies – of Interventions (ROBINS-I) tool evaluates internal validity (risk of bias) in non-randomized studies of interventions in comparison to an ideal (hypothetical) randomized trial. The use of ROBINS-I in studies dealing with exposures or interventions in environmental health has not yet been explored. This study evaluated the usability and applicability of ROBINS-I in studies of environmental health (EH) exposure. Methods: Three researchers in sequential rounds applied ROBINS-I to three systematic reviews of EH exposures: bisphenol-A and obesity; perfluorooctanoic acid and birth weight; and polybrominated diphenyl ethers and thyroid function. We began by providing instructions for application of ROBINS-I to EH studies, including possible confounders and co-exposures specific to the exposures considered in the three reviews. For the first two rounds of testing, two reviewers independently applied ROBINS-I and provided feedback on usability of the tool. Barriers and facilitators to the appropriateness of ROBINS-I for environmental health were identified and modifications made to the tool, as necessary. For the third round of testing, three reviewers independently applied the tool and came to consensus on item-level and overall study risk of bias. Results: Suggested modifications ranged from syntax and wording to conceptual changes to the tool. The term "intervention" was replaced with "exposure" throughout the document. Additional instructions were provided to address assessment of cross-sectional studies. Fields to collect information on measurement of exposures and outcomes of interest was added to the project protocol. Additional granularity was added to the measurement of interventions/exposure domain. Conclusion: Modifications made to the risk of bias tool to tailor it to studies of EH exposure increased understanding and application of the tool, as well as consistency in responses.

Introduction: Randomized controlled trials (RCTs) are the gold standard to evaluate the efficacy of interventions (e.g., drugs and vaccines), yet the sample size of RCTs is often limited for safety assessment. Non-randomized studies of interventions (NRSIs) had been proposed as an important alternative source for safety assessment. In this study, we aimed to investigate whether there is any difference between RCTs and NRSIs in the evaluation of adverse events.Methods: We used the dataset of systematic reviews with at least one meta-analysis including both RCTs and NRSIs and collected the 2 × 2 table information (i.e., numbers of cases and sample sizes in intervention and control groups) of each study in the meta-analysis. We matched RCTs and NRSIs by their sample sizes (ratio: 0.85/1 to 1/0.85) within a meta-analysis. We estimated the ratio of the odds ratios (RORs) of an NRSI against an RCT in each pair and used the inverse variance as the weight to combine the natural logarithm of ROR (lnROR).Results: We included systematic reviews with 178 meta analyses, from which we confirmed 119 pairs of RCTs and NRSIs. The pooled ROR of NRSIs compared to that of RCTs was estimated to be 0.96 (95% confidence interval: 0.87 and 1.07). Similar results were obtained with different sample size subgroups and treatment subgroups. With the increase in sample size, the difference in ROR between RCTs and NRSIs decreased, although not significantly.Discussion: There was no substantial difference in the effects between RCTs and NRSIs in safety assessment when they have similar sample sizes. Evidence from NRSIs might be considered a supplement to RCTs for safety assessment.

To provide a framework for conducting rigorous nonrandomized studies of interventions in fertility treatment research, addressing their role as complements to randomized controlled trials (RCTs) in evaluating treatment outcomes. Multidisciplinary expert consensus on best practices for nonrandomized studies of interventions, informed by advancements in novel methodologies, including causal inference. Patients undergoing assisted reproductive technologies (ARTs) procedures, such as ovarian stimulation, laboratory techniques, and embryo transfer. None. Guidance on methodological rigor, transparency, and relevance in nonrandomized studies of interventions study design and analysis. Randomized controlled trials are the gold standard for determining the efficacy and safety of fertility treatment/ART interventions but can face logistical, practical, and sometimes ethical challenges. Nonrandomized studies of interventions, when conducted with high methodological rigor, complement RCTs by offering insights into real-world clinical practices and diverse patient populations. Key limitations of nonrandomized studies of interventions include susceptibility to confounding and selection bias, which require meticulous study design and advanced analytical techniques to address. Recent innovations, such as target trial emulation studies, have enhanced the validity of causal inferences based on nonrandomized studies of interventions. This article outlines 7 recommendations to improve the credibility of nonrandomized studies of interventions in ART research: clearly define research questions with precise estimands; design nonrandomized studies of interventions as emulated trials; use directed acyclic graphs to clarify causal assumptions; preregister study protocols; separate data analysis from study planning; incorporate negative controls to detect biases; and use appropriate analytical methods to account for confounding and selection bias. Integrating evidence from RCTs and well-conducted nonrandomized studies of interventions enhances clinical decision making in fertility treatment research. By adhering to these recommendations, researchers can improve the quality, transparency, and impact of nonrandomized studies of interventions, ultimately fostering robust, evidence-based clinical practices in fertility treatment/ART.