Meta-research Study Research Articles

Frequency of multiple changes to prespecified primary outcomes of clinical trials completed between 2009 and 2017 in German university medical centers: A meta-research study

Background Clinical trial registries allow assessment of deviations of published trials from their protocol, which may indicate a considerable risk of bias. However, since entries in many registries can be updated at any time, deviations may go unnoticed. We aimed to assess the frequency of changes to primary outcomes in different historical versions of registry entries, and how often they would go unnoticed if only deviations between published trial reports and the most recent registry entry are assessed. Methods and findings We analyzed the complete history of changes of registry entries in all 1746 randomized controlled trials completed at German university medical centers between 2009 and 2017, with published results up to 2022, that were registered in ClinicalTrials.gov or the German WHO primary registry (German Clinical Trials Register; DRKS). Data were retrieved on 24 January 2022. We assessed deviations between registry entries and publications in a random subsample of 292 trials. We determined changes of primary outcomes (1) between different versions of registry entries at key trial milestones, (2) between the latest registry entry version and the results publication, and (3) changes that occurred after trial start with no change between latest registry entry version and publication (so that assessing the full history of changes is required for detection of changes). We categorized changes as major if primary outcomes were added, dropped, changed to secondary outcomes, or secondary outcomes were turned into primary outcomes. We also assessed (4) the proportion of publications transparently reporting changes and (5) characteristics associated with changes. Of all 1746 trials, 23% (n = 393) had a primary outcome change between trial start and latest registry entry version, with 8% (n = 142) being major changes, that is, primary outcomes were added, dropped, changed to secondary outcomes, or secondary outcomes were turned into primary outcomes. Primary outcomes in publications were different from the latest registry entry version in 41% of trials (120 of the 292 sampled trials; 95% confidence interval (CI) [35%, 47%]), with major changes in 18% (54 of 292; 95% CI [14%, 23%]). Overall, 55% of trials (161 of 292; 95% CI [49%, 61%]) had primary outcome changes at any timepoint over the course of a trial, with 23% of trials (67 of 292; 95% CI [18%, 28%]) having major changes. Changes only within registry records, with no apparent discrepancy between latest registry entry version and publication, were observed in 14% of trials (41 of 292; 95% CI [10%, 19%]), with 4% (13 of 292; 95% CI [2%, 7%]) being major changes. One percent of trials with a change reported this in their publication (2 of 161 trials; 95% CI [0%, 4%]). An exploratory logistic regression analysis indicated that trials were less likely to have a discrepant registry entry if they were registered more recently (odds ratio (OR) 0.74; 95% CI [0.69, 0.80]; p<0.001), were not registered on ClinicalTrials.gov (OR 0.41; 95% CI [0.23, 0.70]; p = 0.002), or were not industry-sponsored (OR 0.29; 95% CI [0.21, 0.41]; p<0.001). Key limitations include some degree of subjectivity in the categorization of outcome changes and inclusion of a single geographic region. Conclusions In this study, we observed that changes to primary outcomes occur in 55% of trials, with 23% trials having major changes. They are rarely transparently reported in the results publication and often not visible in the latest registry entry version. More transparency is needed, supported by deeper analysis of registry entries to make these changes more easily recognizable. Protocol registration: Open Science Framework (https://osf.io/t3qva; amendment in https://osf.io/qtd2b).

Reasons for missing evidence in rehabilitation meta-analyses: a cross-sectional meta-research study

BackgroundSystematic reviews of randomized controlled trials are the best evidence for informing on intervention effectiveness. Their results, however, can be biased due to omitted evidence in the quantitative analyses. We aimed to assess the proportion of randomized controlled trials omitted from meta-analyses in the rehabilitation field and explore related reasons.MethodsThis is a cross-sectional meta-research study. For each systematic review included in a published selected sample in the rehabilitation field, we identified an index meta-analysis on the primary outcome and the main comparison. We then looked at all the studies considered eligible for the chosen comparison in the systematic review and identified those trials that have been omitted (i.e., not included) from each index meta-analysis. Reasons for omission were collected based on an eight-reason classification. We used descriptive statistics to describe the proportion of omitted trials overall and according to each reason.ResultsStarting from a cohort of 827 systematic reviews, 131 index meta-analyses comprising a total of 1761 eligible trials were selected. Only 16 index meta-analyses included all eligible studies while 15 omitted studies without providing references. From the remaining 100 index meta-analyses, 717 trials (40,7%) were omitted overall. Specific reasons for omission were: "unable to distinguish between selective reporting and inadequate planning" (39,3%, N = 282), "inadequate planning" (17%, N = 122), "justified to be not included" (15,1%, N = 108), "incomplete reporting" (8,4%, N = 60), "selective reporting" (3,3%, N = 24) and other situations (e.g., outcome present but no motivation for omission) (5,2%, N = 37). The 11,7% (N = 84) of omitted trials were not assessed due to non-English language or full text not available.ConclusionsAlmost half of the eligible trials were omitted from their index meta-analyses. Better reporting, protocol registration, definition and adoption of core outcome sets are needed to prevent omission of evidence in systematic reviews.

A commentary on 'Meta-research studies in surgery: a field that should be encouraged to assess and improve the quality of surgical evidence'.

A commentary on 'Meta-research studies in surgery: a field that should be encouraged to assess and improve the quality of surgical evidence'.

1715P Analysis of the adequacy of control arms in oncology randomized clinical trials published between 2017 and 2021: A meta-research study

1715P Analysis of the adequacy of control arms in oncology randomized clinical trials published between 2017 and 2021: A meta-research study

AMSTAR 2 is only partially applicable to systematic reviews of non-intervention studies: a meta-research study

ObjectivesA measurement tool to assess systematic reviews 2 (AMSTAR 2) was originally developed for systematic reviews (SRs) of health-care interventions. The aim of this study was to assess the applicability of AMSTAR 2 to SRs of non-intervention studies. Study design and settingThis was a meta-research study. We used 20 SRs for each of the following four types of SRs: Diagnostic Test Accuracy reviews, Etiology and/or Risk reviews, Prevalence and/or Incidence reviews, and Prognostic reviews (80 in total). Three authors applied AMSTAR 2 independently to each included SRs. Then, the authors assessed the applicability of each item to that SR type and any SR type. ResultsResearchers unanimously indicated that 7 of 16 AMSTAR 2 items were applicable for all four specific SR types and any SR type (items 2, 5, 6, 7, 10, 14 and 16), but 8 of 16 items for any SR type. These items could cover generic SR methods that do not depend on a specific SR type. ConclusionAMSTAR 2 is only partially applicable for non-intervention SRs. There is a need to adapt/extend AMSTAR 2 for SRs of non-intervention studies. Our study can help to further define generic methodological aspects shared across SR types and methodological expectations for non-intervention SRs.

Clinical trial registration has become more prevalent in physical therapy but it is still inadequate: A meta-research study

BackgroundA study using data from 2009 showed low prevalence and inadequate trial registration in physiotherapy. In 2013, a joint editorial recommended prospective registration in physiotherapy journals. Ten years later it is unclear whether the joint editorial achieved its intended benefit. ObjectivesTo investigate the proportion of randomized trials adequately registered and the extent of selective reporting of outcomes in trials of physiotherapy interventions published in 2019 and to compare these data with equivalent published data from 2009. DesignMeta-research study. MethodA random sample of 200 trials published in 2019 was used. Evidence of registration was sought on trial registers and by contacting authors. Data from the article was compared with data from the trial registration. Data from this sample of trial published in 2019 were compared with equivalent published data from 2009. ResultsIn 2019, the proportion of trials that were registered was 63% versus 34% in 2009 (absolute difference 29%). In 2019, 18% of the trials were prospectively registered compared to 6% in 2009 (absolute difference 12%). Unambiguous primary outcomes (i.e., method and timepoints of measurement clearly defined in the trial registry entry) were registered for 30% in 2019. Registration was adequate (i.e., prospective with unambiguous primary outcomes) for 8%, compared with 3% in 2009 (absolute difference 5%). Selective outcome reporting occurred in 73% of the trials in which it was assessable; in 2009 this proportion was 47% (absolute difference 26%). ConclusionsRegistration of randomized trials in physiotherapy increased in the past decade, but it is still inadequate. More effort is still required to implement and enforce adequate registration.

Metaresearch study highlights the gender gap in randomized controlled trials in dentistry

Background and ObjectivesThis study aimed to assess the gender gap in randomized controlled trials (RCTs) in dentistry in terms of authorship, collaborations, metrics, funding and reporting of good research practice and transparecy. MethodsThe search was performed in PubMed for RCTs restricted to English texts in the dental field, indexed from 12/31/2016 to 12/31/2021. Two reviewers screened the studies in line with the eligibility criteria. A total of 844 articles were included. The name and gender of authors, citation metrics, funding, reporting of characteristics of good research practice and transparency were extracted. We considered "collaboration between authors" when the corresponding author was different from the first author. ResultsThe proportion of women as first authors was 46.56% and 40.12% for corresponding authors. The analysis showed that when a woman is the corresponding author, the probability of the first author also being a woman is 57% higher compared to the first author being a man. For “protocol registration” and “data sharing,” the prevalence of reporting was higher when women were first authors. ConclusionA gender gap in dentistry RCTs was identified and related to the participation of women as first and corresponding authors and the collaboration between authors.

Transparency of COVID-19-related research: A meta-research study.

We aimed to assess the adherence to five transparency practices (data availability, code availability, protocol registration and conflicts of interest (COI), and funding disclosures) from open access Coronavirus disease 2019 (COVID-19) related articles. We searched and exported all open access COVID-19-related articles from PubMed-indexed journals in the Europe PubMed Central database published from January 2020 to June 9, 2022. With a validated and automated tool, we detected transparent practices of three paper types: research articles, randomized controlled trials (RCTs), and reviews. Basic journal- and article-related information were retrieved from the database. We used R for the descriptive analyses. The total number of articles was 258,678, of which we were able to retrieve full texts of 186,157 (72%) articles from the database Over half of the papers (55.7%, n = 103,732) were research articles, 10.9% (n = 20,229) were review articles, and less than one percent (n = 1,202) were RCTs. Approximately nine-tenths of articles (in all three paper types) had a statement to disclose COI. Funding disclosure (83.9%, confidence interval (CI): 81.7-85.8 95%) and protocol registration (53.5%, 95% CI: 50.7-56.3) were more frequent in RCTs than in reviews or research articles. Reviews shared data (2.5%, 95% CI: 2.3-2.8) and code (0.4%, 95% CI: 0.4-0.5) less frequently than RCTs or research articles. Articles published in 2022 had the highest adherence to all five transparency practices. Most of the reviews (62%) and research articles (58%) adhered to two transparency practices, whereas almost half of the RCTs (47%) adhered to three practices. There were journal- and publisher-related differences in all five practices, and articles that did not adhere to transparency practices were more likely published in lowest impact journals and were less likely cited. While most articles were freely available and had a COI disclosure, adherence to other transparent practices was far from acceptable. A much stronger commitment to open science practices, particularly to protocol registration, data and code sharing, is needed from all stakeholders.

Exploring the use of preprints in dentistry

ObjectiveThis study aims to assess the use, impact, and dissemination of preprints in dentistry. MethodsThis is a meta-research study with a cross-sectional design. We included preprints published in dentistry, regardless of the year of publication. Searches were performed in the medRxiv.org and Preprints.org platforms and restricted to English. One researcher extracted the data, and another researcher verified data consistency. The following data were extracted: year of publication, country of the corresponding author, number of abstract and full-text views and downloads, Altmetric attention score, whether the preprint was mentioned in other servers such as Twitter and Publons, number of mentions in other servers, number of citations in the Dimensions database, and whether the preprint had already been published in a peer-reviewed journal. If already published, we extracted the journal's impact factor (JCR 2021) and the number of citations in the Dimensions database. We conducted a descriptive analysis of the extracted characteristics and explored relationships between metrics using the Spearman correlation. ResultsWe identified 276 preprints. Most of the studies were published between 2020 and 2022 (n = 229), especially those from ten countries. The most-cited preprint and published article are the same study. Only the correlation between the number of preprint citations and peer-reviewed article citations in the Dimensions database showed a large positive association (Spearman's rho = 0.5809). ConclusionPreprints gained popularity over the last several years due to the COVID-19 pandemic and reached a larger audience, especially on platforms such as Twitter. Clinical significancePreprint publishing allows faster dissemination of science for the benefit of society.

Assessment of redundancy, methodological and reporting quality, and potential discrepancies of results of systematic reviews of early mobilisation of critically ill adults: a meta-research protocol

IntroductionSeveral systematic reviews (SRs) have been conducted to determine the effectiveness of early mobilisation in critically ill adults with heterogeneous methodology and results. Redundancy in conducting SRs, unclear justification when...

Prevalence and predictors of data and code sharing in the medical and health sciences: systematic review with meta-analysis of individual participant data

ObjectivesTo synthesise research investigating data and code sharing in medicine and health to establish an accurate representation of the prevalence of sharing, how this frequency has changed over time, and...

Mapping Strategies to Assess and Increase the Validity of Published Disproportionality Signals: A Meta-Research Study.

Disproportionality analysis is traditionally used in spontaneous reporting systems to generate working hypotheses about potential adverse drug reactions: the so-called disproportionality signals. We aim to map the methods used by researchers to assess and increase the validity of their published disproportionality signals. From a systematic literature search of published disproportionality analyses up until 1 January 2020, we randomly selected and analyzed 100 studies. We considered five domains: (1) rationale for the study, (2) design of disproportionality analyses, (3) case-by-case assessment, (4) use of complementary data sources, and (5) contextualization of the results within existing evidence. Among the articles, multiple strategies were adopted to assess and enhance the results validity. The rationale, in 95 articles, was explicitly referred to the accrued evidence, mostly observational data (n=46) and regulatory documents (n=45). A statistical adjustment was performed in 34 studies, and specific strategies to correct for biases were implemented in 33 studies. A case-by-case assessment was complementarily performed in 35 studies, most often by investigating temporal plausibility (n=26). Complementary data sources were used in 25 articles. In 78 articles, results were contextualized using accrued evidence from the literature and regulatory documents, the most important sources being observational (n=45), other disproportionalities (n=37), and case reports (n=36). This meta-research study highlighted the heterogeneity in methods and strategies used by researchers to assess the validity of disproportionality signals. Mapping these strategies is a first step towards testing their utility in different scenarios and developing guidelines for designing future disproportionality analysis.

Pilot trials may improve the quality of full-scale trials: a meta-research study

ObjectivesEvidence on the value of pilot trials for subsequent trial's quality is scarce. This study aims to determine if a pilot trial improves the quality of the full-scale trial. Study Design and SettingWe searched PubMed for pilot trials and their subsequent full-scale trials. The meta-analysis of the full-scale trials was used to identify other full-scale trials on the same research topic but without a pilot trial. Markers of trial quality included publication outcomes and Cochrane Risk of Bias (RoB) assessment. ResultsFifty-eight full-scale trials with a pilot trial and 151 full-scale trials without were identified from 47 meta-analyses. Trials with a pilot trial were published 0.9 years sooner (mean ± standard deviation: 1.7 ± 1.0 vs. 2.6 ± 2.0, P = 0.005) and in peer-reviewed journals with higher impact factors (60.9 ± 75.0 vs. 24.8 ± 50.3, P < 0.001). A pilot trial's presence was associated with lower risk of bias in full-scale trial random sequence generation (OR [95% CI]: 4.05 [1.27–12.91]), allocation concealment (2.89 [1.07–7.83]), and participants/researchers masking (4.31 [1.37–13.50]), but not outcome assessment masking (1.03 [0.49–2.18]), incomplete outcome data (1.27 [0.47–3.42]), and selective reporting (1.23 [0.44–3.46]). ConclusionConducting a pilot trial may enhance the quality of the subsequent full-scale trial.

Methods and results of studies on reporting guideline adherence are poorly reported: a meta-research study

ObjectivesWe investigated recent meta-research studies on adherence to four reporting guidelines to determine the proportion that provided (1) an explanation for how adherence to guideline items was rated and (2) results from all included individual studies. We examined conclusions of each meta-research study to evaluate possible repetitive and similar findings. Study Design and SettingA cross-sectional meta-research study. MEDLINE (Ovid) was searched on July 5, 2022 for studies that used any version of the Consolidated Standards of Reporting Trials, Preferred Reporting Items for Systematic Reviews and Meta-Analyses, Standards for the Reporting of Diagnostic Accuracy Studies, or Strengthening the Reporting of Observational Studies in Epidemiology reporting guidelines or their extensions to evaluate reporting. ResultsOf 148 included meta-research studies published between August 2020 and June 2022, 14 (10%, 95% confidence interval [CI] 6%–15%) provided a fully replicable explanation of how they coded the adherence ratings and 49 (33%, 95% CI 26%–41%) completely reported individual study results. Of 90 studies that classified reporting as adequate or inadequate in the study abstract, six (7%, 95% CI 3%–14%) concluded that reporting was adequate, but none of those six studies provided information on how items were coded or provided item-level results for included studies. ConclusionAlmost all included meta-research studies found that reporting in health research is suboptimal. However, few of these reported enough information for verification or replication.

Meta-research studies in surgery: a field that should be encouraged to assess and improve the quality of surgical evidence.

Lozada-Martinez, Ivan Davida; Ealo-Cardona, Cristina Isabelab; Marrugo-Ortiz, Ariel Camilob; Picón-Jaimes, Yelson Alejandroc; Cabrera-Vargas, Luis Feliped; Narvaez-Rojas, Alexis Rafael MDe,f, Author Information

Analysis of the adequacy of control arms in oncology randomised clinical trials published between 2017 and 2021: a meta-research study

IntroductionRandomised controlled trials (RCTs) are usually considered the highest level of evidence for clinical practice. Patients assigned to control arm in RCTs should always receive the best available treatments to protect participants while also allowing for proper interpretation and applicability of study results. Here we analysed RCTs published in oncology between 2017 and 2021 to describe the frequency of suboptimal control arms. MethodsWe identified phase III studies testing active treatments in patients with solid tumours among 11 major oncology journals. Each control arm was analysed, and the standard of care was determined according to international guidelines and scientific evidence at accrual beginning and until accrual completion. We identified studies with suboptimal control arm from the beginning (type 1) and studies with an initially optimal control arm which became outdated during the accrual period (type 2). ResultsThis analysis included 387 studies. Forty-three (11.1%) control arms were judged as suboptimal: 24 (6.2%) type 1 and 19 (4.9%) type 2. These rates were higher in industry-sponsored compared to academic trials: 9.3% versus 1.9% for type 1 (p = 0.003); 7.9% versus 0.6% for type 2 (p = 0.001). Rates of suboptimal control arms were higher in studies with positive results: 8.1% versus 4.0% for type 1 (p = 0.09); 7.6% versus 1.7% for type 2 (p = 0.007). ConclusionsMany trials have suboptimal control arms, even in journals with high-impact factors, leading to suboptimal treatment of control patients and biased evaluation of trial results.

Prolific non-research authors in high impact scientific journals: meta-research study.

Journalistic papers published in high impact scientificjournals can be very influential, especially in hot fields. This meta-research analysis aimed to evaluate the publication profiles, impact, and disclosures of conflicts of interest of non-research authors who had published > 200 Scopus-indexed papers in Nature, Science, PNAS, Cell, BMJ, Lancet, JAMA or New England Journal of Medicine. 154 prolific authors were identified, 148 of whom had published 67,825 papers in their main affiliated journal in a non-researcher capacity. Nature, Science, and BMJ have the lion's share of such authors. Scopus characterized 35% of the journalistic publications as full articles and another 11% as short surveys. 264 papers had received more than 100 citations. 40/41 most-cited papers in 2020-2022 were on hot COVID-19 topics. Of 25 massively prolific authors with > 700 publications in one of these journals, many were highly-cited (median citations 2273), almost all had published little or nothing in the Scopus-indexed literature other than in their main affiliated journal, and their influential writing covered diverse hot topics over the years. Of the 25, only 3 had a PhD degree in any subject matter, and 7 had a Master's degree in journalism. Only the BMJ offered conflicts of interest disclosures for prolific science writers in its website, but even then only 2 of the 25 massively prolific authors disclosed potential conflicts with some specificity. The practice of assigning so much power to non-researchers in shaping scientific discourse should be further debated and disclosures of potential conflicts of interest should be emphasized.

NEgatiVE results in Radiomics research (NEVER): A meta-research study of publication bias in leading radiology journals

PurposeThe purpose of this study was to conduct a meta-research of radiomics-related articles for the publication of negative results, with a focus on the leading clinical radiology journals due to their purportedly high editorial standards. MethodsA literature search was performed in PubMed to identify original research studies on radiomics (last search date: August 16th, 2022). The search was restricted to studies published in Q1 clinical radiology journals indexed by Scopus and Web of Science. Following an a priori power analysis based on our null hypothesis, a random sampling of the published literature was conducted. Besides the six baseline study characteristics, a total of three items about publication bias were evaluated. Agreement between raters was analyzed. Disagreements were resolved through consensus. Statistical synthesis of the qualitative evaluations was presented. ResultsFollowing a priori power analysis, we included a random sample of 149 publications in this study. Most of the publications were retrospective (95%; 142/149), based on private data (91%; 136/149), centered on a single institution (75%; 111/149), and lacked external validation (81%; 121/149). Slightly fewer than half (44%; 66/149) made no comparison to non-radiomic approaches. Overall, only one study (1%; 1/149) reported negative results for radiomics, yielding a statistically significant binomial test (p < 0.0001). ConclusionThe top clinical radiology journals almost never publish negative results, having a strong bias toward publishing positive results. Almost half of the publications did not even compare their approach with a non-radiomic method.

Co-authorship group significantly impacts reported arterial blood pressure variations in rodents exposed to intermittent hypoxia: a meta-research study

Co-authorship group significantly impacts reported arterial blood pressure variations in rodents exposed to intermittent hypoxia: a meta-research study

Completeness of Reporting in Diet- and Nutrition-Related Randomized Controlled Trials and Systematic Reviews With Meta-Analysis: Protocol for 2 Independent Meta-Research Studies.

Journal articles describing randomized controlled trials (RCTs) and systematic reviews with meta-analysis of RCTs are not optimally reported and often miss crucial details. This poor reporting makes assessing these studies' risk of bias or reproducing their results difficult. However, the reporting quality of diet- and nutrition-related RCTs and meta-analyses has not been explored. We aimed to assess the reporting completeness and identify the main reporting limitations of diet- and nutrition-related RCTs and meta-analyses of RCTs, estimate the frequency of reproducible research practices among these RCTs, and estimate the frequency of distorted presentation or spin among these meta-analyses. Two independent meta-research studies will be conducted using articles published in PubMed-indexed journals. The first will include a sample of diet- and nutrition-related RCTs; the second will include a sample of systematic reviews with meta-analysis of diet- and nutrition-related RCTs. A validated search strategy will be used to identify RCTs of nutritional interventions and an adapted strategy to identify meta-analyses in PubMed. We will search for RCTs and meta-analyses indexed in 1 calendar year and randomly select 100 RCTs (June 2021 to June 2022) and 100 meta-analyses (July 2021 to July 2022). Two reviewers will independently screen the titles and abstracts of records yielded by the searches, then read the full texts to confirm their eligibility. The general features of these published RCTs and meta-analyses will be extracted into a research electronic data capture database (REDCap; Vanderbilt University). The completeness of reporting of each RCT will be assessed using the items in the CONSORT (Consolidated Standards of Reporting Trials), its extensions, and the TIDieR (Template for Intervention Description and Replication) statements. Information about practices that promote research transparency and reproducibility, such as the publication of protocols and statistical analysis plans will be collected. There will be an assessment of the completeness of reporting of each meta-analysis using the items in the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) statement and collection of information about spin in the abstracts and full-texts. The results will be presented as descriptive statistics in diagrams or tables. These 2 meta-research studies are registered in the Open Science Framework. The literature search for the first meta-research retrieved 20,030 records and 2182 were potentially eligible. The literature search for the second meta-research retrieved 10,918 records and 850 were potentially eligible. Among them, random samples of 100 RCTs and 100 meta-analyses were selected for data extraction. Data extraction is currently in progress, and completion is expected by the beginning of 2023. Our meta-research studies will summarize the main limitation on reporting completeness of nutrition- or diet-related RCTs and meta-analyses and provide comprehensive information regarding the particularities in the reporting of intervention studies in the nutrition field. DERR1-10.2196/43537.