Validity Of Clinical Research Research Articles

Evidence-based medicine or statistically manipulated medicine? Are we slaves to the P-value?

Abstract First popularized almost a century ago in epidemiologic research by Ronald Fisher and Jerzy Neyman, the P-value has become perhaps the most misunderstood and even misused statistical value or descriptor. Indeed, modern clinical research has now come to be centered around and guided by an arbitrary P-value of <0.05 as a magical threshold for significance, so much so that experimental design, reporting of experimental findings, and interpretation and adoption of such findings have become largely dependent on this “significant” P-value. This has given rise to multiple biases in the overall body of biomedical literature that threatens the very validity of clinical research. Ultimately, a drive toward reporting a “significant” P-value (by various statistical manipulations) risks creating a falsely positive body of science, leading to (i) wasted resources in pursuing fruitless research and (ii) futile or even harmful policies/therapeutic recommendations. This article reviews the history of the P-value, the conceptual basis of P-value in the context of hypothesis testing and challenges in critically appraising clinical evidence vis-à-vis the P-value. This review is aimed at raising awareness of the pitfalls of this rigid observation of the threshold of statistical significance when evaluating clinical trials and to generate discussion regarding whether the scientific body needs a rethink about how we decide clinical significance.

195-LB: Type 2 Diabetes Data Quality in the All of Us Dataset

Data quality significantly impacts the reproducibility and validity of clinical research. The All of Us Research Program aims to collect biomedical data from one million or more participants, with the goal of enabling discoveries and improving targeted management of diseases like type 2 diabetes (T2D). All of Us data include whole genome sequences (WGS), electronic health records (EHRs), device measurements of physical activity, physical measures, and surveys available for analysis on the Researcher Workbench. The program's Spring 2023 curated data release makes All of Us the world's largest, most diverse genomic dataset of its kind available for broad research use, including 413,457 participants among which 60.4% are female, 44.6% are non-white. To ensure maximum utility of this dataset for T2D researchers, we set out to quantify the data quality among participants who have T2D diagnosis codes in their EHRs. We identified the measurements and laboratory tests (hemoglobin A1C, height, weight, body mass index), medications (Insulin and non-insulin), and procedures commonly used in T2D research. We quantified data fitness using five dimensions of quality: completeness, concordance (i.e., agreement), conformance to data standards, plausibility, and temporality. Among 287,012 participants who shared EHRs, 40,093 (14%) had T2D diagnosis codes where the mean age at first diagnosis was 56. Regarding measurements, 68% had A1C, 89% had height, and 70% had weight recorded in their EHR. For plausibility, 99.7% of weights, 99.9% of heights, and 99.3% of BMI were valid, and 97% of heights and weights have high concordance values. For medications, 99.7% of T2D participants had T2D medications, and 47.6% were prescribed after T2D diagnosis. Of these T2D participants, 69% have WGS or array data available. Our analysis shows that the All of Us dataset offers a valuable and high-quality dataset for T2D phenotyping and diagnosis research. Our quality report and code will be available for replication and reuse by researchers with the upcoming release. Disclosure L. Sulieman: None. J. Giannini: None. E. Dede yildirim: None. Y. Ostchega: None. E. Ochsenfaber: None. L. Berman: None. A. Ramirez: n/a.

Healthcare utilization is a collider: an introduction to collider bias in EHR data reuse.

Collider bias is a common threat to internal validity in clinical research but is rarely mentioned in informatics education or literature. Conditioning on a collider, which is a variable that is the shared causal descendant of an exposure and outcome, may result in spurious associations between the exposure and outcome. Our objective is to introduce readers to collider bias and its corollaries in the retrospective analysis of electronic health record (EHR) data. Collider bias is likely to arise in the reuse of EHR data, due to data-generating mechanisms and the nature of healthcare access and utilization in the United States. Therefore, this tutorial is aimed at informaticians and other EHR data consumers without a background in epidemiological methods or causal inference. We focus specifically on problems that may arise from conditioning on forms of healthcare utilization, a common collider that is an implicit selection criterion when one reuses EHR data. Directed acyclic graphs (DAGs) are introduced as a tool for identifying potential sources of bias during study design and planning. References for additional resources on causal inference and DAG construction are provided.

Missing data: the impact of what is not there.

The validity of clinical research is potentially threatened by missing data. Any variable measured in a study can have missing values, including the exposure, the outcome, and confounders. When missing values are ignored in the analysis, only those subjects with complete records will be included in the analysis. This may lead to biased results and loss of power. We explain why missing data may lead to bias and discuss a commonly used classification of missing data.

Musculoskeletal Ultrasound Scanning Protocol Consensus Statements on Scanning Conventions and Documentation in the US.

European rheumatology and radiology-determined standards have largely driven the execution of ultrasound in rheumatology (RhUS). How this translates to American rheumatologic practice has not been examined. A rheumatology-driven consensus on documentation, scanning conventions, and tiered-mastery designation for anatomic region views was developed in 2011 and served as the framework for training and clinical research validation. The present study was undertaken to update this consensus to reflect current utilization of musculoskeletal RhUS evaluation in the US. A 3-round Delphi method study was conducted using a 96-item questionnaire sent via Qualtrics survey software to 101 respondents experienced in RhUS education and scholarship. The target participant number was 38. High agreement was defined as ≥85% agreement on each item. McNemar's chi-square test was used to analyze changes in agreement in the responses. Comments were reviewed for content analysis. A total of 46 respondents completed all 3 rounds. Of documentation and scanning convention statements, 80% and 100%, respectively, reached high agreement. Comments reflected the need for rheumatology-defined and disease-specific complete scan and limited scan definitions, separate from radiology-defined definitions. Many scanning conventions from 2011 remain relevant in current practice. There is a need to determine rheumatology-defined descriptions for common procedural terminology codes for complete and limited scans that accurately reflect the current state of RhUS.

Preprints: a Timely Counterbalance for Big Data-Driven Research.

Big data promises to spark new discoveries but may also distort clinical research. Large datasets that permit numerous analyses could increase the number of spurious findings and threaten the reproducibility and validity of clinical research. The publication of unreproducible research is incentivized by a scientific culture that rewards novelty over rigor. Introducing preprint publication to clinical research could change the culture. The first clinical preprint platform, medRxiv, allows researchers to publish working papers in advance of peer-review to more easily share preliminary findings. Preprint publishing aims to be fast and frictionless, which fundamentally changes the incentive structure of academic publishing. Preprints offer a relatively weak reward (a preprint publication) for substantially less effort than peer-review publication. By reducing barriers to publication, preprints may help encourage scientists to publish null findings, which could mitigate publication bias. By enabling scientists to share preliminary work and publish evolving versions of manuscripts, preprints may also facilitate "workshopping" of ideas and detailed methodological review. This would better reflect the iterative nature of observational research than peer-reviewed publications, which immutably document the "final" results of a study. Preprint platforms are a timely innovation that may buffer the undesired effects of big data on clinical research.

The contribution and attitudes of research ethics committees to complete registration and non-selective reporting of clinical trials: A European survey

Background: For many years, studies have shown that the results of clinical trials are often published or reported selectively with a statistically significant bias in favour of positive trial results. Trial registration as a precondition for publication had only limited effects on current practice. Results of trials which were approved by research ethics committees (RECs) are often published only partially, with a substantial time lag or not at all. This study examined existing procedures of RECs in the European Union to monitor and prevent incomplete registration of trials and selective reporting of trial results. It further investigated opinions of REC members about the need to update current legislation on this matter. Methods: Web-based survey on members of RECs in 22 European countries. Results: Over 90 percent of respondents agreed that the incomplete publication of trial results had a strong or somewhat negative impact on public health and on healthcare professionals’ trust in the validity of clinical research, yet only 30 percent reported that their REC had some (often unsystematic) mechanism in place to check that findings of approved studies are published in some form. Less than 10 percent stated that their REC has further specific procedures in place to prevent or minimize selective reporting of study results. Respondents stated variously that their REC did not have the resources to follow up on this matter. Conclusions: The existing legislation to regulate trial registration and encourage complete publication of trial results leaves room for improvement. REC members welcome guidelines to adequately address both problems. The new Regulation EU No. 536/2014 as well as the FDA Amendment Act from 2007 require the reporting of summary results within 1 year after study end. As recent reviews demonstrated, without any systematic approach to monitor the adherence to these regulations, publication rates remain rather low.

Determinants of access to experimental antiretroviral drugs in an Italian cohort of patients with HIV: a multilevel analysis

BackgroundIdentification of the determinants of access to investigational drugs is important to promote equity and scientific validity in clinical research. We aimed to analyze factors associated with the use of experimental antiretrovirals in Italy.MethodsWe studied participants in the Italian Cohort of Antiretroviral-Naive Patients (ICoNA). All patients 18 years or older who had started cART (≥ 3 drugs including at least two NRTI) after their enrolment and during 1997-2007 were included in this analysis. We performed a random effect logistic regression analysis to take into account clustering observations within clinical units. The outcome variable was the use of an experimental antiretroviral, defined as an antiretroviral started before commercial availability, in any episode of therapy initiation/change. Use of an experimental antiretroviral obtained through a clinical trial or an expanded access program (EAP) was also analyzed separately.ResultsA total of 9,441 episodes of therapy initiation/change were analyzed in 3,752 patients. 392 episodes (360 patients) involved an experimental antiretroviral. In multivariable analysis, factors associated with the overall use of experimental antiretrovirals were: number of experienced drugs (≥ 8 drugs versus "naive": adjusted odds ratio [AOR] = 3.71) or failed antiretrovirals(3-4 drugs and ≥ 5 drugs versus 0-2 drugs: AOR = 1.42 and 2.38 respectively); calendar year (AOR = 0.80 per year) and plasma HIV-RNA copies/ml at therapy change (≥ 4 log versus < 2 log: AOR = 1.55). The probability of taking an experimental antiretroviral through a trial was significantly lower for patients suffering from liver co-morbidity(AOR = 0.65) and for those who experienced 3-4 drugs (vs naive) (AOR = 0.55), while it increased for multi-treated patients(AOR = 2.60). The probability to start an experimental antiretroviral trough an EAP progressively increased with the increasing number of experienced and of failed drugs and also increased for patients with liver co-morbidity (AOR = 1.44; p = 0.053). and for male homosexuals (vs heterosexuals: AOR = 1.67). Variability of the random effect associated to clinical units was statistically significant (p < 0.001) although no association was found with specific characteristics of clinical unit examined.ConclusionsAmong patients with HIV infection in Italy, access to experimental antiretrovirals seems to be influenced mainly by exhaustion of treatment options and not by socio-demographic factors.

The validity of research/clinical trials of CAM

This is the first in a series of papers broadly titled ‘Research Without Tears’ that will discuss fundamental and sometimes controversial areas of research methods. In general, papers in this section aim to inform or update the readership of theoretical and practical issues for research in the clinical sciences. Some of the review papers will be conceptual in nature whilst others may address specific topics related to research design and data handling. It is hoped that this will facilitate the individuals and teams to complete and disseminate their research. Another important aim of these papers will be to try to demystify the chosen topics for those people whose role is not exclusively research orientated.The aim of this specific review paper is to address the concept of validity in clinical research. This article will be followed, firstly, by a review paper that will investigate the topic of reliability in clinical research, and then by an original research article that places the determination of validity and reliability within a specific clinical context. The current paper will initially define key components of validity with regard to the general aims of research and the research design. This will include definitions of internal, external and ecological validity. Threats to internal and external validity will be discussed, as will remedies to these treats through the application of appropriate sampling and design. This paper will then investigate the narrower concept of the validity of the measuring tool or process. A brief definition will be followed by a discussion of the processes by which validity may be determined. Future papers will address other issues of relevance to the readership. The Editorial Board welcomes any specific ideas and/or submissions.

Acknowledgment of uncertainty: a fundamental means to ensure scientific and ethical validity in clinical research.

Recognition of the importance of uncertainty in the design of randomized, controlled trials (RCT) has reached the status of a principle. The "uncertainty principle," or less ambiguously, equipoise, holds that a patient should be enrolled in an RCT only if there is substantial uncertainty about which of the trial treatments would benefit the patient most. In fact, the "uncertainty principle" addresses the most important issue of a clinical trial--the choice of an adequate comparative control. Studies in which intervention and control group are believed to be non-equivalent violate the uncertainty principle. Therefore, one would expect that both editors and authors would be particularly careful to include a statement concerning prior beliefs of the investigator(s) about the uncertainty of the treatments that are reported. However, we found no evidence of such a policy in the randomized, controlled trials we examined. We also show that there is a predictable relationship between the uncertainty principle, that is, the moral principle upon which trials are based, and the ultimate outcomes of clinical trials. We postulate that about 50% of innovations are successful, leading to the conclusion that preserving the ethics of clinical research may be the best investment strategy available.

Vascular viewpoint: an evidence-based approach to vascular medicine

Given the large number of peer review articles that are published in the medical literature each week (e.g. 3000 new postings on MEDLINE per week!), and the easy accessibility of these papers using internet databases, clinicians and researchers are inundated with materials to digest and process in order to keep up with the latest therapeutic interventions they can use to treat their patients. To assist in the interpretation of this huge volume of information, clinicians and researchers need to develop skills that will enable them to assess the validity of clinical research, and to understand when enough evidence has been accumulated for a therapeutic intervention for it to become ‘clearly’ indicated. Although this evidence-based approach to assessing the medical literature is not new, we believe a greater emphasis should be placed on critical appraisal in the area of vascular medicine, and an ideal forum for this is the quarterly publication of Vascular Medicine. Thus, we are introducing a new feature called ‘Vascular viewpoint’, which is intended to assist the reader in synthesizing and evaluating clinical studies related to the treatment of vascular diseases. To avoid complicated ranking scales of levels of evidence, we will use understandable ‘at a glance’ terminology to categorize studies. The type of study will be labeled as Randomized—large ‐ meaning a randomized trial of adequate sample size; Randomized—small ‐ meaning a randomized study of small size; Systematic overview ‐ a quantitative review of randomized and non-randomized studies; Nonrandomized studies such as cohort, case‐control, cross-sectional, and case series; and Investigational studies ‐ referring to physiologic investigations (see Table 1). Study methods will be classified as being high, moderate or low quality, and the validity of the conclusions will be classified

Validity in clinical research: a review of basic concepts and definitions

This is the first in a series of papers broadly titled ‘Research Without Tears’ that will discuss fundamental and sometimes controversial areas of research methods. In general, papers in this section aim to inform or update the readership of theoretical and practical issues for research in the clinical sciences. Some of the review papers will be conceptual in nature whilst others may address specific topics related to research design and data handling. It is hoped that this will facilitate the individuals and teams to complete and disseminate their research. Another important aim of these papers will be to try to demystify the chosen topics for those people whose role is not exclusively research orientated.The aim of this specific review paper is to address the concept of validity in clinical research. This article will be followed, firstly, by a review paper that will investigate the topic of reliability in clinical research, and then by an original research article that places the determination of validity and reliability within a specific clinical context. The current paper will initially define key components of validity with regard to the general aims of research and the research design. This will include definitions of internal, external and ecological validity. Threats to internal and external validity will be discussed, as will remedies to these treats through the application of appropriate sampling and design. This paper will then investigate the narrower concept of the validity of the measuring tool or process. A brief definition will be followed by a discussion of the processes by which validity may be determined. Future papers will address other issues of relevance to the readership. The Editorial Board welcomes any specific ideas and/or submissions.

The Munroe Check List: A Note on its Validity in Clinical Research

Increasing use of the Rorschach Inkblots Test is being made in medical diagnosis, not only in psychiatry, but in neurology as well. Bibliography is too extensive to quote here, and the reader is referred to the standard work by Klopfer et al. (1956) which lists nearly 3,000 references specific to Rorschach literature. The inclusion of a long chapter on the clinical use of this test in the latest research monograph of the Association for Research in Nervous and Mental Diseases (1954) is evidence of the increasing esteem in which this test is held by specialists in neurology and psychiatry, supported by the growing use of the test in diagnosing intracranial organic pathology. It is interesting to note that in the latter field the Rorschach has proved superior in diagnostic accuracy to any other single neurological test, such as EEG, lumbar puncture, ventriculography, etc., and is often as good as the complete battery together.