False Positive Conclusions Research Articles

A Statistical Significance Test for Necessary Condition Analysis

In this article, we present a statistical significance test for necessary conditions. This is an elaboration of necessary condition analysis (NCA), which is a data analysis approach that estimates the necessity effect size of a condition X for an outcome Y. NCA puts a ceiling on the data, representing the level of X that is necessary (but not sufficient) for a given level of Y. The empty space above the ceiling relative to the total empirical space characterizes the necessity effect size. We propose a statistical significance test that evaluates the evidence against the null hypothesis of an effect being due to chance. Such a randomness test helps protect researchers from making Type 1 errors and drawing false positive conclusions. The test is an “approximate permutation test.” The test is available in NCA software for R. We provide suggestions for further statistical development of NCA.

We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness

False positive findings are a growing problem in many research literatures. We argue that excessive false positives often stem from model uncertainty. There are many plausible ways of specifying a regression model, but researchers typically report only a few preferred estimates. This raises the concern that such research reveals only a small fraction of the possible results and may easily lead to nonrobust, false positive conclusions. It is often unclear how much the results are driven by model specification and how much the results would change if a different plausible model were used. Computational model robustness analysis addresses this challenge by estimating all possible models from a theoretically informed model space. We use large-scale random noise simulations to show (1) the problem of excess false positive errors under model uncertainty and (2) that computational robustness analysis can identify and eliminate false positives caused by model uncertainty. We also draw on a series of empirical applications to further illustrate issues of model uncertainty and estimate instability. Computational robustness analysis offers a method for relaxing modeling assumptions and improving the transparency of applied research.

Improving network inference: The impact of false positive and false negative conclusions about the presence or absence of links

BackgroundA reliable inference of networks from data is of key interest in the Neurosciences. Several methods have been suggested in the literature to reliably determine links in a network. To decide about the presence of links, these techniques rely on statistical inference, typically controlling the number of false positives, paying little attention to false negatives. New methodIn this paper, by means of a comprehensive simulation study, we analyse the influence of false positive and false negative conclusions about the presence or absence of links in a network on the network topology. We show that different values to balance false positive and false negative conclusions about links should be used in order to reliably estimate network characteristics. We propose to run careful simulation studies prior to making potentially erroneous conclusion about the network topology. ResultsOur analysis shows that optimal values to balance false positive and false negative conclusions about links depend on the network topology and characteristic of interest. Comparison with existing methodsExisting methods rely on a choice of the rate for false positive conclusions. They aim to be sure about individual links rather than the entire network. The rate of false negative conclusions is typically not investigated. ConclusionsOur investigation shows that the balance of false positive and false negative conclusions about links in a network has to be tuned for any network topology that is to be estimated. Moreover, within the same network topology, the results are qualitatively the same for each network characteristic, but the actual values leading to reliable estimates of the characteristics are different.

Consequences of relying on statistical significance: Some illustrations.

Despite regular criticisms of null hypothesis significance testing (NHST), a focus on testing persists, sometimes in the belief to get published and sometimes encouraged by journal reviewers. This paper aims to demonstrate known key limitations of NHST using simple nontechnical illustrations. The first illustration is based on simulated data of 20000 studies that compare two groups for an outcome event. The true effect size (difference in event rates) and sample size (20-100 per group) were varied. The second illustration used real data from a meta-analysis on alpha-blockers for the treatment of ureteric stones. The simulations demonstrated the large between-study variability in P-values (range between <.0001 and 1 for most simulation conditions). A focus on statistically significant effects (P<.05), notably in small to moderate samples, led to strongly overestimated effect sizes (up to 240%) and many false-positive conclusions, that is statistically significant effects that were, in fact, true null effects. Effect sizes also exerted strong between-study variability, but confidence intervals accounted for this: the interval width decreased with larger sample size, and the percentage of intervals that contained the true effect size was accurate across simulation conditions. Reducing alpha level, as recently suggested, reduced false-positive conclusions but strongly increased the overestimation of significant effects (up to 320%). Researchers and journals should abandon statistical significance as a pivotal element in most scientific publications. Confidence intervals around effect sizes are more informative, but should not merely be reported to comply with journal requirements.

Multiple Testing and the Distributional Effects of Accountability Incentives in Education

Economic theory that underlies many empirical microeconomic applications predicts that treatment responses depend on individuals’ characteristics and location on the outcome distribution. Using data from a large-scale Pakistani school report card experiment, we consider tests for treatment effect heterogeneity that make corrections for multiple testing to avoid an overestimation of positive treatment effects. These tests uncover evidence of policy-relevant heterogeneous effects from information provision on child test scores. Further, our analysis reinforces the importance of preventing the inflation of false positive conclusions since over 65% of the estimated statistically significant quantile treatment effects become insignificant once these corrections are applied.

Considering common sources of exposure in association studies - Urinary benzophenone-3 and DEHP metabolites are associated with altered thyroid hormone balance in the NHANES 2007–2008

Epidemiological studies have shown that thyroid hormone balances can be disrupted by chemical exposure. However, many association studies have often failed to consider multiple chemicals with possible common sources of exposure, rendering their conclusions less reliable.In the 2007–2008 National Health and Nutrition Examination Survey (NHANES) from the U.S.A., urinary levels of environmental phenols, parabens, and phthalate metabolites as well as serum thyroid hormones were measured in a general U.S. population (≥12years old, n=1829). Employing these data, first, the chemicals or their metabolites associated with thyroid hormone measures were identified. Then, the chemicals/metabolites with possible common exposure sources were included in the analytical model to test the sensitivities of their association with thyroid hormone levels.Benzophenone-3 (BP-3), bisphenol A (BPA), and a metabolite of di(2-ethylhexyl) phthalate (DEHP) were identified as significant determinants of decreased serum thyroid hormones. However, significant positive correlations were detected (p-value<0.05, r=0.23 to 0.45) between these chemicals/metabolites, which suggests that they might share similar exposure sources. In the subsequent sensitivity analysis, which included the chemicals/metabolite with potentially similar exposure sources in the model, we found that urinary BP-3 and DEHP exposure were associated with decreased thyroid hormones among the general population but BPA exposure was not. In association studies, the presence of possible common exposure sources should be considered to circumvent possible false-positive conclusions.

Trial Sequential Analysis in systematic reviews with meta-analysis

BackgroundMost meta-analyses in systematic reviews, including Cochrane ones, do not have sufficient statistical power to detect or refute even large intervention effects. This is why a meta-analysis ought to be regarded as an interim analysis on its way towards a required information size. The results of the meta-analyses should relate the total number of randomised participants to the estimated required meta-analytic information size accounting for statistical diversity. When the number of participants and the corresponding number of trials in a meta-analysis are insufficient, the use of the traditional 95% confidence interval or the 5% statistical significance threshold will lead to too many false positive conclusions (type I errors) and too many false negative conclusions (type II errors).MethodsWe developed a methodology for interpreting meta-analysis results, using generally accepted, valid evidence on how to adjust thresholds for significance in randomised clinical trials when the required sample size has not been reached.ResultsThe Lan-DeMets trial sequential monitoring boundaries in Trial Sequential Analysis offer adjusted confidence intervals and restricted thresholds for statistical significance when the diversity-adjusted required information size and the corresponding number of required trials for the meta-analysis have not been reached. Trial Sequential Analysis provides a frequentistic approach to control both type I and type II errors. We define the required information size and the corresponding number of required trials in a meta-analysis and the diversity (D2) measure of heterogeneity. We explain the reasons for using Trial Sequential Analysis of meta-analysis when the actual information size fails to reach the required information size. We present examples drawn from traditional meta-analyses using unadjusted naïve 95% confidence intervals and 5% thresholds for statistical significance. Spurious conclusions in systematic reviews with traditional meta-analyses can be reduced using Trial Sequential Analysis. Several empirical studies have demonstrated that the Trial Sequential Analysis provides better control of type I errors and of type II errors than the traditional naïve meta-analysis.ConclusionsTrial Sequential Analysis represents analysis of meta-analytic data, with transparent assumptions, and better control of type I and type II errors than the traditional meta-analysis using naïve unadjusted confidence intervals.

The difficulties of cytologic diagnostic of breast diseases based on trepanobiopsy samples.

The article analyses, on the basis of scientific publications and original data, the causes of erroneous cytological conclusion concerning samples of trepanobiopsies of mammary gland. The most prevalent benign affections that are characterized by most frequent hyper diagnostic of cancer are fibro-adenoma with proliferation of epithelium and sclerosing adenosis. At the same time, certain modifications of lobular carcinoma of mammary gland, tubular, papillary cancer as well as highly differentiated invasive carcinoma of nonspecific type from cells with ulterior nuclear atypism sometimes it is difficult to diagnose at the cellular level. The sensitivity of cytological diagnostic of pathology of mammary gland using samples of trepanobiopsies makes up 97.5%, specificity - 98.5%. The number of false positive cytological conclusions about presence of malignant tumor makes up to 0.6%, false negative - 1.5%. The reliability of cytological analysis makes up to 97.4%, efficiency - 96.3%.

Regulatory Pathways for New Antimicrobial Agents: Trade-offs to Keep the Perfect From Being the Enemy of the Good.

In 2002, Shlaes and Moellering warned that pharmaceutical companies were abandoning antibiotic research and development due to changing regulatory standards regarding noninferiority (NI) clinical trials. NI trials are subject to unique biases that may yield false-positive conclusions. The US Food and Drug Administration (FDA) developed guidance to ensure that NI results truly reflect drug efficacy. These changes, intended to reduce uncertainty in trial results, have shaped trial enrollment and conduct in ways that now require reflection.

The Emperor's Old Clothes: An Inconvenient Truth About Currently Accepted Vapor Intrusion Assessment Methods

The Emperor's Old Clothes: An Inconvenient Truth About Currently Accepted Vapor Intrusion Assessment Methods

Self-Reported Sunscreen Use and Urinary Benzophenone-3 Concentrations in the United States: NHANES 2003-2006 and 2009-2012

Epidemiological studies have shown that thyroid hormone balances can be disrupted by chemical exposure. However, many association studies have often failed to consider multiple chemicals with possible common sources of exposure, rendering their conclusions less reliable.In the 2007–2008 National Health and Nutrition Examination Survey (NHANES) from the U.S.A., urinary levels of environmental phenols, parabens, and phthalate metabolites as well as serum thyroid hormones were measured in a general U.S. population (≥ 12 years old, n = 1829). Employing these data, first, the chemicals or their metabolites associated with thyroid hormone measures were identified. Then, the chemicals/metabolites with possible common exposure sources were included in the analytical model to test the sensitivities of their association with thyroid hormone levels.Benzophenone-3 (BP-3), bisphenol A (BPA), and a metabolite of di(2-ethylhexyl) phthalate (DEHP) were identified as significant determinants of decreased serum thyroid hormones. However, significant positive correlations were detected (p-value < 0.05, r = 0.23 to 0.45) between these chemicals/metabolites, which suggests that they might share similar exposure sources. In the subsequent sensitivity analysis, which included the chemicals/metabolite with potentially similar exposure sources in the model, we found that urinary BP-3 and DEHP exposure were associated with decreased thyroid hormones among the general population but BPA exposure was not. In association studies, the presence of possible common exposure sources should be considered to circumvent possible false-positive conclusions.

Simulating policy diffusion through learning: Reducing the risk of false positive conclusions

This article uses agent-based computer simulation to investigate the dynamics of policy diffusion through learning. It compares these dynamics across state systems in which policy-makers possess different capabilities to learn about policy effectiveness: independent decision-makers focusing on own experiences vs. interdependent social learners relying heavily on experiences of others. The simulation can thus compare policy adoption patterns in the presence and absence of policy diffusion within a controlled setting. The simulation makes two propositions. First, it supports the existing critique that relying on the identification of policy clusters can lead researchers to draw false positive conclusions about the relevance of policy diffusion. Second, it suggests that relying on the identification of policy volatility under political stability minimizes this risk.

Practical rules for electrodiagnosis in suspected multifocal motor neuropathy.

Multifocal motor neuropathy (MMN) with conduction block (CB) is a rare chronic immune-mediated neuropathy, but important to diagnose as it is treatable. The key features in prototypic MMN are electrodiagnostic demonstration of focal CB away from common sites of entrapment and normal sensory conduction across these sites. However, there are challenges in distinguishing CB from the effects of abnormal temporal dispersion. Consensus electrodiagnostic criteria, reinforced by modeling studies, are available to support definite or probable CB. In addition, consideration of technical issues can guard against false-positive and false-negative conclusions. These include limb temperature, stimulus site, inadvertent stimulating electrode movement, and supramaximal and submaximal responses, as well as the possibility of Martin-Gruber anastamosis. Robust evidence supports the treatment of MMN with intravenous immunoglobulin, and guidelines have been developed. Application of practical and simple rules including a 4-step diagnostic algorithm can help practitioners correctly diagnose this treatable condition and improve patient outcomes.

Satellite symposium: Emerging role of microwave imaging technology (organized by the biomedicine and molecular biosciences COST action TD1301): Microwave Imaging for Breast Cancer Detection

Breast cancer is the most common cancer in women worldwide, with nearly 1.7 million new cases diagnosed in 2012. This represents about 12% of all new cancer cases and 25% of all cancers in women. The current standard method for detecting non-palpable early stage breast cancer is X-ray mammography. Despite the fact that X-rays provide high-resolution images at low radiation doses, its limitations are well documented. In the U.S., up to 75% of all malignancies identified by X-ray mammography are later found to be benign after biopsies. These false positive conclusions result in unnecessary biopsies, causing considerable distress to the patient and an unnecessary financial burden on the health service. Much more worryingly, up to 15% of all breast cancers present at the time of screening are missed by conventional mammography, often delaying treatment to the point where it’s no longer effective. One of the most promising alternative imaging modalities is Microwave Imaging. Microwave Imaging is based on the dielectric contrast between healthy and cancerous breast tissue at microwave frequencies. Microwave imaging is non-ionising, non-invasive, does not require uncomfortable breast compression, and is potentially low cost.

Assessing the strength of directed influences among neural signals: an approach to noisy data.

Measurements in the neurosciences are afflicted with observational noise. Granger-causality inference typically does not take this effect into account. We demonstrate that this leads to false positives conclusions and spurious causalities. State space modelling provides a convenient framework to obtain reliable estimates for Granger-causality. Despite its previous application in several studies, the analytical derivation of the statistics for parameter estimation in the state space model was missing. This prevented a rigorous evaluation of the results. In this manuscript we derive the statistics for parameter estimation in the state space model. We demonstrate in an extensive simulation study that our novel approach outperforms standard approaches and avoids false positive conclusions about Granger-causality. In comparison with the naive application of Granger-causality inference, we demonstrate the superiority of our novel approach. The wide-spread applicability of our procedure provides a statistical framework for future studies. The application to mice electroencephalogram data demonstrates the immediate applicability of our approach. The analytical derivation of the statistics presented in this manuscript enables a rigorous evaluation of the results of Granger causal network inference. It is noteworthy that the statistics can be readily applied to various measures for Granger causality and other approaches that are based on vector autoregressive models.

Diverse convergent evidence in the genetic analysis of complex disease: coordinating omic, informatic, and experimental evidence to better identify and validate risk factors.

In omic research, such as genome wide association studies, researchers seek to repeat their results in other datasets to reduce false positive findings and thus provide evidence for the existence of true associations. Unfortunately this standard validation approach cannot completely eliminate false positive conclusions, and it can also mask many true associations that might otherwise advance our understanding of pathology. These issues beg the question: How can we increase the amount of knowledge gained from high throughput genetic data? To address this challenge, we present an approach that complements standard statistical validation methods by drawing attention to both potential false negative and false positive conclusions, as well as providing broad information for directing future research. The Diverse Convergent Evidence approach (DiCE) we propose integrates information from multiple sources (omics, informatics, and laboratory experiments) to estimate the strength of the available corroborating evidence supporting a given association. This process is designed to yield an evidence metric that has utility when etiologic heterogeneity, variable risk factor frequencies, and a variety of observational data imperfections might lead to false conclusions. We provide proof of principle examples in which DiCE identified strong evidence for associations that have established biological importance, when standard validation methods alone did not provide support. If used as an adjunct to standard validation methods this approach can leverage multiple distinct data types to improve genetic risk factor discovery/validation, promote effective science communication, and guide future research directions.

Exploration of heterogeneity in distributed research network drug safety analyses.

Distributed data networks representing large diverse populations are an expanding focus of drug safety research. However, interpreting results is difficult when treatment effect estimates vary across datasets (i.e., heterogeneity). In a previous study, risk estimates were generated for selected drugs and potential adverse outcomes. Analyses were replicated across eight distributed data sources using an identical analytic structure. To evaluate heterogeneity of risk estimates across data sources, the estimates were combined with summary-level data characterizing the population of each data source. Meta-analysis, meta-regression, and plots of the influence on overall results versus contribution to heterogeneity were examined and used to illustrate an approach to heterogeneity assessment. Heterogeneity, as measured by the I-squared statistic, was high with variability across outcomes. Plots of the relationship between influence on overall results and contribution to heterogeneity suggest that certain datasets and characteristics were influential but there was variability dependent on the drug and outcome being assessed. Exploratory meta-regression identified many possible influential factors, but may be subject to ecological bias and false positive conclusions. Distributed data network drug safety analyses can produce heterogeneous risk estimates that may not be easily explained. Approaches illustrated here can be useful for research that is subject to similar problems with heterogeneity.

Cancer molecular pathobiology in the clinics: concluding remarks.

This interesting issue in which the Institute of Oncology “Prof. Dr. Ion Chiricuta” of Cluj and the Cancer Institute “G. Paolo II” of Bari approach the problem of cancer biomarkers from different perspectives represents an attempt to provide a novel view of biological, clinical and methodological innovative aspects in this field. In fact, there is a critical need for reliable biomarkers in cancer in order to improve (i) early detection, (ii) accuracy of prognosis, (iii) patient stratification for treatments, and (iv) monitoring of tumor response and relapse. Although technological advances in the field of genomics, metabolomics, and proteomics have resulted in the identification of many biomarker candidates, only a few of them have been approved and display a value in clinical practice. This is particularly true for biomarkers suitable for early diagnosis of cancers, despite the considerable efforts devoted to this goal. In the field of diagnosis, one important challenge is getting biomarkers specific enough to avoid false positive conclusions that lead to unnecessarily expensive (imaging) and invasive (biopsies) interventions. Although novel technologies have produced detailed information on the molecular aspects of cancer biology, for the most part, the specificity displayed by single or multiple biomarkers is insufficient to allow the assays to be used in large populations. Very sensitive techniques to address specific hallmarks of cancer and combined approaches (metabolomics and imaging, for instance) need to be improved in the future in order to succeed in this objective. Next-generation sequencing techniques and especially RNAseq provide data about cancer-specific splice variants that can be developed as very useful biomarkers, either by detection of the variants as circulating DNA or by using antibodies when the splice variant results in novel epitopes in transmembrane or soluble proteins. In terms of prognosis and prediction of response, the situation is more favorable because of access to tumor biopsies, which allows the evaluation of different biomarkers at the protein, RNA or DNA levels. A number of signatures are commercialized or under evaluation in large clinical trials and will help clinicians to make therapeutic decisions. In the case of gene profiling in breast cancer, patients who can benefit from adjuvant chemotherapy can be selected based on a high risk of local relapse/ metastasis or to exclude low-risk patients from unnecessary therapeutic intervention. In the era of evidence-based medicine, there is an emergence of biomarkers predicting the response to targeted therapy. These biomarkers allow the selection of the patients who would or would not benefit from a given treatment. Determination of HER2 expression or KRAS mutations are mandatory for selecting patients to be treated with Trastuzumab or Cetuximab in breast or colorectal cancers, respectively. Looking at the expression of immunoregulatorymolecules at the surface of tumor cells or immune infiltrates by IHC will be a requisite to treating patients with novel neutralizing immunomodulatory antibodies recognizing these proteins. Obviously, many pharmaceutical companies develop companion diagnostics together with new drugs and invest a great deal of money in identifying pharmacokinetic, toxicological and predictive biomarkers in both preclinical settings and during early and late clinical trials.

Net risk reclassification p values: valid or misleading?

The Net Reclassification Index (NRI) and its P value are used to make conclusions about improvements in prediction performance gained by adding a set of biomarkers to an existing risk prediction model. Although proposed only 5 years ago, the NRI has gained enormous traction in the risk prediction literature. Concerns have recently been raised about the statistical validity of the NRI. Using a population dataset of 10000 individuals with an event rate of 10.2%, in which four biomarkers have no predictive ability, we repeatedly simulated studies and calculated the chance that the NRI statistic provides a positive statistically significant result. Subjects for training data (n = 420) and test data (n = 420 or 840) were randomly selected from the population, and corresponding NRI statistics and P values were calculated. For comparison, the change in the area under the receiver operating characteristic curve and likelihood ratio statistics were calculated. We found that rates of false-positive conclusions based on the NRI statistic were unacceptably high, being 63.0% in the training datasets and 18.8% to 34.4% in the test datasets. False-positive conclusions were rare when using the change in the area under the curve and occurred at the expected rate of approximately 5.0% with the likelihood ratio statistic. Conclusions about biomarker performance that are based primarily on a statistically significant NRI statistic should be treated with skepticism. Use of NRI P values in scientific reporting should be halted.

Apparently Abnormal Wechsler Memory Scale Index Score Patterns in the Normal Population

Interpretation of the Wechsler Memory Scale-Fourth Edition may involve examination of multiple memory index score contrasts and similar comparisons with Wechsler Adult Intelligence Scale-Fourth Edition ability indexes. Standardization sample data suggest that 15-point differences between any specific pair of index scores are relatively uncommon in normal individuals, but these base rates refer to a comparison between a single pair of indexes rather than multiple simultaneous comparisons among indexes. This study provides normative data for the occurrence of multiple index score differences calculated by using Monte Carlo simulations and validated against standardization data. Differences of 15 points between any two memory indexes or between memory and ability indexes occurred in 60% and 48% of the normative sample, respectively. Wechsler index score discrepancies are normally common and therefore not clinically meaningful when numerous such comparisons are made. Explicit prior interpretive hypotheses are necessary to reduce the number of index comparisons and associated false-positive conclusions. Monte Carlo simulation accurately predicts these false-positive rates.