Statistical Pitfalls Research Articles

Statistical Inference Enables Bad Science; Statistical Thinking Enables Good Science

ABSTRACT Scientific research of all kinds should be guided by statistical thinking: in the design and conduct of the study, in the disciplined exploration and enlightened display of the data, and to avoid statistical pitfalls in the interpretation of the results. However, formal, probability-based statistical inference should play no role in most scientific research, which is inherently exploratory, requiring flexible methods of analysis that inherently risk overfitting. The nature of exploratory work is that data are used to help guide model choice, and under these circumstances, uncertainty cannot be precisely quantified, because of the inevitable model selection bias that results. To be valid, statistical inference should be restricted to situations where the study design and analysis plan are specified prior to data collection. Exploratory data analysis provides the flexibility needed for most other situations, including statistical methods that are regularized, robust, or nonparametric. Of course, no individual statistical analysis should be considered sufficient to establish scientific validity: research requires many sets of data along many lines of evidence, with a watchfulness for systematic error. Replicating and predicting findings in new data and new settings is a stronger way of validating claims than blessing results from an isolated study with statistical inferences.

Biostatistics as a Tool for Medical Research: What are we Doing Wrong?

SUMMARYAs the data resulting from modern clinical research are astonishingly complex, statistics constitute an integral component of the research project, from the study conception and design to the reporting of the results. At the same time, the Medical Literature is not immune to statistical pitfalls. In the following lines we identify eleven common statistical mistakes, elucidate their effects on the study credibility, and provide tips and tricks to avoid them.

Learning the Structure of Bayesian Networks: A Quantitative Assessment of the Effect of Different Algorithmic Schemes

One of the most challenging tasks when adopting Bayesian networks (BNs) is the one of learning their structure from data. This task is complicated by the huge search space of possible solutions and by the fact that the problem isNP-hard. Hence, a full enumeration of all the possible solutions is not always feasible and approximations are often required. However, to the best of our knowledge, a quantitative analysis of the performance and characteristics of the different heuristics to solve this problem has never been done before. For this reason, in this work, we provide a detailed comparison of many different state-of-the-art methods for structural learning on simulated data considering both BNs with discrete and continuous variables and with different rates of noise in the data. In particular, we investigate the performance of different widespread scores and algorithmic approaches proposed for the inference and the statistical pitfalls within them.

Top ten errors of statistical analysis in observational studies for cancer research.

Observational studies using registry data make it possible to compile quality information and can surpass clinical trials in some contexts. However, data heterogeneity, analytical complexity, and the diversity of aspects to be taken into account when interpreting results makes it easy for mistakes to be made and calls for mastery of statistical methodology. Some questionable research practices that include poor analytical data management are responsible for the low reproducibility of some results; yet, there is a paucity of information in the literature regarding specific statistical pitfalls of cancer studies. In addition to proposing how to avoid or solve them, this article seeks to expose ten common problematic situations in the analysis of cancer registries: convenience, dichotomization, stratification, regression to the mean, impact of sample size, competing risks, immortal time and survivor bias, management of missing values, and data dredging.

Implementing Machine Learning in Radiology Practice and Research.

The purposes of this article are to describe concepts that radiologists should understand to evaluate machine learning projects, including common algorithms, supervised as opposed to unsupervised techniques, statistical pitfalls, and data considerations for training and evaluation, and to briefly describe ethical dilemmas and legal risk. Machine learning includes a broad class of computer programs that improve with experience. The complexity of creating, training, and monitoring machine learning indicates that the success of the algorithms will require radiologist involvement for years to come, leading to engagement rather than replacement.

Reliability and minimal detectable change of transcranial magnetic stimulation outcomes in healthy adults: A systematic review.

Transcranial magnetic stimulation (TMS) is used worldwide for noninvasively testing human motor systems but its psychometric properties remain unclear. This work systematically reviewed studies on the reliability of TMS outcome measures of primary motor cortex (M1) excitability in healthy humans, with an emphasis on retrieving minimal detectable changes (MDC). The literature search was performed in three databases (Pubmed, CINAHL, Embase) up to June 2016 and additional studies were identified through hand-searching. French and English-written studies had to report the reliability of at least one TMS outcome of M1 in healthy humans. Two independent raters assessed the eligibility of potential studies, and eligible articles were reviewed using a structured data extraction form and two critical appraisal scales. A total of 34 articles met the selection criteria, which tested the intra- and inter-rater reliability (relative and absolute subtypes) of several TMS outcomes. However, our critical appraisal of studies raised concerns on the applicability and generalization of results because of methodological and statistical pitfalls. Importantly, MDC were generally large and likely affected by various factors, especially time elapsed between sessions and number of stimuli delivered. This systematic review underlined that the evidence about the reliability of TMS outcomes is scarce and affected by several methodological and statistical problems. Data and knowledge of the review provided however relevant insights on the ability of TMS outcomes to track plastic changes within an individual or within a group, and recommendations were made to level up the quality of future work in the field.

STATISTICS, BIOMEDICINE AND SCIENTIFIC FRAUD

A consistent fraction of published data on scientific journals is not reproducible mainly due to insufficient knowledge of statistical methods. Here, we discuss on the use of proper statistical tools in biomedical research and statistical pitfalls potentially undermining the scientific validity of published data. Apart from unaware errors, a growing concern exists regarding data fabrication and scientific misconduct. Indeed, the social impact of false scientific data can be largely unpredictable and devastating, as shown by the worldwide dramatic effects on vaccinations coverage following a retracted paper published on a highly authoritative medical journal. Unfortunately, statistics shows a quite limited power in detecting false science, although a few statistical tools, such as the Benford’s law, are known. Taken together, statistics in biomedical sciences i) is a powerful tool to interpret experimental data; ii) has limited power in detecting false science; and iii) first and foremost, is not the result of a simple “click of a mouse”, but should be the result of accurate research planning by experienced and knowledgeable users.

Detecting and avoiding likely false-positive findings-a practical guide.

Recently there has been a growing concern that many published research findings do not hold up in attempts to replicate them. We argue that this problem may originate from a culture of 'you can publish if you found a significant effect'. This culture creates a systematic bias against the null hypothesis which renders meta-analyses questionable and may even lead to a situation where hypotheses become difficult to falsify. In order to pinpoint the sources of error and possible solutions, we review current scientific practices with regard to their effect on the probability of drawing a false-positive conclusion. We explain why the proportion of published false-positive findings is expected to increase with (i) decreasing sample size, (ii) increasing pursuit of novelty, (iii) various forms of multiple testing and researcher flexibility, and (iv) incorrect P-values, especially due to unaccounted pseudoreplication, i.e. the non-independence of data points (clustered data). We provide examples showing how statistical pitfalls and psychological traps lead to conclusions that are biased and unreliable, and we show how these mistakes can be avoided. Ultimately, we hope to contribute to a culture of 'you can publish if your study is rigorous'. To this end, we highlight promising strategies towards making science more objective. Specifically, we enthusiastically encourage scientists to preregister their studies (including a priori hypotheses and complete analysis plans), to blind observers to treatment groups during data collection and analysis, and unconditionally to report all results. Also, we advocate reallocating some efforts away from seeking novelty and discovery and towards replicating important research findings of one's own and of others for the benefit of the scientific community as a whole. We believe these efforts will be aided by a shift in evaluation criteria away from the current system which values metrics of 'impact' almost exclusively and towards a system which explicitly values indices of scientific rigour.

Common Statistical Pitfalls in Basic Science Research.

The analysis of clinical samples, population samples, and controlled trials is typically subjected to rigorous statistical review. This fact is understandable, given that the results of clinical investigation will often be used to inform patient care or clinical decision making. One would not want

Four Common Pitfalls of Quantitative Analysis in Experimental Research.

A recently published AM Last Page presents five common methodological pitfalls of experimental research in medical education. In this Last Page, we present four statistical pitfalls and their more appropriate alternatives. Pitfalls are illustrated with a case of a fictitious researcher who conducts a study with elements that are common in many medical education experiments.

Can sliding-window correlations reveal dynamic functional connectivity in resting-state fMRI?

During the last several years, the focus of research on resting-state functional magnetic resonance imaging (fMRI) has shifted from the analysis of functional connectivity averaged over the duration of scanning sessions to the analysis of changes of functional connectivity within sessions. Although several studies have reported the presence of dynamic functional connectivity (dFC), statistical assessment of the results is not always carried out in a sound way and, in some studies, is even omitted. In this study, we explain why appropriate statistical tests are needed to detect dFC, we describe how they can be carried out and how to assess the performance of dFC measures, and we illustrate the methodology using spontaneous blood-oxygen level-dependent (BOLD) fMRI recordings of macaque monkeys under general anesthesia and in human subjects under resting-state conditions. We mainly focus on sliding-window correlations since these are most widely used in assessing dFC, but also consider a recently proposed non-linear measure. The simulations and methodology, however, are general and can be applied to any measure. The results are twofold. First, through simulations, we show that in typical resting-state sessions of 10min, it is almost impossible to detect dFC using sliding-window correlations. This prediction is validated by both the macaque and the human data: in none of the individual recording sessions was evidence for dFC found. Second, detection power can be considerably increased by session- or subject-averaging of the measures. In doing so, we found that most of the functional connections are in fact dynamic. With this study, we hope to raise awareness of the statistical pitfalls in the assessment of dFC and how they can be avoided by using appropriate statistical methods.

Flow Cytometry in Transplantation.

Flow Cytometry in Transplantation.

CORR Insights®: What Host Factors Affect Aseptic Loosening After THA and TKA?

A septic loosening is a major cause of construct failure after THA and TKA [2, 4]. Studies suggest that loosening accounts for about 55% of the total failures in THA and 31% in TKA [5, 7]. Excessive production of wear particles induces osteoclast differentiation and macrophage production, leading to inflammatory-mediated osteolysis, which can progress to aseptic loosening [1, 3, 6]. The current systematic review suggested that two host factors (male gender and high activity level) were associated with an increased risk of aseptic loosening for THA, whereas the other two host factors they analyzed (obesity and tobacco use) were not. For TKA, the review did not find any host factors associated with loosening. However,many host (and other clinical) factors can be interrelated, making it difficult to quantitatively analyze how much a single host factor contributes to loosening. Additionally, statistical pitfalls can complicate matters, in particular insufficient statistical power, resulting in the failure to detect the influence of one ormore factors that could result in an increased risk of loosening.

Statistical Pitfalls: Problems with the Statistical Representation of Fieldwork Data from Social Research

Students, as well as professional research proposals have been rejected by the proposal committee and some grants awarding bodies for lack of statistical representation. Statistics is the science that is dealing with the collection and organisation, analysis and interpretation of numerical data from which conclusions about the variables under study, are made. This contribution to the pathway to ensuring students’ success argues that it would be misleading and inappropriate to suggest that only studies, which are backed-up with statistical tools, are rigorous and meaningful. Previous research has argued that statistical techniques play crucial role for uncovering hidden knowledge. However, in the present paper, the researchers argue that to assign such potency to statistical tools appear misleading especially to students and newer researchers. The paper notes that of about twenty-three statistical tools available for use by researchers, none possesses any investigative quality. What statistics possesses is an analytic strength, whereby numerical and calculative meanings are assigned to assembled data that cannot uncover beyond the strength of the data-collecting instrument. In this paper, the researchers examine the meaning of statistics, the use of language as a tool of statistics, statistical data and its types; scale of measurement and appropriate statistics, and some of the pitfalls in the use of statistical tools in analysing data from social and behavioural studies. The implication of this for students’ research and success are equally highlighted. DOI: 10.5901/mjss.2014.v5n20p1448

A review of prognostic factors in thymic malignancies

Prognosis is a central piece of information we would like to know when someone is diagnosed with a disease. Prognosis is affected by many factors. For a cancer, one important aspect is the anatomic extent of the tumor, typically described by a stage classification. Nevertheless, a stage classification system should not be confused with a system to predict prognosis—tumor stage is only one aspect. Prognosis is affected by the treatment given, by patient-specific factors (i.e., comorbidities), tumor-related factors, and others, collectively known as prognostic factors. The International Thymic Malignancy Interest Group (ITMIG) is currently engaged in the development of a validated formal stage classification system for thymic malignancies, together with the International Association for the Study of Lung Cancer and under the auspices of the Union International Contre le Cancer (UICC) and American Joint Commission on Cancer (AJCC). Assessment of prognostic factors is an important corollary to development of an anatomic stage classification system. A review of the current literature identifying prognostic factors is an important baseline to have for this initiative. The study and classification of prognostic factors, in general, is very rudimentary. Often there is confusion because people are not clear about the outcome in question: a factor may be prognostic relative to overall survival, to cure from the disease, to response to treatment, etc. Prognostic factors are often specific to a subgroup of patients and a particular setting: factors associated with prognosis for a patient with thymoma undergoing surgery may be different from that of a patient with thymic carcinoma undergoing chemotherapy and radiation. Prognostic factors can be divided into “domains” of tumor-related factors, host factors (i.e., that would be present in the patient even if there was no tumor), and environmental factors (including things such as access to optimal care). Furthermore, once we identify a prognostic factor, we immediately begin trying to change the outcome to be what we want to see (i.e., to undermine the predictive value of the prognostic factor). Finally, there are many statistical pitfalls and errors in the definition of prognostic factors, as discussed in another article in this issue.1 Despite deficiencies in our level of sophistication about the science of prognostic factors, the need to estimate future outcomes for patients is great. This article sets out to provide a review of the current state of affairs in the area of thymic malignancies.

Avoidable statistical pitfalls in analyzing length of stay in intensive care units or hospitals

In a study of ventilated patients in a recent issue of Critical Care, Shahin and colleagues [1] compared the length of stay in ICU between patients who acquired a suspected ventilator-associated respiratory infection (sVARI) and patients who remained free of this infection. There are two basic facts that require special attention in the statistical analysis. First, ventilation is a time-dependent study entry since some patients receive ventilation a few days after the time of admission to the ICU. In keeping with the study design, these patients cannot be discharged between admission and first ventilation; ignoring this fact leads to length bias [2]. Second, sVARI is a time-dependent exposure since sVARI might occur a few days after the start of ventilation; treating sVARI incorrectly as time-fixed (that is, assumed to be known at the time of admission) leads to time-dependent bias [2]. The statistical models used by Shahin and colleagues incorrectly assumed that ventilation and sVARI occurrence already happened at the day of admission. Both types of bias can be substantial in such settings: in a study of patients colonized with methicillin-resistant Staphylococcus aureus (MRSA), the biased extra length of hospital stay of MRSA-infected patients was 24.5 days whereas the correct estimate was only 6 days [2]. Length and time-dependent bias are avoidable types of bias since the chronological order (admission, time of first ventilation, time of sVARI, and time of discharge) can be acknowledged in a multi-state model [2]. Researchers are encouraged to use adequate models to receive valid results for hospital-acquired infections [3]. Authors’ response Jason Shahin and Arnold S Kristof We thank Dr Wolkewitz for indicating the effects of length and time-dependent biases in studies that use survival analyses to infer a causal link between a time-dependent exposure and time-based clinical outcomes. The time-dependent bias in the study attempting to establish S. aureus bacteremia as a cause of increased duration of admission is a good example, especially since the exposure tends to occur later in the ICU course. In our study on sVARIs [1], we looked for an association between sVARI and patient co-morbidities or outcomes common to contemporary ICUs. Although we focused on iatrogenic immunosuppression, there was also an association between sVARI and duration of mechanical ventilation or length of stay. However, we did not wish to imply a causal effect of sVARI on the latter outcomes. As Wolkewitz indicates, this would require a prospective proportional hazards-based analysis accounting for the duration of admission or mechanical ventilation prior to sVARI in a multi-state model. In contrast, our findings also incorporate a potential effect of duration of mechanical ventilation or ICU stay on the prevalence of sVARI and in part describe the nature of the patient who develops sVARI. This observation would likely be lost in a multi-state model. Nonetheless, we agree with Dr Wolkewitz that it would be of interest to incorporate the time to event in a separate statistical analysis that attempts to address a potential causal effect between sVARI and time-dependent clinical outcomes.

Applied statistics in ecology: common pitfalls and simple solutions

The most common statistical pitfalls in ecological research are those associated with data exploration, the logic of sampling and design, and the interpretation of statistical results. Although one can find published errors in calculations, the majority of statistical pitfalls result from incorrect logic or interpretation despite correct numerical calculations. There are often simple solutions to avoiding these problems that require only patience, clarity of thinking, probabilistic insight, and a reduced reliance on out‐of‐the‐box approaches. Some of these trouble spots are inherent to all statistical analyses and others are particularly insidious in ecology where true controls or replication are challenging, small sample sizes are common, and correctly linking mathematical constructs and ecological ideas is essential. Here we summarize the most common statistical pitfalls observed over nearly a century of combined consulting and research experience in ecological statistics. We provide short, simple solutions.

PP39 Research using Ankle Brachial Pressure Index Statistically Reassessed

BackgroundAnkle Brachial Pressure Index (ABPI) is the ratio of ankle to brachial systolic blood pressure. It is widely used as a non-invasive, relatively simple method for the diagnosis of peripheral...

Pearls and pitfalls

AbstractThe instructor will highlight some pearls and pitfalls in statistics that both authors and readers benefit from knowing, based on his personal experience as a writer, reviewer and editorial board member of the EVER Journal, Acta Ophthalmologica. The items discussed, based on examples from recent literature, will include: 1. how to avoid giving a false impression of precision when reporting key results, 2. how to communicate effect size; 3. how to differentiate statistical from clinical significance, 4. how to communicate statistically non‐significant results, and 5. how to recognise multiple comparisons and adjust for them. Clear reporting of basic statistics is not difficult as long as one is aware of some basic rules. Putting them into action may improve your manuscripts and reading skills when scanning published evidence to base your practice on.

Is a Tax Amnesty a Good Fiscal Policy? A Review of State Experience in the USA

During and immediately after the recessions of 2001 and 2007-2009, tax amnesties in the American states rebounded in popularity. Is a tax amnesty a good fiscal policy? To address this question, we review the experience of US state tax amnesty between 1982-2012 and identify the literature that support the argument that tax amnesties not only raise higher tax revenues for the state treasury in the short term, but also do not have negative effects on tax compliance in the long term. We then point out some of the statistical pitfalls in the literature and explain why they are important.