Questionable Research Practices Research Articles

Evaluating the Pedagogical Effectiveness of Study Preregistration in the Undergraduate Dissertation

Research shows that questionable research practices (QRPs) are present in undergraduate final-year dissertation projects. One entry-level Open Science practice proposed to mitigate QRPs is “study preregistration,” through which researchers outline their research questions, design, method, and analysis plans before data collection and/or analysis. In this study, we aimed to empirically test the effectiveness of preregistration as a pedagogic tool in undergraduate dissertations using a quasi-experimental design. A total of 89 UK psychology students were recruited, including students who preregistered their empirical quantitative dissertation ( n = 52; experimental group) and students who did not ( n = 37; control group). Attitudes toward statistics, acceptance of QRPs, and perceived understanding of Open Science were measured both before and after dissertation completion. Exploratory measures included capability, opportunity, and motivation to engage with preregistration, measured at Time 1 only. This study was conducted as a Registered Report; Stage 1 protocol: https://osf.io/9hjbw (date of in-principle acceptance: September 21, 2021). Study preregistration did not significantly affect attitudes toward statistics or acceptance of QRPs. However, students who preregistered reported greater perceived understanding of Open Science concepts from Time 1 to Time 2 compared with students who did not preregister. Exploratory analyses indicated that students who preregistered reported significantly greater capability, opportunity, and motivation to preregister. Qualitative responses revealed that preregistration was perceived to improve clarity and organization of the dissertation, prevent QRPs, and promote rigor. Disadvantages and barriers included time, perceived rigidity, and need for training. These results contribute to discussions surrounding embedding Open Science principles into research training.

Developing and Validating the Competency Profile for Teaching and Learning Research Integrity

Since research integrity is not external to research but an integral part of it, it should be integrated into research training. However, several hindrances regarding contemporary research integrity education exist. To address them, we have developed a competency profile for teaching and learning research integrity based on four assumptions: 1) to include all levels of study (BA, MA, and PhD); 2) to integrate research integrity into research education itself; 3) to address research integrity issues in context-specific practices; and 4) to pay particular attention to the ‘grey zone’ or questionable research practices. To assess the validity of the content of the competency profile and to determine if some adjustments to the profile are needed, we translated the competencies of the profile into items of a measurement instrument (a questionnaire) and conducted a survey amongst University of Ljubljana students that allowed us to 1) obtain information about students’ attitudes toward issues of integrity in research; 2) analyse differences in these attitudes among BA, MA, and PhD students; and 3) statistically validate the competency profile and suggest possible improvements. The results showed that 1) students are highly aware of research integrity issues, as scores were high on all items assessed. However, there were some deviations to lower scores, especially in relation to questionable research practises, confirming our assumption that the ‘grey zone’ issues are those that should be particularly addressed and given special attention in contemporary research integrity education. 2) The differences in the attitudes of BA, MA, and PhD students showed that higher-level students showed significantly more awareness of integrity issues than lower-level students did, suggesting that research integrity issues should be given special attention at the BA study level. 3) The measurement characteristics showed that the reliability of the questionnaire was very high, suggesting a good overall structure of the competency profile. The principal component analysis also confirmed the four-field structure of the Competency profile (Values and Principles, Research Practise, Publication and Dissemination, and Violations). However, the analysis also showed that the substructure of the four main areas of the profile did not fully match the results of the factor analysis, suggesting that the distribution of competencies in the competency profile could be reconsidered, especially in the area of Research Practice. The most recent developments in the field of research integrity also suggest that the competency profile should be updated with issues regarding the impact of artificial intelligence on research integrity.

Evidence of questionable research practices in clinical prediction models

BackgroundClinical prediction models are widely used in health and medical research. The area under the receiver operating characteristic curve (AUC) is a frequently used estimate to describe the discriminatory ability of a clinical prediction model. The AUC is often interpreted relative to thresholds, with “good” or “excellent” models defined at 0.7, 0.8 or 0.9. These thresholds may create targets that result in “hacking”, where researchers are motivated to re-analyse their data until they achieve a “good” result.MethodsWe extracted AUC values from PubMed abstracts to look for evidence of hacking. We used histograms of the AUC values in bins of size 0.01 and compared the observed distribution to a smooth distribution from a spline.ResultsThe distribution of 306,888 AUC values showed clear excesses above the thresholds of 0.7, 0.8 and 0.9 and shortfalls below the thresholds.ConclusionsThe AUCs for some models are over-inflated, which risks exposing patients to sub-optimal clinical decision-making. Greater modelling transparency is needed, including published protocols, and data and code sharing.

Open Science Practices in The Journal Human Factors: 2017–2022

Evidence of low replicability, poor reproducibility, and questionable research practices has prompted a broad effort toward transparency and openness in science, termed the open science movement. We examined articles published in HFES’s flagship journal, Human Factors, to gauge the prevalence of open science practices. Empirical articles published in the journal between the years 2017 and 2022 were assessed for evidence of study preregistration, materials sharing, data sharing, and analytic code sharing. The majority of articles made no mention of any open science practices. The most common practices adopted were data and code sharing, though these were reported in no more than 17% of the articles in any year. Journal policies may be worthwhile to better incentivize openness and transparency.

The trustworthiness of the cumulative knowledge in industrial/organizational psychology: The current state of affairs and a path forward

The goal of industrial/organizational (IO) psychology, is to build and organize trustworthy knowledge about people-related phenomena in the workplace. Unfortunately, as with other scientific disciplines, our discipline may be experiencing a “crisis of confidence” stemming from the lack of reproducibility and replicability of many of our field's research findings, which would suggest that much of our research may be untrustworthy. If a scientific discipline's research is deemed untrustworthy, it can have dire consequences, including the withdraw of funding for future research. In this focal article, we review the current state of reproducibility and replicability in IO psychology and related fields. As part of this review, we discuss factors that make it less likely that research findings will be trustworthy, including the prevalence of scientific misconduct, questionable research practices (QRPs), and errors. We then identify some root causes of these issues and provide several potential remedies. In particular, we highlight the need for improved research methods and statistics training as well as a re-alignment of the incentive structure in academia. To accomplish this, we advocate for changes in the reward structure, improvements to the peer review process, and the implementation of open science practices. Overall, addressing the current “crisis of confidence” in IO psychology requires individual researchers, academic institutions, and publishers to embrace system-wide change.

Research anomalies in criminology: How serious? How extensive over time? And who was responsible?

ABSTRACT A variety of ways to detect questionable research practices in small sample social science surveys have been discussed by a variety of authors. However, some of those approaches (e.g., GRIM test, SPRITE test) do not work well for results obtained from larger samples. Here several approaches for detecting anomalies in larger samples are presented and illustrated by an analysis of 78 journal articles in the area of criminology, 59 by Dr. Eric Stewart, published since 1998 with similar methods and/or authors. Of all 59 articles, 28 (47.5%, p < .001, d = 0.94) had two or more major anomalies compared to none of the 19 control group articles. It was also found that the larger the role of Dr. Stewart in article authorship, the greater the number of anomalies detected (p < .001, d = 1.01) while for his coauthors, there were few significant relationships between their roles and total anomalies. Our results demonstrate that extensive problematic results can remain undetected for decades despite several levels of peer review and other scientific controls; however, use of two types of control groups and the use of statistical methods for measuring and evaluating anomalies can improve detection.

Overcoming the “Ostrich Effect”: A Narrative Review on the Incentives and Consequences of Questionable Research Practices in Kinesiology

Increasing transparency and openness in science is an ongoing endeavor, one that has stimulated self-reflection and reform in many fields. However, kinesiology and its related disciplines are among those exhibiting an “ostrich effect” and a reluctance to acknowledge their methodological shortcomings. Notwithstanding several high-profile cases of scientific misconduct, scholars in the field are frequently engaged in questionable research practices (QRPs) such as biased experimental designs, inappropriate statistics, and dishonest/inexplicit reporting. To advance their careers, researchers are also “gaming the system” by manipulating citation metrics and publishing in predatory and/or pay-to-publish journals that lack robust peer review. The consequences of QRPs in the discipline may be profound: from increasing the false positivity rate to eroding public trust in the very institutions tasked with informing public health policy. But what are the incentives underpinning misconduct and QRPs? And what are the solutions? This narrative review is a consciousness raiser that explores (a) the manifestations of QRPs in kinesiology; (b) the excessive publication pressures, funding pressures, and performance incentives that are likely responsible; and (c) possible solutions for reform.

A critical assessment of sparse PCA (research): why (one should acknowledge that) weights are not loadings

Principal component analysis (PCA) is an important tool for analyzing large collections of variables. It functions both as a pre-processing tool to summarize many variables into components and as a method to reveal structure in data. Different coefficients play a central role in these two uses. One focuses on the weights when the goal is summarization, while one inspects the loadings if the goal is to reveal structure. It is well known that the solutions to the two approaches can be found by singular value decomposition; weights, loadings, and right singular vectors are mathematically equivalent. What is often overlooked, is that they are no longer equivalent in the setting of sparse PCA methods which induce zeros either in the weights or the loadings. The lack of awareness for this difference has led to questionable research practices in sparse PCA. First, in simulation studies data is generated mostly based only on structures with sparse singular vectors or sparse loadings, neglecting the structure with sparse weights. Second, reported results represent local optima as the iterative routines are often initiated with the right singular vectors. In this paper we critically re-assess sparse PCA methods by also including data generating schemes characterized by sparse weights and different initialization strategies. The results show that relying on commonly used data generating models can lead to over-optimistic conclusions. They also highlight the impact of choice between sparse weights versus sparse loadings methods and the initialization strategies. The practical consequences of this choice are illustrated with empirical datasets.

A toolbox to evaluate the trustworthiness of published findings

During the past few years, researchers have criticized their professions for providing an entry point for false-positive results arising from publication bias and questionable research practices such as p-hacking (i.e., selectively reporting analyses that yield a p-value below 5 %). Researchers are advocating replication studies and the implementation of open-science practices, like preregistration, in order to identify trustworthy effects. Nevertheless, because such consumer research developments are still emerging, most prior research findings have not been replicated, leaving researchers in the dark as to whether a line of research or a particular effect is trustworthy. We tackle this problem by providing a toolbox containing multiple heuristics to identify data patterns that might, from the information provided in published articles, indicate publication bias and p-hacking. Our toolbox is an easy-to-use instrument with which to initially assess a given set of findings.

Quantifying, and correcting for, the impact of questionable research practices on false discovery rates in psychological science

Large-scale replication failures have shaken confidence in the social sciences, psychology in particular. Most researchers acknowledge the problem, yet there is widespread debate about the causes and solutions. Using “big data,” the current project demonstrates that unintended consequences of three common questionable research practices (retaining pilot data, adding data after checking for significance, and not publishing null findings) can explain the lion’s share of the replication failures. A massive dataset was randomized to create a true null effect between two conditions, and then these three questionable research practices were applied. They produced false discovery rates far greater than 5% (the generally accepted rate), and were strong enough to obscure, or even reverse, the direction of real effects. These demonstrations suggest that much of the replication crisis might be explained by simple, misguided experimental choices. This approach also produces empirically-based statistical corrections to account for these practices when they are unavoidable, providing a viable path forward.

The Scientific Status of Acceptance and Commitment Therapy: An Analysis From the Philosophy of Science

How good is the science in the Acceptance and Commitment Therapy (ACT) program? This article examines ACT philosophy, theory, and research on five dimensions: (1) the quality of its meta-science; (2) the clarity of its constructs; (3) the psychometrics of its principal measures; (4) the adequacy of its account of values; and (5) the quality of its research. Significant problems are found in each dimension, and suggestions for improvements are offered. ACT aligns with a Machiavellianism that is problematic in accurately describing these commitments and constituting a meta-stance that permits problematic values to be embraced. Relatedly, there is evidence of a positive bias in ACT research that has been ignored methodologically and in summaries of ACT. These problems justify significant skepticism regarding any claims from the science associated with ACT. Avoiding questionable research practices, psychometrically problematic measures, and research designs that weaken valid causal inference is recommended. Finally, an increased commitment to open science, intellectual humility, and severe testing is recommended.

On the frequency, prevalence, and perceived severity of questionable research practices

Questionable Research Practices (QRPs) fall in the gray zone between responsible research conduct and absolute misconduct (e.g., falsification and fabrication). Whereas other fields such as medicine have a long tradition of discussing and studying QRPs, there has been very limited focus on this topic in quantitative humanities research, and in applied linguistics specifically. Drawing on a community-generated list of quantitative humanities-specific QRPs, the present study investigates the self-reported frequency, prevalence, and perceived severity of QRPs among researchers in the US and Sweden. We also explored relationships between frequency of QRP engagement and researcher background factors, such as years since Ph.D. and publication rate. With regard to prevalence, the results showed that 96% of the respondents reported having used one or more of the practices listed. The most prevalent item was also the one that occurred with the highest frequency and the one that was reported as the least severe (‘Presenting the same presentation at multiple conferences’). Overall, there was a strong negative correlation between frequency and severity (ρ < -.77) of QRPs, suggesting that it is uncommon for researchers to engage in an activity considered to be severe. With this exploratory study, we hope to contribute to an open and respectful discussion about QRPs.

"Conducted Properly, Published Incorrectly": The Evolving Status of Gel Electrophoresis Images Along Instrumental Transformations in Times of Reproducibility Crisis.

For the last ten years, within molecular life sciences, the reproducibility crisis discourse has been embodied as a crisis of trust in scientific images. Beyond the contentious perception of "questionable research practices" associated with a digital turn in the production of images, this paper highlights the transformations of gel electrophoresis as a family of experimental techniques. Our aim is to analyze the evolving epistemic status of generated images and its connection with a crisis of trust in images within that field. From the 1980s to the 2000s, we identify two key innovations (precast gels and gel docs) leading to a "two-tiered" gel electrophoresis with different standardization procedures, different epistemic statuses of the produced images and different ways of generating (dis)trust in images. The first tier, exemplified by differential gel electrophoresis (DIGE), is characterized by specialized devices processing images as quantitative data. The second tier, exemplified by polyacrylamide gel electrophoresis (PAGE), is described as a routine technique making use of image as qualitative "virtual witnessing." The difference between these two tiers is particularly apparent in the ways images are processed, even though both tiers involve image digitization. Our account thus highlights different views on reproducibility within the two tiers. Comparability of images is insisted upon in the first tier while traceability is expected in the second tier. It is striking that these differences occur not only within the same scientific field, but even within the same family of experimental techniques. In the second tier, digitization entails distrust, whereas it implies a collective sentiment of trust in the first tier.

Don't hate the player, hate the game: Realigning incentive structures to promote robust science and better scientific practices in marketing

Marketing research draws heavily from the methods of Psychology and Economics, where questionable research practices (QRPs) are evident and replication rates are low; it is thus likely that QRPs and low replicability rates are pervasive in Marketing. Here, we review proximate and systemic issues that contribute to this state, and survey prominent solutions currently available to researchers for combating QRPs, namely, preregistration, registered reports, open-science practices, and multi-verse analysis. We argue that the core of replicability issues is rooted in a misalignment of academic incentives structures, rather than placing blame on individual researchers, and we make systemic recommendations for a pathway forward towards more robust and replicable marketing science.

Responsible research practices could be more strongly endorsed by Australian university codes of research conduct

BackgroundThis study aimed to investigate how strongly Australian university codes of research conduct endorse responsible research practices.MethodsCodes of research conduct from 25 Australian universities active in health and medical research were obtained from public websites, and audited against 19 questions to assess how strongly they (1) defined research integrity, research quality, and research misconduct, (2) required research to be approved by an appropriate ethics committee, (3) endorsed 9 responsible research practices, and (4) discouraged 5 questionable research practices.ResultsOverall, a median of 10 (IQR 9 to 12) of 19 practices covered in the questions were mentioned, weakly endorsed, or strongly endorsed. Five to 8 of 9 responsible research practices were mentioned, weakly, or strongly endorsed, and 3 questionable research practices were discouraged. Results are stratified by Group of Eight (n = 8) and other (n = 17) universities. Specifically, (1) 6 (75%) Group of Eight and 11 (65%) other codes of research conduct defined research integrity, 4 (50%) and 8 (47%) defined research quality, and 7 (88%) and 16 (94%) defined research misconduct. (2) All codes required ethics approval for human and animal research. (3) All codes required conflicts of interest to be declared, but there was variability in how strongly other research practices were endorsed. The most commonly endorsed practices were ensuring researcher training in research integrity [8 (100%) and 16 (94%)] and making study data publicly available [6 (75%) and 12 (71%)]. The least commonly endorsed practices were making analysis code publicly available [0 (0%) and 0 (0%)] and registering analysis protocols [0 (0%) and 1 (6%)]. (4) Most codes discouraged fabricating data [5 (63%) and 15 (88%)], selectively deleting or modifying data [5 (63%) and 15 (88%)], and selective reporting of results [3 (38%) and 15 (88%)]. No codes discouraged p-hacking or hypothesising after results are known.ConclusionsResponsible research practices could be more strongly endorsed by Australian university codes of research conduct. Our findings may not be generalisable to smaller universities, or those not active in health and medical research.

We must improve conditions and options for Australian ECRs.

Early-career researchers in Australia report dissatisfaction, bullying and questionable research practices. We discuss how this may contribute to the replication crisis and suggest local and international strategies to improve the industry.

Development of a Concept Inventory on Open and Transparent Research Practices

Over the past decade psychology researchers have begun adopting practices that promote openness and transparency. While these practices are increasingly reflected in undergraduate psychology curricula, pedagogical research has not systematically examined whether instruction on open science practices improves students’ conceptual understanding of research methods. We developed the Open Science Concept Inventory (OSCI) to evaluate the impact of integrating open science practices into research methods courses. First, we created a set of hypothetical dilemmas related to a range of open science concepts and elicited open-ended responses from undergraduates (N = 64, Study 1). Based on the responses, we created a 40-item multiple-choice questionnaire, which we administered to a new group of participants (N = 262, Study 2) and used item response theory to select 33 items for the final OSCI. Finally, in two implementation rounds across two semesters (Study 3, total N = 37), we evaluated students’ learning gains with the OSCI in a pre-test/post-test design. The implementation rounds involved new materials on open science for a psychology research methods course, including video lectures that situated questionable research practices in the current norms of science and introduced emerging solutions. After excluding extremely fast survey responders, an exploratory analysis showed learning gains among students who expended appropriate effort when completing the OSCI. By systematically evaluating a tool that is easily integrated into existing curricula, we aim to facilitate the adoption of open science practices in undergraduate instruction and the assessment of students’ conceptual foundations for conducting robust and transparent research.

Nature and causes of questionable research practice and research misconduct from a philosophy of science perspective

ABSTRACT Misconduct in science is often viewed and analyzed through the lenses of normative ethics and moral philosophy. However, notions and methods in the philosophy of science could also provide rather penetrative explanatory insights into the nature and causes of scientific misconduct. A brief illustration in this regard, using as examples the widely popular Popperian falsification and the Kuhnian scientific paradigm, is provided. In multiple areas of scientific research, failure to seek falsification in a Popperian manner constitutes a questionable research practice and could lead to “falsification” in the context of research misconduct. On the other hand, scientific misconduct is often facilitated by its perpetrators using the familiarity, expectations and confines of a Kuhnian paradigm to blend in fabricated data/results. A rudimentary application of these philosophical notions could be useful in our understanding of the nature and cause of research misconduct, and facilitate mitigation of the latter through educational means.

Changes in methodological study characteristics in psychology between 2010-2021.

In 2015, the Open Science Collaboration repeated a series of 100 psychological experiments. Since a considerable part of these replications could not confirm the original effects and some of them pointed in the opposite direction, psychological research is said to lack reproducibility. Several general criticisms can explain this finding, such as the standardized use of undirected nil-null hypothesis tests, samples being too small and selective, lack of corrections for multiple testing, but also some widespread questionable research practices and incentives to publish positive results only. A selection of 57,909 articles from 12 renowned journals is processed with the JATSdecoder software to analyze the extent to which several empirical research practices in psychology have changed over the past 12 years. To identify journal- and time-specific changes, the relative use of statistics based on p-values, the number of reported p-values per paper, the relative use of confidence intervals, directed tests, power analysis, Bayesian procedures, non-standard α levels, correction procedures for multiple testing, and median sample sizes are analyzed for articles published between 2010 and 2015 and after 2015, and in more detail for every included journal and year of publication. In addition, the origin of authorships is analyzed over time. Compared to articles that were published in and before 2015, the median number of reported p-values per article has decreased from 14 to 12, whereas the median proportion of significant p-values per article remained constant at 69%. While reports of effect sizes and confidence intervals have increased, the α level is usually set to the default value of .05. The use of corrections for multiple testing has decreased. Although uncommon in each case (4% in total), directed testing is used less frequently, while Bayesian inference has become more common after 2015. The overall median estimated sample size has increased from 105 to 190.

Questionable Metascience Practices

Questionable research practices may reduce the public’s trust in science. The present article considers some questionable metascience practices (QMPs) that may threaten scientists’ trust in metascience. A QMP is a research practice, assumption, or perspective that has been questioned by several commentators as being potentially problematic for the credibility of metascience and/or the science reform movement. The present article reviews 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science. Specifically, the following QMPs are considered: (1) rejecting or ignoring self-criticism; (2) a fast ‘n’ bropen scientific criticism style; (3) overplaying the role of replication in science; (4) assuming a replication rate is “too low” without specifying an “acceptable” rate; (5) an unacknowledged metabias towards explaining the replication crisis in terms of researcher bias; (6) assuming that researcher bias can be reduced; (7) devaluing exploratory results as more “tentative” than confirmatory results; (8) presuming that QRPs are problematic research practices; (9) focusing on knowledge accumulation as an index of scientific progress; and (10) focusing on specific scientific methods. It is stressed that only some metascientists engage in some QMPs some of the time, and that these QMPs may not always be problematic. Research is required to estimate the prevalence and impact of QMPs. In the meantime, QMPs should be viewed as invitations to ask “questions” about how we go about doing metascience rather than as grounds for mistrusting the credibility of metascience.