Questionable Research Practices Research Articles

Investigating the links between questionable research practices, scientific norms and organisational culture

BackgroundThis study investigates the determinants of engagement in questionable research practices (QRPs), focusing on both individual-level factors (such as scholarly field, commitment to scientific norms, gender, contract type, and career stage) and institution-level factors (including industry type, researchers' perceptions of their research culture, and awareness of institutional policies on research integrity).MethodsUsing a multi-level modelling approach, we analyse data from an international survey of researchers working across disciplinary fields to estimate the effect of these factors on QRP engagement.ResultsOur findings indicate that contract type, career stage, academic field, adherence to scientific norms and gender significantly predict QRP engagement. At the institution level, factors such as being outside of a collegial culture and experiencing harmful publication pressure, and the presence of safeguards against integrity breaches have small associations. Only a minimal amount of variance in QRP engagement is attributable to differences between institutions and countries.ConclusionsWe discuss the implications of these findings for developing effective interventions to reduce QRPs, highlighting the importance of addressing both individual and institutional factors in efforts to foster research integrity.

Discontinuous Distribution of Test Statistics Around Significance Thresholds in Empirical Accounting Studies

ABSTRACTExamining test statistics from articles in six leading accounting journals, we detect discontinuities in their distributions around conventional significance thresholds (p‐values of 0.05 and 0.01) and find an unusual abundance of test statistics that are just significant. Further analysis reveals that these discontinuities are more prominent in studies with smaller samples and are more salient in experimental than in archival studies. The discontinuity discrepancy between experimental and archival studies relates to several proxies for researcher degrees of freedom. Nevertheless, this evidence does not imply that experimental research is more prone to questionable research practices than archival studies. Overall, our findings speak to the concern of whether accounting researchers could exercise undisclosed discretion to obtain and report statistically significant results. Based on our results, a healthy skepticism of some just‐significant test statistics is warranted.

Developing Surveys on Questionable Research Practices: Four Challenging Design Problems

AbstractThe exposure of scientific scandals and the increase of dubious research practices have generated a stream of studies on Questionable Research Practices (QRPs), such as failure to acknowledge co-authors, selective presentation of findings, or removal of data not supporting desired outcomes. In contrast to high-profile fraud cases, QRPs can be investigated using quantitative, survey-based methods. However, several design issues remain to be solved. This paper starts with a review of four problems in the QRP research: the problem of precision and prevalence, the problem of social desirability bias, the problem of incomplete coverage, and the problem of controversiality, sensitivity and missing responses. Various ways to handle these problems are discussed based on a case study of the design of a large, cross-field QRP survey in the social and medical sciences in Sweden. The paper describes the key steps in the design process, including technical and cognitive testing and repeated test versions to arrive at reliable survey items on the prevalence of QRPs and hypothesized associated factors in the organizational and normative environments. Partial solutions to the four problems are assessed, unresolved issues are discussed, and tradeoffs that resist simple solutions are articulated. The paper ends with a call for systematic comparisons of survey designs and item quality to build a much-needed cumulative knowledge trajectory in the field of integrity studies.

Is something rotten in the state of Denmark? Cross-national evidence for widespread involvement but not systematic use of questionable research practices across all fields of research.

Questionable research practices (QRP) are believed to be widespread, but empirical assessments are generally restricted to a few types of practices. Furthermore, conceptual confusion is rife with use and prevalence of QRPs often being confused as the same quantity. We present the hitherto most comprehensive study examining QRPs across scholarly fields and knowledge production modes. We survey perception, use, prevalence and predictors of QRPs among 3,402 researchers in Denmark and 1,307 in the UK, USA, Croatia and Austria. Results reveal remarkably similar response patterns among Danish and international respondents (τ = 0.85). Self-reported use indicates whether respondents have used a QRP in recent publications. 9 out of 10 respondents admitted using at least one QRP. Median use is three out of nine QRP items. Self-reported prevalence reflects the frequency of use. On average, prevalence rates were roughly three times lower compared to self-reported use. Findings indicated that the perceived social acceptability of QRPs influenced self-report patterns. Results suggest that most researchers use different types of QRPs within a restricted time period. The prevalence estimates, however, do not suggest outright systematic use of specific QRPs. Perceived pressure was the strongest systemic predictor for prevalence. Conversely, more local attention to research cultures and academic age was negatively related to prevalence. Finally, the personality traits conscientiousness and, to a lesser degree, agreeableness were also inversely associated with self-reported prevalence. Findings suggest that explanations for engagement with QRPs are not only attributable to systemic factors, as hitherto suggested, but a complicated mixture of experience, systemic and individual factors, and motivated reasoning.

Understanding QRPs and Blatant Misconduct in Applied Linguistics Research: A Book Review of Addressing Questionable Research Practices in Applied Linguistics: A Practical Guide

his book explores the ethical gray area of Questionable Research Practices (QRPs), including its similarities and differences (Butler et al., 2017; Ravn & Sørensen, 2021). It lies between responsible research conduct and clear misconduct like falsification, fabrication, plagiarism. It helps Applied Linguistics (henceforth AL) researchers to carry out ethical research which addresses the ambiguity and debate surrounding certain practices, such as data sharing and outlier management. They are often not well covered in traditional research ethics training. There is a need for AL researcher to identify the good, the bad, and the ugly in the research practices (Banks et al., 2016). To enhance understanding and ethical decision-making, the book provides a taxonomy of 58 QRPs, lesson plans, suggested readings, and discussion scenarios for use in research methods courses and workshops.

Investigating questionable research practices among Iranian applied linguists: Prevalence, severity, and the role of artificial intelligence tools

The concept of Questionable Research Practices (QRPs) has recently gained recognition in the field of applied linguistics. This field, which previously differentiated between macro- and micro-ethics, is now grappling with QRPs that originate from the core of microethics. Despite the importance of studying QRPs, the applied linguistics community has not paid due attention to them. The present mixed-methods study investigates the frequency and severity of QRPs among Iranian applied linguists. Additionally, it explores QRPs related to artificial intelligence (AI) tools, a topic not previously examined, using researcher-constructed scenarios. A total of 160 Iranian applied linguists participated in the study, completing a 48-item questionnaire adapted from Larsson et al. (2023). Fifteen participants also took part in follow-up scenario-based interviews with regard to AI-related QRPs. The most common QRPs identified were the selection of variables for convenience and P-hacking, while the most severe ones were plagiarism-like behaviours and authorship issues. Lack of AI-related regulations, excessive use of AI as a threat, AI disclosure dilemma, and Al deficiencies and human oversight were the themes representing AI-related QRPs. The study concludes that QRPs among Iranian applied linguists mainly stem from inadequate ethics training, limited statistical knowledge, and a lack of AI guidelines.

Conflicting Results and Statistical Malleability: Embracing Pluralism of Empirical Results

Abstract Conflicting results undermine making inferences from the empirical literature. So far, the replication crisis is mainly seen as resulting from honest errors and questionable research practices such as p-hacking or the base-rate fallacy. I discuss the malleability (researcher degrees of freedom) of quantitative research and argue that conflicting results can emerge from two studies using different but plausible designs (e.g., eligibility criteria, operationalization of concepts, outcome measures) and statistical methods. I also explore how the choices regarding study design and statistical techniques bias results in a way that makes them more or less relevant for a given policy or clinical question.

Protocol for a meta-analysis of stereotype threat in African Americans.

Stereotype threat is a well-known construct in psychology wherein individuals who belong to a negatively stereotyped demographic group underperform on cognitive or academic tasks due to the detrimental effects of a stereotype. Many psychologists have suggested that stereotype threat may be one of the reasons that some demographic groups are underrepresented in advanced academic programs and STEM fields. However, others have raised concerns about the quality of the stereotype threat research, suggesting that its apparent effects are inflated and that the phenomenon may be an illusion of questionable research practices and publication bias. The purpose of this proposed meta-analysis is to evaluate the existence of stereotype threat by (1) identifying the average effect size of stereotype threat studies in different types of studies, (2), investigating whether publication bias and p-hacking are present in the empirical research on stereotype threat, (3) testing for the influence of theoretical and methodological moderators, (4) assessing the overall quality of the research on stereotype threat, (5) and identifying the average effect in the methodologically strongest studies. This meta-analysis will be limited to studies that report data from African Americans because this population is a theoretically important group in stereotype threat research, and the size of score gaps between the African American and non-stereotyped populations in the United States should make the stereotype threat effect easiest to detect.

The practicalities of a partial lottery to allocate research funding

Abstract This article explores the practical dimension of organizing a partial lottery as an alternative funding allocation mechanism in academic institutions. Following an assessment of the recent literature that presents criticism of traditional ranking-based system derived from academic peer assessment and the need for improved decision-making procedures in research funding, we highlight four key arguments for organizing a partial lottery: improved decision quality (more fair, transparent, and divers), an alternative to ranking-based selection, prevention of questionable research practices, and increased efficiency. Based on our observations from a lottery-based ‘seed money call’ at Tilburg University, we provide insights into the organization and implementation of a lottery procedure, including the criteria for proposal eligibility, the administrative process, and the weighting mechanism based on budget categories. We furthermore share six lessons we learned that might be useful for future implementation in other institutional contexts, emphasizing the need for transparency, external supervision, and pre-registration of the lottery procedure to ensure trust and minimize gaming. As such, we not only contribute to the ongoing conceptual discussion about improving funding allocation mechanisms in academia. Our article also offers an account of how partial lotteries could be put into practice.

Stage 2 Registered Report: Anomalous perception in a Ganzfeld condition - A meta-analysis of more than 40 years investigation

This meta-analysis is an investigation into anomalous perception (i.e., conscious identification of information without any conventional sensorial means). The technique used for eliciting an effect is the Ganzfeld condition (a form of sensory homogenization that eliminates distracting peripheral noise). The database consists of studies published between January 1974 and December 2020 inclusive. The overall effect size estimated both with a frequentist and a Bayesian random-effect model, were in close agreement yielding an effect size of approximately .08 (.04 -.12). This result passed four publication bias tests and seems not contaminated by questionable research practices. Trend analysis carried out with a cumulative meta-analysis and a meta-regression model with year of publication as a covariate, did not indicate sign of decline of this effect size. The moderators’ analyses show that the selected participants’ effect size was almost three-times that obtained by non-selected participants and that tasks that simulate telepathic communication show a two-fold effect size for tasks requiring the participants to guess a target. The Stage 1 Registered Report can be accessed here: https://doi.org/10.12688/f1000research.24868.3

Paclitaxel - a Product of Fungal Secondary Metabolism or an Artefact?

Taxol (common name: paclitaxel) is an extremely important component of drugs for the treatment of various cancers. Thirty years after the discovery of its effectiveness, a metabolic precursor of Taxol (10-deacetylbaccatin III) is still primarily extracted from needles of European yew trees. In order to meet the considerable demand, hopes were pinned on the possibilities of biotechnological production from the very beginning. In 1993, as if by chance, Taxol was supposedly discovered in fungi that grow endobiotically in yew trees. This finding aroused hopes of biotechnological use to produce fungal Taxol in large quantities in fermenters. It never came to that. Instead, a confusing flood of publications emerged that claimed to have detected Taxol in more and more eukaryotic and even prokaryotic species. However, researchers never reproduced these rather puzzling results, and they could certainly not be applied on an industrial scale. This paper will show that some of the misguided approaches were apparently based on a seemingly careless handling of sparse evidence and on at least questionable publications. Apparently, the desired gold rush of commercial exploitation was seductive. Scientific skepticism as an indispensable core of good scientific practice was often neglected, and the peer review process has not exerted its corrective effect. Self-critical reflection and more healthy skepticism could help to reduce the risk of such aberrations in drug development. This article uses this case study as a striking example to show what can be learned from the Taxol case in terms of research ethics and the avoidance of questionable research practices.

Values, bias and replicability

The Value-free ideal of science (VFI) is a view that claims that scientists should not use non-epistemic values when they are justifying their hypotheses, and is widely considered to be obsolete in the philosophy of science. I will defend the ideal by demonstrating that acceptance of non-epistemic values, prohibited by VFI, necessitates legitimizing certain problematic scientific practices. Such practices, including biased methodological decisions or Questionable Research Practices (QRP), significantly contribute to the Replication Crisis. I will argue that the realizability of VFI is not a necessary condition for its validity. Then, I will show how some of the prominent proposals of value-laden science legitimize problematic scientific practices, provide real-world examples, and generalize the argument. Finally, I will show how value-laden methodological decisions contribute to the Replicability Crisis and discuss two strategies for realizing VFI.

Researchers Training Researchers: Ethics Training in Quantitative Applied Linguistics

AbstractThis forum piece reports on a brief survey of current quantitative research ethics training materials in Applied Linguistics (AL). This survey was conducted as a step towards an increased understanding of the ways in which we, as a field, train students and researchers to conduct quantitative research ethically. The survey was carried out in the process of creating research ethics training materials as part of a grant‐funded project related to questionable research practices (QRPs). Through manual and computer‐assisted searches in twenty‐four textbooks and twenty‐three course syllabi from the past five years, three themes emerged: (1) research ethics is most often conceptualized in terms of IRB‐related topics, (2) when ‘ethical gray‐zone’ issues are mentioned, the focus is primarily on methodological transparency, data sharing/open science, and selecting the appropriate research design and statistical tests, and (3) materials tend to address the topic of research ethics in a single section or chapter, or in a single day or week of a course. Against this background, we provide three recommendations that our field can implement to provide more robust and thorough research ethics training for students and researchers in TESOL and Applied Linguistics: (1) expand our conceptualization of research ethics to include ethical gray‐zone issues, (2) be thorough and explicit in our discussion of ethical issues related to research decisions, and (3) incorporate research ethics as a recurring theme throughout textbooks and courses.

The psychological reality of the learned “p < .05” boundary

The .05 boundary within Null Hypothesis Statistical Testing (NHST) “has made a lot of people very angry and been widely regarded as a bad move” (to quote Douglas Adams). Here, we move past meta-scientific arguments and ask an empirical question: What is the psychological standing of the .05 boundary for statistical significance? We find that graduate students in the psychological sciences show a boundary effect when relating p-values across .05. We propose this psychological boundary is learned through statistical training in NHST and reading a scientific literature replete with “statistical significance”. Consistent with this proposal, undergraduates do not show the same sensitivity to the .05 boundary. Additionally, the size of a graduate student’s boundary effect is not associated with their explicit endorsement of questionable research practices. These findings suggest that training creates distortions in initial processing of p-values, but these might be dampened through scientific processes operating over longer timescales.

How variable are the classic ERP effects during sentence processing? A systematic resampling analysis of the N400 and P600 effects

Although event-related potential (ERP) research on language processing has capitalized on key, theoretically influential components such as the N400 and P600, their measurement properties—especially the variability in their temporal and spatial parameters—have rarely been examined. The current study examined the measurement properties of the N400 and P600 effects elicited by semantic and syntactic anomalies, respectively, during sentence processing. We used a bootstrap resampling procedure to randomly draw many thousands of resamples varying in sample size and stimulus count from a larger sample of 187 participants and 40 stimulus sentences of each type per condition. Our resampling investigation focused on three issues: (a) statistical power; (b) variability in the magnitudes of the effects; and (c) variability in the temporal and spatial profiles of the effects. At the level of grand averages, the N400 and P600 effects were both robust and substantial. However, across resamples, there was a high degree of variability in effect magnitudes, onset times, and scalp distributions, which may be greater than is currently appreciated in the literature, especially for the P600 effects. These results provide a useful basis for designing future studies using these two well-established ERP components. At the same time, the results also highlight challenges that need to be addressed in future research (e.g., how best to analyze the ERP data without engaging in such questionable research practices as p-hacking).

Knowledge, Attitude, and Practices About Research Integrity and Scientific Misconduct Among the Faculty and Medical Postgraduates Working in Medical Colleges in North Karnataka and Central India: A Cross-Sectional Online Survey.

Introduction Research integrity is an active adherence to the ethical principles and professional standards essential for the responsible practice of research. Research or scientific misconduct stands like child abuse today. The survey of National Institutes of Health (NIH)-funded scientists calculated an absolute minimum of 2325 incidents of scientific misconduct per year. A report has also shown that Iran (6.60), India (5.68), Turkey (5.38), South Korea (3.59), and China (2.00) had higher ratios of publication misconduct to distrust data or interpretations than other countries. Hence, to determine the knowledge, attitude, and practices (KAPs) of the research integrity/scientific misconduct among the faculty and postgraduates working in the medical colleges in North Karnataka (NK) and Central India (CI), this study has been carried out. Methods It is a web-based, cross-sectional study carried out with the use of Google Forms (Google, Mountain View, California). A pretested, unstructured questionnaire consisting of 25 questions was posted in the way of a link to the faculty and postgraduates working in various disciplines within the colleges of NK and CI either by using an e-mail or other social platforms like WhatsApp. Institutional Ethics Committee approval was obtained in both regions before conducting the survey. Results A total of 146 participants responded to the e-questionnaire posted to them. Participants from CI displayed better awareness in several areas compared to NK. Citing articles and/ or materials that have not been read is the common questionable research practice (QRP) they have come across, as mentioned by participants in both groups. Discussion The study reveals a moderate level of knowledge and variable attitudes toward research integrity. The "publish or perish" culture is a major contributor to misconduct. Training and awareness programs are needed to enhance ethical research practices. Conclusion This study highlights the need for improved education and policy implementation to uphold research integrity in medical colleges, emphasizing the role of academic culture in shaping ethical research practices.

Lost in Statistics

The present paper investigates one aspect of questionable research practices relating to Arabic L1 learners of foreign languages, namely the use of statistics. The objective of the paper is to argue that reproducible research requires adopting wise practices in linguistics and that the excessive focus on quantification does not seem to serve this purpose. Statistical significance tests in quantitative research are routinely used in linguistic inquiry as well as language teaching and learning studies with a view to supporting the relevant explanatory insights in linguistics. In this article, I will expose the misuse of statistics by doctoral students in English departments of Morocco working on Arabic L1 learners&amp;rsquo; data, by highlighting some practices that are at odds with international good practices in academic research in linguistics. I will take stock of the current questionable practices in this regard to dispel some of the misunderstanding about the use of statistics which is now gaining grounds lest this becomes an orthodoxy. I will argue that research on Arabic L1 learners&amp;rsquo; data should be focused more on exploration and discovery, as well as the validation of epistemological insights than on mere descriptive quantification geared to hypothesis verification. These areas of focus constitute the crux of academic research in linguistics, but they seem to be lost in statistics in doctoral students&amp;rsquo; theses. Recommendations and solutions are provided for enhancing transparency and improving reproducibility of doctoral research outcomes to advance theory building and the delivery of new research lines in linguistics as well as to avoid the risk of research waste, in line with the requirements of open science.

Registered reports and associated benefits for agricultural economics

Abstract The distribution of published scientific evidence is biased. There is accumulating evidence of the frequent use of questionable publishing practices and questionable research practices, and current solutions are limited. This has important undesirable consequences for policy-oriented areas such as agricultural economics. In this article, we discuss a potential solution: Registered Reports. We first provide an overview of the Registered Report publication format. We demonstrate how Registered Reports outperform pre-registration, while also efficiently contributing to reducing questionable research practices, canceling questionable publishing practices, and providing more reliable results that can inform evidence-based policymaking. Second, we present a benchmark of Registered Reports in economics today and discuss potential barriers using survey data from economic journal editors. Lastly, we highlight the importance of developing the Registered Report publication format in agricultural economics.

Minimizing Questionable Research Practices – The Role of Norms, Counter Norms, and Micro-Organizational Ethics Discussion

AbstractBreaches of research integrity have gained considerable attention due to high-profile scandals involving questionable research practices by reputable scientists. These practices include plagiarism, manipulation of authorship, biased presentation of findings and misleading reports of significance. To combat such practices, policymakers tend to rely on top-down measures, mandatory ethics training and stricter regulation, despite limited evidence of their effectiveness. In this study, we investigate the occurrence and underlying factors of questionable research practices (QRPs) through an original survey of 3,005 social and medical researchers at Swedish universities. By comparing the role of the organizational culture, researchers´ norms and counter norms, and individual motivation, the study reveals that the counter norm of Biasedness—the opposite of universalism and skepticism—is the overall most important factor. Thus, Biasedness was related to 40–60% of the prevalence of the questionable practices. The analysis also reveals the contradictory impact of other elements in the organizational environment. Internal competition was positively associated with QRP prevalence, while group-level ethics discussions consistently displayed a negative association with such practices. Furthermore, in the present study items covering ethics training and policies have only a marginal impact on the prevalence of these practices. The organizational climate and normative environment have a far greater influence. Based on these findings, it is suggested that academic leaders should prioritize the creation and maintenance of an open and unbiased research environment, foster a collaborative and collegial climate, and promote bottom-up ethics discussions within and between research groups.

Hyper-ambition and the Replication Crisis: Why Measures to Promote Research Integrity can Falter

AbstractThis paper introduces the concept of ‘hyper-ambition’ in academia as a contributing factor to what has been termed a ‘replication crisis’ across some sciences. The replication crisis is an umbrella term that covers a range of ‘questionable research practices’, from sloppy reporting to fraud. There are already many proposals to address questionable research practices, some of which focus on the values, norms, and motivations of researchers and institutes, and suggest measures to promote research integrity. Yet it is not easy to promote integrity in hyper-competitive academic environments that value high levels of ambition. I argue that in such contexts, it is as likely that a kind of hyper-ambition is fostered that (inadvertently or otherwise) prioritises individual success above all, including to the detriment of scientific quality. In addition, efforts to promote values like integrity falter because they rely on sufficient uniformity in motivations or tendencies. Codes and guidance promoting integrity are, however, likely to influence those for whom such values are not optional, while others simply find ways around them. To demonstrate this I offer a thought experiment in which we consider the imaginary working situations of two ordinary academics. I conclude that tackling questionable research practices in the light of the replication crisis requires robust ‘top down’ measures that expect and accommodate a broader range of academic values, motivations, and tendencies, while challenging those that help to promote hyper-ambition.