Questionable Research Practices Research Articles

­Questionable Research Practices and Open Science in Quantitative Criminology

Questionable research practices (QRPs) lead to incorrect research results and contribute to irreproducibility in science. Researchers and institutions have proposed open science practices (OSPs) to improve the detectability of QRPs and the credibility of science. We examine the prevalence of QRPs and OSPs in criminology, and researchers’ opinions of those practices. We administered an anonymous survey to authors of articles published in criminology journals. Respondents self-reported their own use of 10 QRPs and 5 OSPs. They also estimated the prevalence of use by others, and reported their attitudes toward the practices. QRPs and OSPs are both common in quantitative criminology, about as common as they are in other fields. Criminologists who responded to our survey support using QRPs in some circumstances, but are even more supportive of using OSPs. We did not detect a significant relationship between methodological training and either QRP or OSP use. Support for QRPs is negatively and significantly associated with support for OSPs. Perceived prevalence estimates for some practices resembled a uniform distribution, suggesting criminologists have little knowledge of the proportion of researchers that engage in certain questionable practices. Most quantitative criminologists in our sample have used QRPs, and many have used multiple QRPs. Moreover, there was substantial support for QRPs, raising questions about the validity and reproducibility of published criminological research. We found promising levels of OSP use, albeit at levels lagging what researchers endorse. The findings thus suggest that additional reforms are needed to decrease QRP use and increase the use of OSPs.

Reversing Bonferroni.

It is common for conclusions of empirical studies to depend on multiple significant outcomes. This practice may seem reasonable, but it has some unintended effects. In particular, the compound Type I error rate for multiple studies (the likelihood of concluding that an effect exists when it does not) can be much lower than that of the individual studies. This in itself is not a problem since a low Type I error rate is desirable. However, there is also an accompanying drop in power, meaning that the probability of finding support for a true effect is low. Currently, there is no standard statistical method for dealing with the hyper-conservative error rate and accompanying low power that results from investigations requiring multiple significant outcomes. Here, we propose a novel solution to this problem: We show that it is sometimes appropriate to reverse the logic of the classic Bonferroni correction and increase the significance criterion in order to maintain an intended compound Type I error rate across multiple tests. This reverse Bonferroni approach dramatically improves statistical power and encourages careful planning of statistical analyses prior to data collection. To avoid adding to the list of questionable research practices that seem to contaminate some psychological research, we suggest that reverse Bonferroni be restricted to situations where authors pre-register their analysis plans.

Rethinking success, integrity, and culture in research (part 2) \u2014 a multi-actor qualitative study on problems of science

BackgroundResearch misconduct and questionable research practices have been the subject of increasing attention in the past few years. But despite the rich body of research available, few empirical works also include the perspectives of non-researcher stakeholders.MethodsWe conducted semi-structured interviews and focus groups with policy makers, funders, institution leaders, editors or publishers, research integrity office members, research integrity community members, laboratory technicians, researchers, research students, and former-researchers who changed career to inquire on the topics of success, integrity, and responsibilities in science. We used the Flemish biomedical landscape as a baseline to be able to grasp the views of interacting and complementary actors in a system setting.ResultsGiven the breadth of our results, we divided our findings in a two-paper series with the current paper focusing on the problems that affect the integrity and research culture. We first found that different actors have different perspectives on the problems that affect the integrity and culture of research. Problems were either linked to personalities and attitudes, or to the climates in which researchers operate. Elements that were described as essential for success (in the associate paper) were often thought to accentuate the problems of research climates by disrupting research culture and research integrity. Even though all participants agreed that current research climates need to be addressed, participants generally did not feel responsible nor capable of initiating change. Instead, respondents revealed a circle of blame and mistrust between actor groups.ConclusionsOur findings resonate with recent debates, and extrapolate a few action points which might help advance the discussion. First, the research integrity debate must revisit and tackle the way in which researchers are assessed. Second, approaches to promote better science need to address the impact that research climates have on research integrity and research culture rather than to capitalize on individual researchers’ compliance. Finally, inter-actor dialogues and shared decision making must be given priority to ensure that the perspectives of the full research system are captured. Understanding the relations and interdependency between these perspectives is key to be able to address the problems of science.Study registrationhttps://osf.io/33v3m

Reproducibility and replicability crisis: How management compares to psychology and economics – A systematic review of literature

The past decade has been marked by concerns regarding the replicability and reproducibility of published research in the social sciences. Publicized failures to replicate landmark studies, along with high-profile cases of research fraud, have led scholars to reconsider the trustworthiness of both findings and institutionalized research practices. This paper considers two questions: (1) Relative to psychology and economics, what is the state of replication and reproduction research in management? (2) Are the disciplines equally advanced in the use of methods applied to study the replication problem? A systematic literature review identified 67 studies pertinent to these questions. The results indicate that the replication prevalence rate in management studies lies almost exactly between those of psychology and economics, while a high level of variation between management and other business-related disciplines can be noted. Further, similarly to psychology, but unlike economics, the surveys of published replications tend to report high replication success rates for management and other business-related disciplines. However, a comparison with recently obtained results in preregistered multi-study replications in psychology and economics suggests that these rates are almost certainly inflated. Method and data transparency are medium to low, often rendering attempts to reproduce or replicate studies impossible. Finally, the understanding of the replicability problem in management is held back by the underutilization of methods developed in other disciplines. The review also reveals that management, psychology, and economics exhibit strikingly different practices and approaches to replication, despite facing similar incentive structures. Disciplines in which replication and reproduction attempts are rare and which frequently involve authors of the original study in replication attempts lack strong deterrents against questionable research practices; thus, they are less likely to deliver replicable results.

A survey of early-career researchers in Australia.

Early-career researchers (ECRs) make up a large portion of the academic workforce and their experiences often reflect the wider culture of the research system. Here we surveyed 658 ECRs working in Australia to better understand the needs and challenges faced by this community. Although most respondents indicated a 'love of science', many also expressed an intention to leave their research position. The responses highlight how job insecurity, workplace culture, mentorship and 'questionable research practices' are impacting the job satisfaction of ECRs and potentially compromising science in Australia. We also make recommendations for addressing some of these concerns.

The "Goldilocks Zone": (Too) many confidence intervals in tests of mediation just exclude zero.

Questionable research practices (QRPs) can occur whenever one result is favored over another, and tests of mediation are no exception. Given mediation's ubiquity and importance to both theory and practice, QRPs in tests of mediation pose a serious threat to the advancement of psychology. We investigate this issue through the introduction of a straightforward means of detecting the presence and magnitude of QRPs in tests of mediation and validate this methodology with a series of sensitivity tests and simulations. We then apply this method to 2,569 tests of mediation published in five leading psychology journals in 2018 and 2019. We find that despite most hypothesized tests of mediation being likely underpowered, most (76%) were nevertheless supported. Furthermore, confidence intervals (CIs) that just barely exclude zero are 3.6 to 4.4 times as prevalent as those CIs that just barely include zero. We also find a number of study- and test-level factors, such as whether the test of mediation was hypothesized, explain both whether the CI excluded zero (odds ratio [OR] = 17.87, p < .001) as well as the CI's proximity to zero (b = .27, p < .001). In addition, other factors, most notably sample size, do predict the CI's proximity to zero (γ = .00, p < .001), but surprisingly do not predict the CI's exclusion of zero (OR = .99, p = .803). We conclude with actionable QRP curtailment strategies so that both, academics and practitioners, can have greater and well-founded confidence in tests of mediation in psychological research. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Infant and Child Development: A journal for open, transparent and inclusive science from prenatal through emerging adulthood

<i>Infant and Child Development</i>: A journal for open, transparent and inclusive science from prenatal through emerging adulthood

Spanish Scientists’ Opinion about Science and Researcher Behavior

We surveyed 348 Psychology and Education researchers within Spain, on issues such as their perception of a crisis in Science, their confidence in the quality of published results, and the use of questionable research practices (QRP). Their perceptions regarding pressure to publish and academic competition were also collected. The results indicate that a large proportion of the sample of Spanish academics think there is a crisis in Science, mainly due to a lack of economic investment, and doubts the quality of published findings. They also feel strong pressure to publish in high impact factor journals and a highly competitive work climate.

An Exploratory Analysis of 4844 Withdrawn Articles and Their Retraction Notes

The objective of our study was to obtain an updated image of the dynamic of retractions and retraction notes, retraction reasons for questionable research (QRP) and publication practices (QPP), countries producing retracted articles, and the scientific impact of retractions, by studying 4844 PubMed indexed retracted articles published between 2009 and 2020 and their retraction notes. Results: Mistakes/inconsistent data account for 32% of total retractions, followed by images (22.5%), plagiarism (13.7%) and overlap (11.5%). There were 163 cases of duplicate submission (3.36%), 180 cases of lack of reproducibility (3.72%), 181 cases of editorial errors (3.73%), 229 cases of fabricated data (4.73%) and 350 cases of fraudulent peer review (7.22%). Journals failed to properly report the retraction in 247 cases (5.1%). Thirty countries account for 94.79% of 4844 retractions. Top five are: China (32.78%), United States (18.95%), India (7.24%), Japan (4.37%) and Italy (3.75%). The total citations number for all articles is 140,810 (Google Scholar), 96,000 (Dimensions). Average exposure time (ET) is 28.89 months. The largest ET is for image retractions (49.3 months), the lowest ET is for editorial errors (11.2 months). The impact of retracted research is higher for Spain, Sweden, United Kingdom, United States, and other nine countries and lower for Pakistan, Turkey, Malaysia, and other six countries, including China. Conclusions: Mistakes and data inconsistencies represent the main retraction reason; images and ethical issues show a growing trend, while plagiarism and overlap continue to represent a significant problem. There is a steady increase in QRP and QPP article withdrawals with a peak of 878 retractions issued in 2020. Retraction of articles seems to be a technology-dependent process. The number of citations of retracted articles shows a high impact of papers published by authors from certain countries. The number of retracted articles per country does not always accurately reflect the scientific impact of QRP/QPP articles. The distribution of retraction reasons shows structural problems in the organization and quality control of scientific research, which have different images depending on geographical location, economic development, and cultural model.

Psihološka nauka od "objavi ili nestani" do "veruj, ali proveri"

At the time when social psychologists believed they could be proud of their discipline, there was the devastating news that Diederik Stapel had committed a major scientific fraud. This event coincided with the start of the discussion on trust in psychological findings. It was soon followed by the report of a series of nine studies that failed to replicate the 'professor's study'. These replication results were astounding due to earlier reports of successful replications. Due to the crisis of confidence in the results of field research, the Open Science Collaboration subsequently replicated 100 correlation and experimental studies published in 2008 in Psychological Science, Journal of Personality and Social Psychology, and Journal of Experimental Psychology: Learning, Memory, and Cognition. Of the 97% of the original studies that had a positive effect, 36% were replicated. However, their findings have also been called into question by calculating the Bayesian factor. In addition to fraud, questionable research practices resulting from publication bias that results in false positives undermine confidence in the validity of psychological research findings. Perhaps the most costly mistake of false-positive findings is to erroneously reject the null hypothesis. However, that Stapel (2011) confirmed the null hypothesis, or that Bargh (1996) found that admission of participants did not affect walking speed, or that Dijksterhuis and van Knipenberg (1998) reported that participants received with the word 'professor' did not improve their performance on task, no one would be interested in their findings. Zero findings are only interesting if they contradict the main hypothesis derived from the theory or contradict a number of previous studies. The fact that good experimental research is usually conducted in order to test theories, researchers can never be sure whether they have chosen the optimal operationalization of a given construct. As researchers can never be sure that they have properly operationalized the theoretical constructs they are evaluating and whether they have been successful in controlling the third variables that may be responsible for their results, the theory can never be proven true.

The Costs of HARKing

Kerr ([1998]) coined the term ‘HARKing’ to refer to the practice of ‘hypothesizing after the results are known’. This questionable research practice has received increased attention in recent years because it is thought to have contributed to low replication rates in science. The present article discusses the concept of HARKing from a philosophical standpoint and then undertakes a critical review of Kerr’s ([1998]) twelve potential costs of HARKing. It is argued that these potential costs are either misconceived, misattributed to HARKing, lacking evidence, or that they do not take into account pre- and post-publication peer review and public availability to research materials and data. It is concluded that it is premature to conclude that HARKing has led to low replication rates.

The contribution of Open Science to the quality of educational research

Ausgangspunkt dieses Beitrags sind Diskussionen um die Belastbarkeit empirischer Befunde in benachbarten Disziplinen, namentlich der Sozialpsychologie, die in der sog. „Replication Crisis“ gipfelten. Von derartigen Diskussionen um Replikationen und „Questionable Research Practices“ ist die Bildungsforschung bisher noch nicht in dem Maße betroffen, aber die Problemlagen sind in Teilbereichen ähnlich. Es mag daher nur eine Frage der Zeit sein, bevor diese Kontroversen auch in der Bildungsforschung aufkommen. Vor diesem Hintergrund argumentieren wir, wie Open Science einen Beitrag leisten kann, um die Belastbarkeit von Befunden der Bildungsforschung zu erhöhen. Im Besonderen greifen wir drei Open Science Praktiken auf: Präregistrierung, Open Materials und Open Data. Wir stellen diese vor und beleuchten, wie sie in der Bildungsforschung implementiert werden können. Wir thematisieren dabei die spezifischen Verhältnisse der Bildungsforschung im Vergleich zu Nachbardisziplinen, und gehen auf Limitationen und Besonderheiten der Bildungsforschung ein. Wir schließen mit einem Plädoyer für Transparenz.

Boosting Immunity of the Registered Reports System in Psychology to the Pandemic.

In psychology, a Registered Reports system is key to preventing questionable research practices. Under this system, manuscripts, including their detailed protocols (i.e., hypothesis, experimental design, sample size, and methods of statistical analysis), are reviewed prior to data collection. If a protocol manuscript is accepted, publication of the full manuscript including the results and discussion is guaranteed in principle regardless of whether the collected data support the registered hypothesis. However, this assurance of publication might be broken under the impact of the COVID-19 pandemic: Begrudging withdrawal of an accepted protocol manuscript due to a difficulty to meet the deadline by compelling reasons (e.g., pandemic) has occurred. The present paper reports the first author’s real-life experience related to the collapse of the assurance of publication in the Registered Reports system and discusses the disbenefits of this collapse. Furthermore, we propose the implementation of a journal section specific to protocol manuscripts as a solution to the crisis of the Registered Reports system.

P-Curve analysis of the Taylor Aggression Paradigm: Estimating evidentiary value and statistical power across 50 years of research.

The overall reliability or evidentiary value of any body of literature is established in part by ruling out publication bias for any observed effects. Questionable research practices have potentially undermined the evidentiary value of commonly used research paradigms in psychological science. Subsequently, the evidentiary value of these common methodologies remains uncertain. To quantify the severity of these issues in the literature, we selected the Taylor Aggression Paradigm (TAP) as a case study and submitted 170 hypothesis tests spanning over 50 years of research to a preregistered p-curve analysis. The TAP literature (N = 24,685) demonstrated significant evidentiary value but yielded a small average effect size (d = 0.29) and inadequate power (38%). The main effects demonstrated greater evidentiary value, power, and effect sizes than interactions. Studies that tested the effects of measured traits did not differ in evidentiary value or power to those that tested the effects of experimentally manipulated states. Exploratory analyses revealed that evidentiary value, statistical power, and effect sizes have improved over time. We provide recommendations for researchers who seek to maximize the evidentiary value of their psychological measures.

Bias in science: natural and social

Moral, social, political, and other “nonepistemic” values can lead to bias in science, from prioritizing certain topics over others to the rationalization of questionable research practices. Such values might seem particularly common or powerful in the social sciences, given their subject matter. However, I argue first that the well-documented phenomenon of motivated reasoning provides a useful framework for understanding when values guide scientific inquiry (in pernicious or productive ways). Second, this analysis reveals a parity thesis: values influence the social and natural sciences about equally, particularly because both are so prominently affected by desires for social credit and status, including recognition and career advancement. Ultimately, bias in natural and social science is both natural and social—that is, a part of human nature and considerably motivated by a concern for social status (and its maintenance). Whether the pervasive influence of values is inimical to the sciences is a separate question.

Plagiarism, FakePeer-Review, and Duplication: Predominant Reasons Underlying Retractions of Iran-Affiliated Scientific Papers.

Retractions of scientific papers published by some Iran-affiliated scientists in the preceding decade have attracted much attention and publicity; however, the reasons for these retractions have not been documented. We searched the Retraction Watch Database to enumerate the retracted Iran-affiliated papers from December 2001 to December 2019 and aimed to outline the predominant reasons for retractions. The reasons included fake peer-review, authorship dispute, fabricated data, plagiarism, conflict of interest, erroneous data, and duplication. The Fisher's exact test was used to investigate the associations between retractions and their underlying reasons. We selected P < 0.05 to indicate the statistically significant differences. We found 697 retracted papers. Duplication (27%), plagiarism (26%), and fake peer-review (21%) were the most frequent reasons for retractions. Our study highlights the importance of urgent intervention to prevent the misconduct and questionable research practices that lead to retractions in Iran. Continually educating the scientists and postgraduate students about the ethics and norms of scientific publishing is an important measure to ensure publication of reliable, worthy, and impactful papers.

L’évaluation et l’efficacité des psychothérapies psychanalytiques et de la psychanalyse

AimThis article proposes a review of the literature concerning the evaluation and the efficacy of psychoanalytic psychotherapy and psychoanalysis (PPP). MethodAfter proposing a global overview of the evaluation of psychotherapy, we sum up the empirical and quantitative evaluation of PPPs before focusing on their qualitative and process-based evaluation. ResultsPsychotherapies are very effective both in the short and in the long term. Their effectiveness is largely independent of the clinician's theoretical affiliation. Common factors such as the therapeutic alliance or the particularities of the therapist are prevalent, as are the duration and the frequency of the session throughout the therapy. Concerning more specifically the evaluation of PPPs, they have been empirically demonstrated to be effective for most psychiatric conditions. Several characteristics of PPPs are also specifically correlated with therapeutic effectiveness. Moreover, they produce long-term therapeutic effects, particularly concerning personality and after the end of the therapy. DiscussionThe qualitative and process-based evaluation of PPPs appears complementary to empirical evaluation, the latter suffering from several limitations (allegiance bias, indistinction of processes, questionable research practices, etc.) highlighted in particular by the recent crisis of reproducibility. The methodology of Randomized Controlled Trials should be associated with research based on clinical material. The approach of the Boston group, the analysis of psychotherapeutic processes with the Psychotherapy Q-Sort (PQS) and the modeling of the symbolization processes by the School of Lyon are three qualitative research paradigms that are particularly rich from this point of view. ConclusionPPPs are effective for most psychiatric disorders in the short term, at the end of the therapy, and for several years afterwards. They generate lasting transformations in terms of the reduction of symptoms and of personality evolution. They often appear to be more effective than pharmacotherapy and lead to substantial savings when they are implemented in health care facilities for patients suffering from a large variety of pathologies.

P‐Hacking, P‐Curves, and the PSM–Performance Relationship: Is There Evidential Value?

AbstractRecent developments in the social sciences have demonstrated that we cannot uncritically aggregate the published research on a particular effect to conclude about its presence or absence. Instead, questionable research practices such as p‐hacking (conducting additional analyses or collecting new data to obtain significant results) and selective publication of significant results can produce a body of published research that misleads readers even if it contains many significant results. It is, therefore, necessary to assess the evidential value of the research on a certain effect; that is, one must rule out that it is the result of questionable research practices. We introduce the p‐curve method to public administration research and apply it to the research on the relationship between public service motivation (PSM) and individual performance, to demonstrate how the evidential value of a body of published research can be assessed. We find that this particular literature contains evidential value.

The myth of ‘older LGBT+’ people: Research shortcomings and policy/practice implications for health/care provision

This article explores the implications of research which takes a collectivised approach to lesbian, gay, bisexual and trans+ (LGBT+) ageing and which engages in Questionable Research Practices (QRPs) in doing so. Collectivised approaches to heterogenous identity-based groups address commonalities but often fail to address internal diversity, i.e. the differences between and among older LGBT+ people. This article explores six key problems associated with collectivised research: (1) Homogenising language and phrases; (2) Uneven numerical representation of sub-groups; (3) Thematic over-representation of sexuality; (4) Non-intersectional analyses; (5) Thematic under-representation of gender; and (6) Inaccurate reporting of data. Research which does not differentiate between ‘older LGBT+’ sub-populations, can provide policy-makers and practitioners with inaccurate and/or misleading information, resulting in services which meet the needs of some, but not all, older LGBT+ people. This article discusses how research can become more inclusive, intersectional and reliable.

Bridging research integrity and global health epidemiology (BRIDGE) statement: guidelines for good epidemiological practice

BackgroundResearch integrity and research fairness have gained considerable momentum in the past decade and have direct implications for global health epidemiology. Research integrity and research fairness principles should be equally...