Poor Research Practices Research Articles

There is nothing WEIRD about basic research: The critical role of convenience samples in psychological science.

Attention to issues of sample diversity and generalizability has increased dramatically in the past 15 years, as psychological scientists have confronted the limitations of relatively homogeneous samples. Though this reckoning was perhaps overdue and has undoubtedly shined a light on some poor research practices, recommendations surrounding sample diversity are sometimes applied to research that does not aim for generalizability across peoples. In this article, I seek to promote discussion about when and why sample diversity and generalizability matter. In doing so, I address problems with language surrounding generalizability, the broader question of generalizability beyond samples, challenges for determining sufficient generalizability, and the inherent question of moderation in psychological science, given the reality of limited time and resources. I then discuss the important roles that basic research plays in understanding group differences, producing generalizable knowledge, and developing applied interventions. Finally, I address issues of equity surrounding sample diversity, emphasizing the distinction between WEIRD samples and convenience samples and the importance of convenience samples for globalizing psychological science. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Four principles for improved statistical ecology

Abstract Increasing attention has been drawn to the misuse of statistical methods over recent years, with particular concern about the prevalence of practices such as poor experimental design, cherry picking and inadequate reporting. These failures are largely unintentional and no more common in ecology than in other scientific disciplines, with many of them easily remedied given the right guidance. Originating from a discussion at the 2020 International Statistical Ecology Conference, we show how ecologists can build their research following four guiding principles for impactful statistical research practices: (1) define a focussed research question, then plan sampling and analysis to answer it; (2) develop a model that accounts for the distribution and dependence of your data; (3) emphasise effect sizes to replace statistical significance with ecological relevance; and (4) report your methods and findings in sufficient detail so that your research is valid and reproducible. These principles provide a framework for experimental design and reporting that guards against unsound practices. Starting with a well‐defined research question allows researchers to create an efficient study to answer it, and guards against poor research practices that lead to poor estimation of the direction, magnitude, and uncertainty of ecological relationships, and to poor replicability. Correct and appropriate statistical models give sound conclusions. Good reporting practices and a focus on ecological relevance make results impactful and replicable. Illustrated with two examples—an experiment to study the impact of disturbance on upland wetlands, and an observational study on blue tit colouring—this paper explains the rationale for the selection and use of effective statistical practices and provides practical guidance for ecologists seeking to improve their use of statistical methods.

Identifying and managing problematic trials: A research integrity assessment tool for randomized controlled trials in evidence synthesis.

Evidence synthesis findings depend on the assumption that the included studies follow good clinical practice and results are not fabricated or false. Studies which are problematic due to scientific misconduct, poor research practice, or honest error may distort evidence synthesis findings. Authors of evidence synthesis need transparent mechanisms to identify and manage problematic studies to avoid misleading findings. As evidence synthesis authors of the Cochrane COVID-19 review on ivermectin, we identified many problematic studies in terms of research integrity and regulatory compliance. Through iterative discussion, we developed a research integrity assessment (RIA) tool for randomized controlled trials for the update of this Cochrane review. In this paper, we explain the rationale and application of the RIA tool in this case study. RIA assesses six study criteria: study retraction, prospective trial registration, adequate ethics approval, author group, plausibility of methods (e.g., randomization), and plausibility of study results. RIA was used in the Cochrane review as part of the eligibility check during screening of potentially eligible studies. Problematic studies were excluded and studies with open questions were held in awaiting classification until clarified. RIA decisions were made independently by two authors and reported transparently. Using the RIA tool resulted in the exclusion of >40% of studies in the first update of the review. RIA is a complementary tool prior to assessing "Risk of Bias" aiming to establish the integrity and authenticity of studies. RIA provides a platform for urgent development of a standard approach to identifying and managing problematic studies.

Observation, experimentation, and replication in linguistics

Abstract In this paper, I propose that replication failure in linguistics may be due primarily to inherent issues with the application of experimental methods to analyze an inextricably social phenomenon like language, as opposed to poor research practices. Because language use varies across social contexts, and because social context must vary across independent experimental replications, linguists should not be surprised when experimental results fail to replicate at the expected rate. To address issues with replication failure in linguistics, and to increase methodological rigor in our field more generally, I argue that linguists must use experimental methods carefully, keeping in mind their inherent limitations, while acknowledging the scientific value of observational methods, which are often the only way to pursue basic questions in our field.

We object to Bad Science: Poor research practices should be discouraged!

This Commentary is a response to a Commentary published in the May/June 2020 issue:
 Nattrass N. Why are black South African students less likely to consider studying biological sciences? S Afr J Sci. 2020;116(5/6), Art. #7864, 2 pages. https://doi.org/10.17159/sajs.2020/7864
 Responses to the Commentary in the May/June 2020 issue have been published collectively in a special issue of Volume 116.

Ethical considerations when conservation research involves people.

Social science is becoming increasingly important in conservation, with more studies involving methodologies that collect data from and about people. Conservation science is a normative and applied discipline designed to support and inform management and practice. Poor research practice risks harming participants and, researchers, and can leave negative legacies. Often, those at the forefront of field-based research are early-career researchers, many of whom enter their first research experience ill-prepared for the ethical conundrums they may face. We draw on our own experiences as early-career researchers to illuminate how ethical challenges arise during conservation research that involves human participants. Specifically, we considered ethical review procedures, conflicts of values, and power relations, and devised broad recommendations on how to navigate ethical challenges when they arise during research. In particular, we recommend researchers apply reflexivity (i.e., thinking that allows researchers to recognize the effect researchers have on the research) to help navigate ethical challenges and encourage greater engagement with ethical review processes and the development of ethical guidelines for conservation research that involves human participants. Such guidelines must be accompanied by the integration of rigorous ethical training into conservation education. We believe our experiences are not uncommon and can be avoided and hope to spark discussion to contribute to a more socially just conservation.

Turning the tables: A university league-table based on quality not quantity.

Background: Universities closely watch international league tables because thesetables influence governments, donors and students. Achieving a high ranking in atable, or an annual rise in ranking, allows universities to promote their achievementsusing an externally validated measure. However, league tables predominantly rewardmeasures of research output, such as publications and citations, and maytherefore be promoting poor research practices by encouraging the "publish or perish"mentality. Methods: We examined whether a league table could be created based on goodresearch practice. We rewarded researchers who cited a reporting guideline, whichhelp researchers report their research completely, accurately and transparently,and were created to reduce the waste of poorly described research. We used theEQUATOR guidelines, which means our tables are mostly relevant to health andmedical research. We used Scopus to identify the citations. Results: Our cross-sectional tables for the years 2016 and 2017 included 14,408papers with 47,876 author affiliations. We ranked universities and included a bootstrapmeasure of uncertainty. We clustered universities in five similar groups in aneffort to avoid over-interpreting small differences in ranks. Conclusions: We believe there is merit in considering more socially responsiblecriteria for ranking universities, and this could encourage better research practiceinternationally if such tables become as valued as the current quantity-focused tables.

Turning the tables: A university league-table based on quality not quantity.

Background: Universities closely watch international league tables because thesetables influence governments, donors and students. Achieving a high ranking in atable, or an annual rise in ranking, allows universities to promote their achievementsusing an externally validated measure. However, league tables predominantly rewardmeasures of research output, such as publications and citations, and maytherefore be promoting poor research practices by encouraging the "publish or perish"mentality. Methods: We examined whether a league table could be created based on goodresearch practice. We rewarded researchers who cited a reporting guideline, whichhelp researchers report their research completely, accurately and transparently,and were created to reduce the waste of poorly described research. We used theEQUATOR guidelines, which means our tables are mostly relevant to health andmedical research. Results: Our cross-sectional tables for the years 2016 and 2017 included 14,408papers with 47,876 author affiliations. We ranked universities and included a bootstrapmeasure of uncertainty. We clustered universities in five similar groups in aneffort to avoid over-interpreting small differences in ranks. Conclusions: We believe there is merit in considering more socially responsiblecriteria for ranking universities, and this could encourage better research practiceinternationally if such tables become as valued as the current quantity-focused tables.

Mind the Hype: A Critical Evaluation and Prescriptive Agenda for Research on Mindfulness and Meditation

During the past two decades, mindfulness meditation has gone from being a fringe topic of scientific investigation to being an occasional replacement for psychotherapy, tool of corporate well-being, widely implemented educational practice, and “key to building more resilient soldiers.” Yet the mindfulness movement and empirical evidence supporting it have not gone without criticism. Misinformation and poor methodology associated with past studies of mindfulness may lead public consumers to be harmed, misled, and disappointed. Addressing such concerns, the present article discusses the difficulties of defining mindfulness, delineates the proper scope of research into mindfulness practices, and explicates crucial methodological issues for interpreting results from investigations of mindfulness. For doing so, the authors draw on their diverse areas of expertise to review the present state of mindfulness research, comprehensively summarizing what we do and do not know, while providing a prescriptive agenda for contemplative science, with a particular focus on assessment, mindfulness training, possible adverse effects, and intersection with brain imaging. Our goals are to inform interested scientists, the news media, and the public, to minimize harm, curb poor research practices, and staunch the flow of misinformation about the benefits, costs, and future prospects of mindfulness meditation.

Core Outcome Sets

‘Clinical trials are only as credible as their outcomes.’1 Clinical trials guide clinical practice. They do this by demonstrating beneficial or detrimental effects of an intervention on patients: the outcomes...

Referee Report For: The culture of scientific research

In 2014, the UK-based Nuffield Council on Bioethics carried out a series of engagement activities, including an online survey to which 970 people responded, and 15 discussion events at universities around the UK to explore the culture of research in the UK and its effect on ethical conduct in science and the quality of research. The findings of the project were published in December 2014 and the main points are summarised here. We found that scientists are motivated in their work to find out more about the world and to benefit society, and that they believe collaboration, multidisciplinarity, openness and creativity are important for the production of high quality science. However, in some cases, our findings suggest, the culture of research in higher education institutions does not support or encourage these goals or activities. For example, high levels of competition and perceptions about how scientists are assessed for jobs and funding are reportedly contributing to a loss of creativity in science, less collaboration and poor research practices. The project led to suggestions for action for funding bodies, research institutions, publishers and editors, professional bodies and individual researchers.

The culture of scientific research.

In 2014, the UK-based Nuffield Council on Bioethics carried out a series of engagement activities, including an online survey to which 970 people responded, and 15 discussion events at universities around the UK to explore the culture of research in the UK and its effect on ethical conduct in science and the quality of research. The findings of the project were published in December 2014 and the main points are summarised here. We found that scientists are motivated in their work to find out more about the world and to benefit society, and that they believe collaboration, multidisciplinarity, openness and creativity are important for the production of high quality science. However, in some cases, our findings suggest, the culture of research in higher education institutions does not support or encourage these goals or activities. For example, high levels of competition and perceptions about how scientists are assessed for jobs and funding are reportedly contributing to a loss of creativity in science, less collaboration and poor research practices. The project led to suggestions for action for funding bodies, research institutions, publishers and editors, professional bodies and individual researchers.

ORI findings of scientific misconduct in clinical trials and publicly funded research, 1992–2002

Since 1992 the Office of Research Integrity (ORI) had reviewed investigations of scientific misconduct in research funded by the US Public Health Service (PHS). ORI defined scientific misconduct as "fabrication, falsification, plagiarism, or other practices that seriously deviate from those that are commonly accepted within the scientific community for proposing, conducting, or reporting research". The purpose of this study was to summarize the findings and administrative actions reported in ORI notices of scientific misconduct in clinical trials occurring between May 1992 and 2002. Findings of misconduct were gathered from publicly available sources: the ORI annual reports and the NIH Guide to Grants and Contracts. Clinical trials accounted for 17 (13%) of the 136 investigations that resulted in findings of scientific misconduct, and they were noted in 12 (11%) of the 113 brief reports of investigations closed with findings of no scientific misconduct. In clinical trials, the most severe sanction, debarment from US Government funding, was applied in six (35%) of the cases of misconduct compared to 79 (66%) of 119 cases from all other types of research combined. Of individuals cited for misconduct in clinical trials, three (18%) held doctorates in contrast to 81 (68%) in other types of research. In clinical trials, junior employees may bear the burden of sanction for scientific misconduct. The most frequently applied sanction was the requirement that a plan of supervision of the sanctioned employees accompany any future application for funding which would include them. This imposition of sanction on an individual employee does not address possible causes of misconduct which may be inherent in the overall pattern of leadership, training and supervision in the trial. Furthermore, the definition of misconduct, as interpreted by the Departmental Appeals Board, excludes carelessness and other poor research practices that may lead to dissemination of more incorrect data than misconduct.

How valid and reliable are patient satisfaction data? An analysis of 195 studies

To assess the properties of validity and reliability of instruments used to assess satisfaction in a broad sample of health service user satisfaction studies, and to assess the level of awareness of these issues among study authors. Examination and analysis of 195 papers published in 1994 in 139 journals. The following databases were searched: British Nursing Index, CINAHL, EMBASE, MedLine, Popline, and PsycLIT. Number and types of strategies used for content, criterion, and construct validity, and for stability and internal consistency. Associations between validity/reliability and other study characteristics. Eighty-nine (46%) of the 195 studies reported some validity or reliability data; 76 reported some element of content validity; 14 reported criterion validity, with patient's intent to return the most commonly used criterion; four reported construct validity. Thirty-four studies reported internal consistency reliability, 31 of which used Cronbach's coefficient alpha; eight studies reported test-retest reliability. Only 11 studies (6% of the 181 quantitative studies) reported content validity and criterion or construct validity and reliability. 'New' instruments designed specifically for the reported study demonstrated significantly less evidence for reliability/validity than did 'old' instruments. With few exceptions, the study instruments in this sample demonstrated little evidence of reliability or validity. Moreover, study authors exhibited a poor understanding of the importance of these properties in the assessment of satisfaction. Researchers must be aware that this is poor research practice, and that lack of a reliable and valid assessment instrument casts doubt on the credibility of satisfaction findings.

An introduction to research ethics

Practical issues throughout scientific research can be found to have an ethical aspect. There is a gray area in which scientific error (“honest error”) may be difficult to distinguish from unacceptably poor research practice or an unethical failure to follow scientific norms. Further, there is no clear margin between deceptive practices which are widely accepted and those which must be considered fraudulent. Practical problems arise in matters of data management and presentation, authorship, publication practices, “grantsmanship”, and rights of research trainees, as well as the well-recognized areas of human and animal experimentation. Beyond the gray areas, the legal definition of research misconduct is discussed in relation to research fraud, and the latest proposed definition of the Commission on Research Integrity is briefly reviewed. It is noted that the standards of ethical research are changing. Finally, there is a comment on the idea of institutional integrity in research, and the critical role of the mentor in transmitting research standards to the next generation.