Positive Outcome Bias Research Articles

Positive Outcome Bias in the Influential Otolaryngology Clinical Trial Literature.

To assess for the presence of positive outcome bias in the otolaryngology clinical trial literature. Specifically, we investigate the prevalence of clinical trials with positive findings (CTP) and clinical trials with negative findings (CTN), as well as their quality of evidence and subsequent impact. Retrospective analysis. Clinical Trials in the Influential Otolaryngology Literature. We reviewed all clinical trials published in 4 major otolaryngology journals between 2000 and 2020. We constructed several multivariable regression models to investigate the relationship of finding direction with randomization status and citation count. Subsequently, we incorporated an interaction term between year and the primary covariate of each model to assess the temporal trajectory of these relationships. All models accounted for sample size, journal, subspecialty, and the affiliated program prestige. Of the 1367 trials analyzed, 1143 (84%) were CTPs, a rate that persisted throughout the study period (aOR 1.00, 95% CI 0.98-1.03). CTPs were significantly less likely to be randomized compared to CTNs (aOR 0.25, 0.17-0.37), a relationship that persisted over time (aOR 1.05, 0.99-1.03). CTPs received significantly more citations compared to CTNs (aIRR 1.41, 1.25-1.60), a disparity that also persisted over time (aIRR 0.99, 0.97-1.01). The otolaryngology clinical trial literature has been heavily dominated by positive findings. CTPs were more frequently cited and published even with a lower level of evidence compared to CTNs. This bias may influence the objectivity of evidence used to guide clinical practice and warrants attention when reviewing findings and changing practices.

Score! What is it Good for? Why Football Managers Need to Look Beyond Results

Outcome bias is a pervasive phenomenon in decision-making, referring to the tendency to evaluate decisions made under identical circumstances as more favourable when it results in the desired outcome. This paper analyses this cognitive bias in the context of top-flight European football, examining whether a Bayesian updating model distorted by a multiplicative outcome bias is valid. Managers make significantly more changes to their strategy following a loss than a win, even having controlled for expected performance, in-game performance, and team and game-specific variation. The results of this paper are consistent with a positive-outcome bias, but not necessarily a multiplicative bias.

Publication bias, time-lag bias, and place-of-publication bias in social intervention research: An exploratory study of 527 Swedish articles published between 1990-2019.

Publication and related biases constitute serious threats to the validity of research synthesis. If research syntheses are based on a biased selection of the available research, there is an increased risk of producing misleading results. The purpose fo this study is to explore the extent of positive outcome bias, time-lag bias, and place-of-publication bias in published research on the effects of psychological, social, and behavioral interventions. The results are based on 527 Swedish outcome trials published in peer-reviewed journals between 1990 and 2019. We found no difference in the number of studies reporting significant compared to non-significant findings or in the number of studies reporting strong effect sizes in the published literature. We found no evidence of time-lag bias or place-of-publication bias in our results. The average reported effect size remained constant over time as did the proportion of studies reporting significant effects.

Be Positive! International Stroke Conference Oral Presentations ‐ 2015 to 2020

Be Positive! International Stroke Conference Oral Presentations ‐ 2015 to 2020

Implications of Eligibility Criteria on the Generalizability of Alcohol and Drug Treatment Outcome Research: A Study of Real-World Treatment Seekers in Sweden and in Australia

ABSTRACTBackground: Clinical studies of alcohol and drug treatment outcomes frequently apply participant eligibility criteria (EC), which may exclude real-world treatment seekers, impairing the representativeness of studied samples. Some research exists on the impact of EC on alcohol treatment seekers. Little is known about drug treatment and country differences. Objectives: We tested and compared the degree to which commonly used EC exclude real-world treatment seekers with problem alcohol and drug use in Sweden and Australia, and compared the impact of EC on outcomes. Methods: Two large naturalistic and comparative service user samples were used. Respondents were recruited in Stockholm County (n = 1,865; data collection 2000–2002), and Victoria and Western Australia (n = 796; in 2012–2013). Follow-up interviews were conducted after 1 year. Cross-tabulations, Chi-square (χ2) tests and logistic regressions were used. Results: Percentages of the samples excluded by individual EC ranged from 5% (lack of education/literacy) to 70% (social instability) among Swedish alcohol cases and from 2% (low alcohol problem severity) to 69% (psychiatric medication) among Australian counterparts; and from 2% (age 60+ years) to 82% (social instability) among Swedish drug cases and from 1% (age 60+ years) to 67% (psychiatric medication) among Australian counterparts. Country differences and differences across substances appeared independent of country effect. Co-morbid psychiatric medication, noncompliance, poly drug use, and low education EC caused positive 1-year outcome bias; whereas female sex and old age introduced negative outcome bias. Conclusions/Importance: Commonly used EC exclude large proportions of treatment seekers. This may impair generalizability of clinical research, and the effects of many EC differ by country and drug type.

Publication fate of abstracts presented at Society of British Neurological Surgeons meetings

Introduction. Society of British Neurological Surgeons (SBNS) meetings are important national events which allow for the presentation of current academic work. The publication rate of presented abstracts is considered a proxy marker of the scientific strength of a conference. We aimed to determine the publication fate of presented abstracts at SBNS meetings over a 5-year period. Methods. A retrospective review of SBNS conference proceedings between 2001 and 2005 was performed. To ascertain whether an abstract resulted in peer-reviewed publication, a range of databases (PubMed, Google Scholar, Medline and Ovid) were interrogated. s published in full were subsequently assessed for journal impact factor (IF), time of publication and number of citations received (per Google Scholar). Results. A total of 494 abstracts were presented. Of these, 181 abstracts were subsequently published in full, giving the conference a publication rate of 36.6%. The mean time to publication from presentation was 22 months (range 35 months pre-presentation to 133 months afterwards). The top three journals for publication were the British Journal of Neurosurgery (23.2%), Neurosurgery (7.7%) and Journal of Neurosurgery (7.7%). The IF of journal destinations ranged from no IF to 38.28 (median = 1.97). Number of citations ranged from 0 to 963 (median = 22). s with positive results were significantly more likely to be published in full compared to those with negative results (p = 0.0001). Conclusions. SBNS conferences have a respectable publication rate. Those abstracts that are published in full have gone on to gain a considerable number of citations reflecting their scientific relevance. However, studies presented at SBNS are susceptible to positive outcome bias.

A Study of the Impact of Thirteen Celebrity Suicides on Subsequent Suicide Rates in South Korea from 2005 to 2009

A number of ecological studies have found a pattern of increasing suicide rates after suicides of several Asian entertainment celebrities. However, the finding may be subject to positive outcome bias where cases with no perceived impact may be routinely excluded. In this study, we deploy interrupted time-series analysis using ARIMA transfer function models to investigate systematically the impact of thirteen celebrity suicides on subsequent suicide rates in South Korea. We find that three out of eleven cases were found to be followed by a significant increase in suicide rate, while controlling for seasonality, secular trends, and unemployment rates. Such significant increases could last for nine weeks. Non-significance cases may be attributable to the small amount of media coverage, the “displacement” effect of preceding case, and the negative connotation of celebrity deaths. We therefore conclude that whether or not the impacts were detected may be largely conditioned by various contextual factors. Current evidence based on ecological studies is insufficient to draw a firm conclusion. Further studies using multiple approaches should be developed.

Eliminating bias/living with bias?

Message from the Editor—Eliminating bias/living with bias? The scientific method and elements of bias are fundamentally at odds with one another. While the prior seeks to identify truth through objective analysis, the latter wishes to prove one’s preconceptions through artificially created or filtered observations. Both the scientific method and bias are purpose-driven notions, both serve an intrinsic value. Arguably the more general and truly-lasting societal value is derived from our attempts at approximating truth through scientific methodology, compared with the more selfserving notions of reaffirming our preconceptions through bias. But what if we cannot really separate the two entities—the scientific method and bias? We have explored the theme of bias in previous EBSJ editorials from the perspectives of investigator, commercial, and even publisher bias [1]. We have also learned that characteristics, which favor publication in major scientific medical journals, are United States–based articles that report good outcomes and are commercially funded [2]. Efforts to eliminate bias for the greater good of medicine are still far from complete but the movement to ‘rate’ research based on its Class of Evidence (CoE) is important, as it is basically a rating of the presence of bias. Please look through the Science in Spine article in this EBSJ issue on “Class or level of evidence” by Dettori (page 9–12). Without oversimplifying its content, the entire article underscores the value of a ‘high-class’ study compared with a ‘lower- class evidence’ by enumerating the incessant awareness it takes to eliminate bias with every step of the way of a research project and why the rating of the evidence is so important. Ultimately it is bias-free evidence that reveals the truth behind a hypothesis, shows previously unknown cross-connections, and allows us to rationally formulate new and more effective treatments. The thought of improving upon nature through our medical interventions is perhaps one of the core motivators that has led us into the healthcare field in the first place. As physicians and healthcare providers, we by nature tend to be more optimists, we hope for improvement with our patients, and value positive achievements. However, this disposition may also lead us to displace negative experiences. Perhaps this is a helpful protective phenomenon as it provides us with capacity to move onward. But what if this core proclivity toward optimism is a problem in itself by introducing bias not only in our clinical medicine practices but also in our approach toward research? This unsettling thought was raised by Emerson et al (full disclosure: some of the authors are fellow employees at the University of Washington) in their recent article titled “Testing for a positive outcome bias in peer review: a randomized controlled trial.” With permission from the editors-inchief of two leading orthopaedic publications, JBJS and CORR, the authors submitted a fabricated sham article either with positive outcomes or the very same sham article with ‘no-difference’ findings to 210 regular reviewers. A statistically significant number of reviewers recommended publication of the positive outcomes version of the sham paper compared with the negative version (97.3% versus 80.0%, P < .005) [3].

Negative results are disappearing from most disciplines and countries

Concerns that the growing competition for funding and citations might distort science are frequently discussed, but have not been verified directly. Of the hypothesized problems, perhaps the most worrying is a worsening of positive-outcome bias. A system that disfavours negative results not only distorts the scientific literature directly, but might also discourage high-risk projects and pressure scientists to fabricate and falsify their data. This study analysed over 4,600 papers published in all disciplines between 1990 and 2007, measuring the frequency of papers that, having declared to have “tested” a hypothesis, reported a positive support for it. The overall frequency of positive supports has grown by over 22% between 1990 and 2007, with significant differences between disciplines and countries. The increase was stronger in the social and some biomedical disciplines. The United States had published, over the years, significantly fewer positive results than Asian countries (and particularly Japan) but more than European countries (and in particular the United Kingdom). Methodological artefacts cannot explain away these patterns, which support the hypotheses that research is becoming less pioneering and/or that the objectivity with which results are produced and published is decreasing.

Publikationsbias in Studien jenseits RCT

Publication bias arises from the preferential publication of studies with positive, statistically significant, or particularly strong effects. In meta-analysis on the effectiveness of interventions this may, for example, lead to an overestimation of an effect. In studies on harm, though, wish bias and publication bias frequently take opposing directions. In consequence the overall effect is more difficult to interpret. Generally, non randomized studies might be expected to have a higher potential for publication bias as they have lower publication rates. Specifically, adverse effects are often not reported in a standardized way and tend to be secondary outcomes. Related biases such as positive outcome bias and location bias (problems in locating studies, retrieval bias) are therefore also more likely. As yet, however, there is no empirical proof to support this since research activities on the subject of publication bias is almost exclusively focused on randomized controlled trials.

Testing for the Presence of Positive-Outcome Bias in Peer Review

If positive-outcome bias exists, it threatens the integrity of evidence-based medicine. We sought to determine whether positive-outcome bias is present during peer review by testing whether peer reviewers would (1) recommend publication of a "positive" version of a fabricated manuscript over an otherwise identical "no-difference" version, (2) identify more purposefully placed errors in the no-difference version, and (3) rate the "Methods" section in the positive version more highly than the identical "Methods" section in the no-difference version. Two versions of a well-designed randomized controlled trial that differed only in the direction of the finding of the principal study end point were submitted for peer review to 2 journals in 2008-2009. Of 238 reviewers for The Journal of Bone and Joint Surgery and Clinical Orthopaedics and Related Research randomly allocated to review either a positive or a no-difference version of the manuscript, 210 returned reviews. Reviewers were more likely to recommend the positive version of the test manuscript for publication than the no-difference version (97.3% vs 80.0%, P < .001). Reviewers detected more errors in the no-difference version than in the positive version (0.85 vs 0.41, P < .001). Reviewers awarded higher methods scores to the positive manuscript than to the no-difference manuscript (8.24 vs 7.53, P = .005), although the "Methods" sections in the 2 versions were identical. Positive-outcome bias was present during peer review. A fabricated manuscript with a positive outcome was more likely to be recommended for publication than was an otherwise identical no-difference manuscript.

Misleading P-Value: Do You Recognise It?

Misleading P-Value: Do You Recognise It?

Publication Bias in Kienböck's Disease: Systematic Review

Kienböck's disease is considered rare and currently affects fewer than 200,000 people in the United States. Given the inherent challenges associated with researching rare diseases, the intense effort in hand surgery to treat this uncommon disorder may be influenced by publication bias in which positive outcomes are preferentially published. The specific aim of this project was to conduct a systematic review of the literature with the hypothesis that publication bias is present for the treatment of Kienböck's disease. We conducted a systematic review of all available abstracts associated with published manuscripts (English and non-English) and abstracts accepted to the 1992 to 2004 American Society for Surgery of the Hand (ASSH) annual meetings. Data collection included various study characteristics, direction of outcome (positive, neutral/negative), complication rates, mean follow-up time, time to publication, and length of patient enrollment. Our study included 175 (124 English, 51 non-English) published manuscripts and 14 abstracts from the 1992 to 2004 annual ASSH meetings. Abstracts from published manuscripts were associated with a 53% positive outcome rate, which is lower than the 74% positive outcome rate found among other surgically treated disorders. Over the past 40 years, studies have become more positive (36% to 68%, p=.007) and are more likely to incorporate statistical analysis testing (0% to 55%, p<.001). Of the 14 abstracts accepted to ASSH, 11 were published in peer-reviewed journals. Ten of the 14 accepted abstracts were considered positive, and there was no significant difference in publication rate between studies with positive (n = 10) and negative (n = 4) outcomes (p>.999). The acceptance rate for negative outcomes studies regarding Kienböck's disease is higher than for other surgical disorders. This may indicate a relative decrease in positive outcome bias among published Kienböck's disease studies compared with other surgical disorders. However, the increasing positive outcome rate for published Kienböck's disease studies over time may suggest a trend of increasing publication bias among journals toward Kienböck's disease studies.

Abstracts Presented at the American Urological Association Annual Meeting: Determinants of Subsequent Peer Reviewed Publication

Abstracts submitted to medical meetings do not undergo the same critical peer review process as published manuscripts. Despite this limited scrutiny presented abstracts often influence clinical thinking and practice. Consequently the peer reviewed publication rate of abstracts becomes critical in judging the quality of this research. We determined this publication rate and factors influencing it. All 1,584 abstracts presented at the 2000 American Urological Association Annual Meeting were reviewed and assessed for subsequent publication with a fixed MEDLINE search protocol. We searched for publications from January 1, 1999 to May 31, 2005. Abstracts were deemed published if 1) at least 1 author of the presented abstract was a manuscript author and 2) at least 1 conclusion in the presented abstract was included in the final publication conclusions. Publication rates according to mode and topic of presentation, country or state of origin and time to publication were calculated. Journal impact factors for publications were compared according to these variables. Of presented abstracts from the 2000 American Urological Association meeting 55% went on to successful publication, including 59% of podium, 55% of poster, 55% of unmoderated poster and 42% of video presentations. Mean time from presentation to publication was 17 months. The average journal impact factor was 3.2. A significant proportion of presentations at the American Urological Association Annual Meeting is never subjected to or fails the critical peer review process. The overall journal impact factor for published manuscripts is modest. Meeting attendees should consider these observations when deciding whether to incorporate the findings of presentations into their clinical practice.

Critical Appraisal and Positive Outcome Bias in Case Reports Published in Brazilian Dental Journals

The aim of this study was to carry out a critical appraisal to detect the occurrence of publication bias, specifically positive-outcome bias, in Brazilian dental journals. A convenience sample of twenty-eight dental journals published in Brazil between 1994 and 2003 was selected (564 issues containing 5,453 articles). Each article was classified according to methodological design; 1,064 were classified as case reports. A simple random sample of 435 articles comprised the final study sample and was submitted to a critical appraisal by previously trained raters. Although a great part of case reports achieve proper structural aspects, most authors did not consider important aspects that could help the decision making process, such as discussion of alternative approaches, limitation of the performed procedures, lack of long-term clinical follow-up, possible variations in treatment outcome, and failure to discuss limitations of study design. In addition, unanimous reports of successful outcomes of clinical interventions strongly suggest that publication of case reports was influenced by positive outcome bias, which may influence readers' clinical decision making process.

Publication Bias of Randomized Controlled Trials in Emergency Medicine

To determine the publication status and time to publication of randomized controlled trials (RCTs) that were presented at the Society for Academic Emergency Medicine (SAEM) meetings from 1995 to 2003. The impact of positive-outcome bias, time-lag bias, and gray literature bias also was assessed. Retrospective cohort study of RCT abstracts presented at nine SAEM scientific meetings. Electronic searches identified publications from the abstracts. Of 4,399 abstracts, 383 (8.7%; 95% confidence interval [CI] = 7.8% to 9.5%) were identified as RCTs. One hundred ninety-four (50.7%; 95% CI = 45.7% to 55.7%) were subsequently published up to May 2004. The median time to publication was 32 months (95% CI = 23 to 41), with 59% of RCT abstracts published within five years of presentation. No evidence of positive-outcome bias or time-lag bias was identified; however, changes from abstract to manuscript were found. Manuscripts were less likely to endorse the experimental intervention than were abstracts (OR, 0.2; 95% CI = 0.0 to 0.6). The proportion of emergency medicine RCT abstracts published is slightly lower than that for other biomedical specialties; however, biases reported by investigators in other biomedical areas do not appear to be as problematic in emergency medicine research. Differences between conclusions from abstracts and manuscripts must be considered when employing meeting abstracts as a source of evidence for future research or for systematic reviews in emergency medicine.

Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals.

Citation by other authors is important in the dissemination of published science, but factors predicting it are little studied. To identify characteristics of published research predicting citation in other journals, we searched the Science Citations Index database for a standardized 3.5 years for all citations of published articles originally submitted to a 1991 emergency medicine specialty meeting. Analysis was conducted by classification and regression trees, a nonparametric modeling technique of regression trees, to determine the impact of previously determined characteristics of the full articles on the outcome measures. We calculated the the number of times an article was cited each year and calculated the mean impact factor (citations per manuscript per year) in other citing journals. Of the 493 submitted manuscripts, 204 published articles met entry criteria. The mean citations per year was 2.04 (95% confidence interval, 1.6-2.4; range, 0-20.9) in 440 different journals. Nineteen articles (9.3%) were never cited. The ability to predict the citations per year was weak (pseudo R(2) = 0.14.). The strongest predictor of citations per year was the impact factor of the original publishing journal. The presence of a control group, the subjective newsworthiness score, and sample size predicted citation frequency (24.3%, 26.0%, and 26.5% as strongly, respectively). The ability to predict mean impact factor of the citing journals was even weaker (pseudo R(2) = 0.09). The impact factor of the publishing journal was the strongest predictor, followed by the newsworthiness score (89.9% as strongly) and a subjective quality score (61.5%). Positive outcome bias was not evident for either outcome measure. In this cohort of published research, commonly used measures of study methodology and design did not predict the frequency of citations or the importance of citing journals. Positive outcome bias was not evident. The impact factor of the original publishing journal was more important than any other variable, suggesting that the journal in which a study is published may be as important as traditional measures of study quality in ensuring dissemination.

Factors associated with acceptance and full publication of GI endoscopic research originally published in abstract form

Background: Many abstracts submitted to annual scientific meetings never come to full publication in peer-reviewed journals. The objective of this study was to determine factors associated with the fate of endoscopic research abstracts submitted to the annual scientific meeting of the American Society for Gastrointestinal Endoscopy (ASGE). Methods: All abstracts (n = 461) submitted to the annual meeting of the ASGE in May of 1994 were retrospectively reviewed. The following databases were searched for evidence of publication of abstracts in full-manuscript form: Medline, HealthSTAR, Current Contents, CINHAL, and Cancerlit. All abstracts were reviewed between May 4, 1998 and June 30, 1998. Univariate and multivariate analysis were performed to determine the association between abstract characteristics and acceptance for presentation at the meeting and for publication. Results: Fifty-five percent (247/451) of submitted abstracts were accepted for presentation. In univariate analysis, pediatric studies, prospective studies, randomized studies, and studies from university-affiliated medical centers (UAMC), were more likely to be accepted for presentation ( p < 0.05). In multivariate analysis, the variables: pediatric studies ( p = 0.01), prospective studies ( p = 0.005), randomized studies ( p = 0.06), and studies from UAMC ( p = 0.01) predicted acceptance of abstracts for presentation at the meeting. The overall publication rate was 25.1%. The publication rates 1, 2, 3, and 4 years after the meeting were 6.7%, 16.2%, 22.8%, and 25.1%, respectively. Multivariate Cox proportional hazards analysis showed that accepted abstracts ( p = 0.0003) studies reporting positive results ( p = 0.0015), and studies from outside the United States ( p = 0.036) were more likely to be published in manuscript form. Conclusions: The overall publication rate of abstracts reporting endoscopic research is 25%, lower than that in any published report from other medical societies. Abstracts from the United States were less likely to be published in full-manuscript form. Although there was no positive outcome bias for acceptance of abstracts for presentation at the meeting, there was bias toward publication of statistically significant results. Further investigations are warranted to determine the variation in the publication of research results according to country of origin and to determine factors that hinder publication of GI endoscopic research in manuscript form. (Gastrointest Endosc 2001;53:275-82.)

Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting.

Studies with positive results are more likely to be published in biomedical journals than are studies with negative results. However, many studies submitted for consideration at scientific meetings are never published in full; bias in this setting is poorly studied. To identify features associated with the fate of research abstracts submitted to a scientific meeting. Prospective observational cohort, with 5-year follow-up of all research submitted for consideration to the major annual 1991 US research meeting in the specialty of emergency medicine. All research abstracts submitted for consideration at the meeting for possible presentation. Characteristics associated with acceptance for presentation at the meeting and subsequent publication as a full manuscript. A total of 492 research abstracts were submitted from programs in emergency medicine and other specialties affiliated with 103 US medical schools. A total of 179 (36%) were accepted for presentation and 214 (43%) were published in 44 journals. Of the 179 abstracts accepted for presentation, 111 studies were published. Scientific quality of abstracts or prestige of the journal in which the study was eventually published did not predict either of these outcomes. The best predictors (by logistic regression) of meeting acceptance were a subjective "originality" factor (odds ratio [OR], 2.07; 95% confidence interval [CI], 1.13-3.89) and positive results (OR, 1.99; 95% CI, 1.07-3.84), and, for publication, meeting acceptance (OR, 2.49; 95% CI, 1.49-4.35) and large sample size (OR, 2.26; 95% CI, 1.23-4.31). Forty-nine percent (241) of abstracts did not report on blinding, and 24% (118) did not report on randomization. Acceptance and publication were both more likely for positive outcomes (P=.03). Funnel plots showed the classic distribution of positive-outcome ("publication") bias at each of the submission, acceptance, and publication phases. Meeting acceptance predicted publication with a sensitivity of only 51%, specificity of 71%, positive predictive value of 57%, and negative predictive value of 66%. Positive-outcome bias was evident when studies were submitted for consideration and was amplified in the selection of abstracts for both presentation and publication, neither of which was strongly related to study design or quality.

The justice and injustice of life events

An experiment employing a sample (N=280) of undergraduates from an urban university was designed to test the general hypothesis that the perception of justice and injustice in life events depends upon the relationship between two variables that are part of the stimulus situation: the valence of the person being observed (good or bad), and the valence of the outcome experienced by that person (positive or negative). The findings from both qualitative (analysis of spontaneous comments) and quantitative scale ratings supported the prediction that justice and injustice perceptions depend respectively on whether the signs of the person–outcome valences are the same or different. Two perceptual biases were revealed by the analysis. The first was a positive outcome bias: respondents rated as more just outcomes that were positive regardless of the goodness or badness of the person in the life event. The second was a justice bias: respondents in both measures found the just life events to be more just than unjust life events to be unjust. Finally the more religious respondents perceived the life events as more just regardless of the patterns of person–outcome valences than did the less religious, suggesting a third, religiosity bias. A number of theoretical implications and questions for future research were discussed, including the quantification of the hypothesis and its cross-cultural generality.