Abstract WP269: Induced Harking: Selective Reporting Bias After Changes in Instructions for Preparing Abstracts for the International Stroke Conference

Abstract

Background: Irreproducibility is a systemic problem in medical research, affecting 75-89% of major reported findings. An important cause of irreproducibility is hypothesizing after the results are known (HARKing): presenting a post hoc hypothesis according with findings as if it were a confirmed a priori hypothesis. Meeting abstract instructions may deter or encourage HARKing. Hypothesis: Compared with abstract instructions NOT indicating a "hypothesis" should be stated, instructions indicating THAT a "hypothesis" should be stated will increase the proportion of abstracts judged by field content experts to exhibit HARKing. Methods: We analyzed purposefully random-sampled abstracts from 3 ISC meeting years with highly forceful (2015), moderately forceful (2017), and no (2013) requirements for statement of a “Hypothesis” in abstract preparation instructions, regardless of whether a hypothesis statement was appropriate (confirmatory science) or inappropriate (exploratory science). Results: The 153 abstracts, 51 in each meeting year, equally sampled 17 ISC topic categories. Explicit “hypothesis” statements were most common in the highly forceful instruction year (35%), intermediate in the moderately forceful year (26%), and least frequent in the no requirement year (11%) (p < 0.02). Among hypothesis statements, 84% were deemed falsifiable, and 16% non-falsifiable. Of falsifiable hypotheses, 74% were adjudicated by two independent content experts as likely to be hypotheses generated after the results were known (HARKing). 0% of hypotheses were stated as pre-registered prior to study conduct. HARKing was most common in the highly and moderately forceful instruction years (24% each) and less frequent in the no requirement year (10%) (p < 0.01). Conclusions: Abstract preparation instructions suggesting use of a hypothesis statement likely promote hypotheses after the results are known. To improve reproducibility of stroke science, ISC abstract preparation instruction revisions may be desirable, recommending hypothesis statements only for the small proportion of stroke studies that are of confirmatory, rather than exploratory, design.