Background: Open science improves methodological rigor, limits duplication, and helps researchers build upon existing findings. Yet despite the importance of open science in cardiovascular outcomes research, it may be limited by difficulties in accessing statistical code after publication. We evaluated the ease of obtaining statistical code from published research in high-profile journals. Methods: We identified all studies published in 6 general medical journals that utilized Medicare Parts A and B data in 2017 and 2018: Annals of Internal Medicine, BMJ, JAMA, JAMA Internal Medicine, Lancet, and New England Journal of Medicine . We focused on studies utilizing national Medicare data given their widespread use in outcomes research. We targeted recent studies to minimize barriers to accessing code as research teams change over time. Using these criteria, we identified 51 research articles with 41 unique corresponding authors. We contacted these authors to request statistical code for their publications via email up to 3 times at 2-week intervals. When code was provided to us, we assessed its integrity and completeness. Results: Of 51 studies we identified, 3 had full or partial code already publicly available. Of the remaining 48 studies, 7 codes (corresponding to 6 authors) were sent to us, leading to a total of 10 out of 51 articles with full or partial code availability (Figure 1). Overall, we received responses from 19 of the 41 authors. Thirteen corresponding authors declined to share code for numerous reasons. Reasons given for not sharing code included researchers leaving prior positions, analysts leaving research teams, and likelihood of significant delays due to seeking permission from research sponsors. We received no responses from 22 of 41 corresponding authors (Figure 1). Conclusion: We found significant barriers to accessing statistical code after publication in research articles utilizing Medicare data in general medical journals, and we were unable to access statistical codes in over 80% of cases. The inability to obtain codes associated with scientific analyses may represent an important challenge for reproducibility in cardiovascular outcomes research.
Read full abstract