Optimizing Peer Review of Software Code—Response

Abstract

Sliz and Morin question the feasibility of our recommendation to both peer-review computer code and release it, and they proffer an alternative: postpublication community review and stronger procedures and facilities for dealing with corrections and retractions of published results. These are not incompatible. Encouraging the broader scientific community to inspect computer code postpublication would help in identifying scientific errors currently unnoticed in the scientific literature. Improving the process of corrections and retractions would have positive benefits far beyond this issue. However, neither negates the need for pre-publication review of code. The scientific publishing process relies on prepublication peer review as a filter for robust results. This is so because, regardless of the strength of processes for dealing with corrections and retractions, putting “the genie back in the bottle” is always going to be a difficult task after a result has been reported in the literature. At a minimum, code needs to be available to reviewers should they choose to scrutinize it. Moreover, prepublication review of code need not necessarily rely on the current review system. Just as English-language editing services have emerged to ensure a minimum standard of accessibility of articles in many major journals, so might software-reviewing services provide a stamp of approval that code actually implements the algorithm reported in a paper. Indeed, in the commercial sector, software escrow providers routinely provide full verification services to companies purchasing (or investing in) business-critical software [e.g., ([ 1 ][1])], and the approaches used by such companies might provide pointers for a new model for academic software verification services. Of course, verification of software is just the first essential step in the process, with by far the more challenging issue being software validation. Addressing this issue, together with the equally pressing issue of uncertainty quantification in complex [computational] models, has been the focus of intensive research efforts in other scientific disciplines ([ 2 ][2]). These efforts might provide a good starting point for equivalent efforts in the life sciences. 1. [↵][3] Iron Mountain, “How verification services fortify your software escrow solution” (Iron Mountain, 2011). 2. [↵][4] National Academies Press, Assessing the Reliability of Complex Models: Mathematical and Statistical Foundations of Verification, Validation, and Uncertainty Quantification (National Academies Press, Washington, DC, 2012). [1]: #ref-1 [2]: #ref-2 [3]: #xref-ref-1-1 View reference 1 in text [4]: #xref-ref-2-1 View reference 2 in text