Peer review on the Internet: A better class of conversation

Abstract

Peer review can be broadly defined as a method of evaluating and improving the content of scholarly work by subjecting it to the critical assessment of other workers in the same field. The principle is one of judgement by equals; the practice is attended by difficulties in finding properly qualified and unbiased reviewers who are willing to do the job fairly and accurately.Peer review is used by funding bodies to assess grant applications, by some institutions to guide employment and promotion decisions, and by journal editors to advise offer web sites or electronic versions. Some electronic journals do not use peer review, and others use a traditional peer-review system conducted off the Internet. However, interesting developments are scattered around the worldwide web. As with other aspects of the Internet, finding out what's going on in Internet peer review is not a precise science, and this is not a comprehensive survey.One of the attractions of early academic publishing on the Internet was that it avoided peer review and other production delays that surrounded the publication of them on what to publish. This last kind is now following scholarly publishing onto the Internet. Editorial peer review in the past has been secretive (because only the editor can see the whole process), narrow (because each article is usually reviewed by only one, two, or three people), and somewhat arbitrary (because the choice of reviewers is limited by the editor's knowledge and the time constraints of the potential reviewers). Some of the new Internet models of peer review seek to overcome these weaknesses, and may offer a more cost-efficient system. Panel 1 summarises some of the pros and cons of electronic peer review.Panel 1Features of electronic peer reviewTabled 1FeatureThe good newsThe bad newsAll or part of review conducted in public forumRapid publication. Wide participation inprocess. Authors have opportunity to use reader feedback to revise article. Late-identified errors are corrected in the article, not in a separate document.No filtering of errors before publication. Reduced pressure on authors to meet the standards of peer review (because they have already achieved publication). Raises questions about archiving versions of the article, and which version is authoritativeMore reviewers per articleBroadens the range of expertise in review. Reduces likelihood of bias. Improves chances that errors will be corrected.Requires more time to assess reviewers' comments. May multiply dissonance, to the confusion of editor and author.Review conducted as debateAuthors placed on even footing with reviewers. Reviewers provide check on each other's performance. Editors' positions made explicit through their participation in and control of the debate. Decision-making process automatically recorded.Debate may wander beyond the purposes of peer review. Debate may lead to arguments and confrontation. Editors may feel a loss of authority attending their more public decision-making.Review using structured review formsReviewers specifically requested to address key points in their review. Reviews collected in database ready for further analysis.Inflexible. Structure may inhibit some reviewers or even bias the content of reviews. Computer programming required.Centrally recorded reviewsImproved monitoring of reviewer performance. Research into peer review.Processing the extra data adds to the time spent managing peer review.Not all electronic peer review systems possess all these features. In this summary, some people might argue about the categorisation of good and bad news but the real purpose is to suggest the issues that are raised by some of the innovations of electronic peer review. Open table in a new tab What is peer review on the Internet?Although experiments began a long time ago (see Panel 2), peer review on the Internet is still in its infancy. Most learned journals are still based in print, even if they now ideas. The Internet made self-publishing possible on a grand scale. For some writers this represented a new freedom of communications. The result, for many readers, was e-mail boxes and file servers full of formal and informal articles, comments and digressions–all unsorted, unevaluated, and unindexed, with little or nothing to separate the wheat from the chaff. None of this diversity is going to disappear, but the demand for selective and reliable information sources suggests that peer review has a bright future on the Internet.Panel 2An early attempt at electronic peer reviewIn 1976 workers at the New Jersey Institute of Technology began to develop the Electronic Information Exchange System (EIES). The system, based on telnet, introduced four publications over the next 4 years. 1Rathie SE Electronic journals and peer review: perils and promises.1994http://www.cs.ubc.ca/spider/rathie/elecpub/paper3.htmlDate: 1994Google Scholar, 2Turoff M Hiltz SR The electronic journal: a progress report.J Am Soc Information Sci. 1982; 33: 195-202Crossref Scopus (59) Google ScholarThe first was a read-only newsletter for communications from the EIES to its users. The second was an open journal, Paper Fair, to which all readers could contribute articles and responses to articles. This facility allowed authors to publish eprints (electronic preprints) for comment before seeking a publication in print, and was successfully used for that purpose. Today the High Energy Physics Eprint Server (http://xxx.lanl.gov) has a similar function, but in medicine such an activity might cause print journal editors to invoke the Ingelfinger rule and refuse to publish3Kassirer JP Angell M The Internet and the Journal.N Engl J Med. 1995; 332: 1709-1710Crossref PubMed Scopus (82) Google Scholar (see panel 3). The third journal, Mental Workload(on “human-machine interfaces in the operation of complex systems”),2Turoff M Hiltz SR The electronic journal: a progress report.J Am Soc Information Sci. 1982; 33: 195-202Crossref Scopus (59) Google Scholar was modelled on a traditional peer-reviewed journal. Articles were sent by e-mail to the editor, who e-mailed them on without author-identifying information to the reviewers, who replied also by e-mail. The journal also allowed readers to comment; comments and authors' responses could be linked to articles, and authors could continue to revise their articles after publication.The fourth journal, Legitech, introduced a new model in which readers contributed questions and responses, ultimately to produce a structured “brief” summarising the collected information.2Turoff M Hiltz SR The electronic journal: a progress report.J Am Soc Information Sci. 1982; 33: 195-202Crossref Scopus (59) Google Scholar This journal was produced and used by a small group of scientific advisers to government, who found it an efficient means of working collaboratively at a distance and disseminating information.These experimental journals were created over a decade before the launch of the worldwide web, at a time when computers were far less user-friendly than they are now (no pictures, and no point-and-clicks). Within the limits of this technology, the EIES described many of the possibilities for Internet publication and peer review: a structured forum for peer-to-peer communications; an open journal for author-controlled publication and postpublication review; and an editorially controlled journal with closed prepublication review and open postpublication review.The least successful project was Mental Workload, the one journal based on traditional peer-review methods, which failed to compete with established print journals of higher academic status. Those involved in the project concluded that, irrespective of any technical advantages of electronic publishing, the established print journals would continue to be the preferred publication medium of academic authors so long as publication status remained important in the academic reward system.2Turoff M Hiltz SR The electronic journal: a progress report.J Am Soc Information Sci. 1982; 33: 195-202Crossref Scopus (59) Google Scholar So far, so true.Examples of Internet peer reviewThe traditional, but wiredAt the simplest level, the Internet can be used to accelerate traditional peer review processes. Several journals, particularly those published electronically, are using the Internet for peer review in this way (one example is Conservation Ecology, http://www.consecol.org/Journal/, another is Journal of High Energy Physics, http://jhep.sissa.it/). The article to be reviewed is transmitted to the reviewers by e-mail, or made available to them on a password-protected website, and the reviewer's comments are returned to the journal electronically. Checklists and structured comment forms, used to ensure that reviewers address particular questions in a comprehensive review, can be implemented as online forms, with the advantage that the responses are automatically collected in a database (for an example of an online review form, see http://www.mja.com.au/guides/cfs/cfscomment.html).This kind of electronic peer review saves paper, time, and communication expenses, but does not change the intellectual process. Some would say that this is a good thing; others that it misses the opportunity to transform peer review into a fairer system. Its disadvantages compared with paper-based review systems are technical. Although the simplest versions of this model can run on e-mail alone, any complex system requires computer programming to create and maintain online reviewing forms and an associated database. What may be more difficult to arrange is the necessary level of computer skills among the peer reviewers: this is particularly a problem in medicine, where skills with e-mail and web browsers have yet to permeate the profession. Nonetheless, rough estimates of the Internet readiness of reviewers used by the Medical Journal of Australia (MJA) suggest that 90% have access to e-mail, 60% have some level of “web awareness”, and that about 30% have used the Internet to find medical information. These percentages will rise rapidly over the next few years.Post-publication commentaryPeer review has never ended with publication. The letters to the editor published in journals carry a great deal of peer review, but these corrections and amplifications appear in print weeks or months after the original article and remain permanently separated by a thick pile of pages, even if indexing services do their best to cross-reference.These delays and discontinuities in post-publication review are easily remedied on the Internet. A good example is Psycoloquy, an interdisciplinary journal of psychology and related sciences that has been published electronically since 1990, appearing originally as an FTP file archive and e-mail list, but now also available in hypertext (http://www.princeton.edu/~harnad/psyc.html).In Psycoloquy, authors are able to publish preliminary reports of new data or ideas on which they wish to receive rapid commentary from their peers. The original article, commentary, and authors' responses are all archived and linked together. The process is designed to enhance the development of the authors' ideas and data, and can be thought of as a kind of peer-review development of an as-yet-unwritten final report.Although Psycoloquy's own peer-review processes are traditional, its innovations set several precedents for the development of open peer review on the Internet. •It encourages the publication of work at a stage that many journals would consider incomplete, specifically for the purpose of gathering the input of others while there is still time to incorporate improvements in ongoing studies. (It has been suggested that medical journals could enhance clinical trials by peer reviewing and/or publishing the protocols before the study is performed.)•It demonstrates the principle that effective peer commentary can be elicited via the Internet.•It shows that “article+commentary” can be a rich form of scholarly presentation for readers, and that the electronic dialogue (faster than print but with the discipline of the written communication)4Harnad S Post-Gutenberg galaxy: the fourth revolution in the means of production of knowledge.Public-Access Computer Systems Review. 1991; 2 (Accessed March 3, 1998): 39-53ftp://ftp.princeton.edu/pub/harnad/Harnad/HTML/harnad91.postgutenberg.htmlGoogle Scholar can be a powerful form of intellectual discourse.Some online journals, particularly in cultural studies, blur the bounds between dialogue and article completely. One example is Rhetnet, “a cyberjournal for rhetoric and writing” (http://www.missouri.edu/~rhemet/). In this journal, the “articles” have diverse origins, sometimes appearing as individual contributions, sometimes as article and commentary, sometimes coalescing out of online conversations. Rhetnet does not seem to have a peer-review process so much as be a peer-review process. It is a method quite alien to biomedical journals, but not unlike a scientific meeting or the consensus processes of a working group.The collaborative style of Rhetnet works well for the discussion of ideas, but it may not appeal to researchers who are seeking to maximise the status of their latest research reports. Even when effective informal methods of communication exist, the academic reward system encourages publication through the formal peer-review processes of well-established journals.Post-publication review systemsOpponents of prepublication peer review and the control exercised by journals over what is published have suggested that authors should publish their own articles in an open archive on a web server.5Ginsparg P Winners and losers in the global research village.1998http://xxx.lanl.gov/blurb/pg96unesco.htmlDate: 1998Google Scholar, 6LaPorte RE Marler E Akazawa S et al.The death of biomedical journals.BMJ. 1995; 310: 1387-1390Crossref PubMed Scopus (104) Google Scholar, 7Odlyzko AM Tragic loss or good riddance?.The impending demise of traditional scholarly journals. 1998; (Accessed March 3, 1998)http://www.arms.org/notices/199501/forum-odlyzko.htmlDate: 1998Google ScholarSuch a system, without any peer-review mechanism, has worked at the High-Energy Physics Eprint Server (http://xxx.lanl.gov) for some years now. However, high-energy physics is a small and concentrated research specialty, with an unusually coherent and technically literate group of authors who are also simultaneously the main readers2Turoff M Hiltz SR The electronic journal: a progress report.J Am Soc Information Sci. 1982; 33: 195-202Crossref Scopus (59) Google Scholar According to Paul Ginsparg, founder of the eprint archive, the peer-review processes of the paper physics journals added relatively little value to the literature, and publication was unconscionably slow (and yet the physics journals still continue).The situation is different in the wide and varied field of medicine, where most readers are necessarily cautious about the quality of information they read, and would in any event be swamped if all articles were published in one archive. To serve the function of selecting the high-quality articles in such an archive, it has been suggested that all readers could act as reviewers and give articles a quality score, with the results being displayed on the server.6LaPorte RE Marler E Akazawa S et al.The death of biomedical journals.BMJ. 1995; 310: 1387-1390Crossref PubMed Scopus (104) Google Scholar This system involves no guarantee that the scores are given by appropriate people, and invites the dubious proposition that the most popular articles are the most scientifically correct. Nonetheless, such a system for biological/biomedical articles was set up by a group called the World Journal Association (World Journal Club) in the USA. For some months the site remained nearly empty, but eventually it published dozens of articles on biological and medical topics. However, the attached scoring system went unused, and the published access data for each article indicated that there were few readers. When I last looked, the site had disappeared.The chief practical problem with democratic scoring systems as a replacement for peer review is that if only a few readers vote, the resulting score is not representative. In our experiments in open electronic peer review at the MJA (described below), we found that less than 2% of readers who were invited to comment on articles actually did so. However, we were asking for comments, not scores, and for this purpose a 2% response rate was sufficient to yield some valuable contributions. The key to successful open systems is the quality, not the quantity, of feedback. A simple score does nothing to improve or correct articles, provide new insights, or teach authors how to improve their writing or research methods.Extensions to traditional reviewFrom March, 1996, to July, 1997, the MJA invited its authors and peer reviewers to participate in a trial of extended peer review on the Internet. Articles that had been accepted for publication after standard peer review were published on the worldwide web together with the peer-review comments. Readers were invited to submit further comments, which could be used by the authors and editors in revising the article before the fmal version was prepared for publication in print. As I said above, few readers took up this opportunity, but seven of 56 articles that entered the trial were modified by the authors in response to readers' comments. Readers' comments contributed new perspectives that had not been elicited in the standard peer-review process, and there were other benefits as well •Articles published in the trial appeared in their preliminary form on the Internet about 10 weeks earlier than usual, and in their final form in print on the same schedule as other articles. In other words, we achieved enhanced peer review at the same time as rapid publication.•Readers gained an unprecedented insight into peer review.•Authors and editors received new feedback on the reception of articles and a new line of contact with readers.The Cochrane Collaboration (http://hiru.mcmaster.ca/cochrane/default.htm) has instituted a similar system for online comment on the “systematic reviews of the effects of healthcare interventions” published in the Cochrane Library. These articles have already undergone traditional peer review (the reviews are not published with the articles), but it is hoped that the online comment system will assist in meeting the “commitment to ensure that Cochrane Reviews are maintained through identification and incorporation of new evidence” (http://hiru.mcmaster.ca/cochrane/stratp96.htm).The Cochrane Library also publishes protocols for planned systematic reviews, and these are also available for online comment and further development before the review is undertaken.The open-discussion modelAmong purely electronic journals, there have been experiments in conducting peer review as an open process on the Internet. Some electronic journals (eg, the Australian Electronic Journal of Nursing Education, http://www.csu.edu.au/faculty/health/nurshealth/aejne/) are prepared to publish unreviewed “preprints”, which are then open to comment by readers with the possibility of revision and a change in status to “reviewed article”. A potential difficulty with this approach is deciding what to do with preprints that go unreviewed, or that are condemned by review. Should these articles remain in the journal archive? If so, does this have negative implications for the journal's standards? If not, what can the authors do with an article that has been published, but has now disappeared from public access?The Electronic Journal of Sociology (EJS), published in Canada (http://www.sociology.org), advertised that “Our pre-print feature and web discussion provides the opportunity for interactive discussion amongst the journals' editors, readers and authors. We pride ourselves in an open, transparent, and democratic review procedure”. However, the EJS fell back on a more traditional peer-review process after the editor found “Very little reader, or even author, interest … we dropped the idea temporarily but still plan on trying again down the road—-perhaps when the technology matures” (Mike Sosteric, personal communication).One of the most (possibly overly) advanced examples of the open-discussion model is the Journal of Interactive Media in Education (JIME, http://www-jime.open.ac.uk/), published by the Knowledge Media Institute of the Open University in the UK. The editors have developed a sophisticated system for organising online commentary and revision of articles. Points in articles that are the subject of discussion are marked with a special symbol that links to the discussion pages; comments are organised into threads of argument and marked with “thumbs-up” or “thumbs-down” symbols to indicate the essence of the commentator's assessment.JIME has a process of editorial selection before publication, but all peer reviewing and article revision is published online. This implies that no articles will be rejected after peer review, an approach that is probably sustainable within the narrow specialisation of this journal (JIME publishes less than one article a month), but one that would not be acceptable for a general medical journal.In theory, readers can select a short path through the review material or elect to participate in an extended debate. As a visitor to the site I found their scheme overly complicated (the interface uses two browser windows, one for the article and one for the commentary, with six frames in each). However, this may not be a problem for regular readers with a fast network connection.Open-review systems modify the nature of the published article and raise some tricky questions for publishers (panel 3).Panel 3Tricky issues in open peer reviewThe Ingelfinger rule This rule, first announced by Franz Ingelfinger when he was Editor of the New England Journal of Medicine (NEJM),8Ingelfinger FJ Shattuck Lecture—the general medical journal: for readers or repositories?.N Engl J Med. 1977; 296: 1258-1264Crossref PubMed Scopus (27) Google Scholar states that a journal will not publish a research report that has already been substantively published elsewhere. Ingelfinger saw it as a necessary defensive measure for peer-reviewed journals if their “newsworthiness” was not to be stolen by the non-peer-reviewed press, which can publish much more quickly. The rule is applied by most medical journals, and the current editor of the NEJM has indicated that it will apply to prior publication on the Internet.3Kassirer JP Angell M The Internet and the Journal.N Engl J Med. 1995; 332: 1709-1710Crossref PubMed Scopus (82) Google Scholar Thus authors who distribute eprints of an article risk foregoing the opportunity to publish in a peer-reviewed journal. Similarly, journals that plan to publish articles online and then conduct an open review process must address the implications of the rule. Can such a journal reject an article after review? If it does, the authors will probably find that they cannot submit it elsewhere, if it does not, the journal has abandoned peer review as a means of selecting articles, having effectively agreed to publish anything it has reviewed. Most medical journals could not function on that basis.Publicity and embargo Print journals usually provide advance information to news media on the understanding that the articles are under embargo until the date of publication. They can then expect a welcome burst of publicity on each day of publication. Articles released for open review on the Internet before publication in print cannot be placed under embargo in the same way. The moment of publicity shifts from the publication of the journal issue to the day the article appears on the Internet, perhaps resulting in reduced or more scattered publicity.Role of the editor Traditional peer review places the journal editor in a powerful and reasonably private position; open review promises to increase the direct scrutiny of the editor by authors and reviewers. Given the decisive influence that editors have over what is published and how it is reviewed, revised, and presented, this is a good thing, but not all editors will like it. However, editors will still represent the interests of the readers-a group not necessarily represented by authors or reviewers. What version is that? Articles published in open-review systems may undergo further revision during the review process. This system presents difficulties for the historical record and when citing references. A partial solution is to cite the date at which an article was accessed—but this is little help if that date's version has passed into the ether without being stored in an accessible archive. Protocols for signposting the changes that articles have undergone are required, plus a rational scheme for determining how much of the historical record needs to be preserved.Anonymous review Open peer-review systems on the Internet can be implemented to require reviewers to identify themselves to both editor and author, to the editor only, or to neither. Arguments for identification are that conscientious reviewers ought to be prepared to “stand by” their reviews, that it helps to expose conflicts of interest, and that a fair debate can take place when the authors know who it is they are addressing. Arguments for anonymous reviewing are that personalities are irrelevant and potentially distracting, and that identification may instigate or inflame bad feeling. Trials are needed to establish whether there is any correlation between identification of the reviewer to the author and the quality of the review.Staged-discussion modelIn a new trial now beginning at the MJA (outlined in the figure), peer review is conducted as an online discussion between the editors, the authors, two peer reviewers specifically commissioned to review the article, and a small “virtual college” of peers who are invited to observe and contribute if they wish. If the article is accepted for publication, the article and discussion moves into the publicly accessible part of our web site, and an open discussion period follows until the article is finalised. At the MJA, the final form will be the one that is printed, but it would be possible in a purely electronic journal for articles to be in a permanent state of development and change (as they are in the Cochrane Library).The first stage of discussion ensures that articles reach a certain standard before they are published. The procedure integrates peer review with the electronic publication process, allowing rapid publication after rigorous peer review. It preserves the virtues of our traditional peer-review system, but brings a new openness to the procedure that we hope will enhance its accuracy and fairness. The two commissioned reviewers continue to carry the traditional responsibility of ensuring that the article is comprehensively reviewed; the extra participants may provide broader perspectives, catch more errors in the article or the peer-review process, and smooth potential biases. The openness of dialogue offers a more democratic and less authoritarian model of peer review, which might improve the authors' satisfaction with the procedure. Reviewers too are better informed about the impact of their comments. Editors are required to make their decisions out in the open, which might be unnerving, but the hoped-for gain is better advice, demonstrated accountability, and, ultimately, an improved service to readers through providing better articlesWill it work? Ask me in a year. Newsgroups Milverton WallaceElectronic mail (e-mail) is the glue of network communication and the workhorse of the Internet. E-mail allows anyone to communicate with anyone else on the net—instantly. E-mail not only lets you communicate one-to-one but one-to-many, which is where the idea of the mailing list comes in.The mailing list is a community of persons with like interests. A mailing list is usually founded and maintained by an individual (known as the list-owner). List-owners take differing views about the management of their lists. Some allow free, unhindered flow of discussion, while others actively intervene to block postings that are off-topic, to enforce rigorous criteria for list-membership, and to filter postings to maintain the integrity of the list.Mailing lists are maintained by special software known genetically as listservers. The software automatically sends a message from one member to all others on the list via e-mail. Anyone on the list can reply to the message in similar fashion. There are thousands of lists in the form of digests, electronic journals, and discussion groups. A search of TileNet (http://www.tile.net/) found 197 medically related lists.Electronic conferences are a variant of mailing lists, but are larger, less restrictive, and with a narrower subject focus. The primary site for electronic conferences is Usenet, (USEr NETwork) where conferences are called newsgroups. Usenet was started in 1979 at Duke University as a means of exchanging information among computers running the Unix operating system. The development in 1986 of NNTP (network news transpsort protocol) allowed transfer of news messages via TCP/IP (transmission control protocol/Internet protocol)—the basic piece of software that controls Internet signal traffic—thus integrating Usenet into the Internet. This development meant that Usenet news articles could be accessed and stored on computers running operating systems other than Unix. Usenet newsgroups are maintained on special servers and operated according to agreed Internet standards and protocols. There are about 20 000 newsgroups, organised in an hier