Breathing New Life into Peer Review: It's Time for a Commission on Peer Review Criteria and Application

Abstract

Peer review is a well-accepted practice among scientists, scholars, and others working in the academy, medicine, and the arts. The process can involve a blind review, such as the means employed by Educator and other refereed journals to evaluate the merit and contribution of submitted articles free of the bias of knowing the author or her previous work. Other times, peer review considers a body of work of a clearly identified individual seeking promotion or tenure based on the attainment of a high level of contribution to a field. For each, peer review can serve an irreplaceable function. And for each, peer review enables professionals to appraise the work of colleagues and in that manner to take responsibility-rather than ceding it to less-qualified bodies or agencies or to the ravages of partisan or religious motivations antithetical to academic pursuitfor the direction and quality of their profession, discipline, or artistic community. Evaluative peer review provides admission to conferences, journals, and juried art and film festivals, as well as the basis for the monetary and status rewards of annual faculty review, promotion, and tenure. Does it work? Pretty well, especially when peer review is based on substantive, as well as procedural, due process-that is, in addition to clear rules that establish who and what will be reviewed, the frequency of review, and the qualifications of reviewers, there is an implicit assumption that peer review is based upon explicit criteria that are applied consistently. These six criteria-who will be reviewed, what will be reviewed, the frequency of review, the qualifications of reviewers, the explicit criteria of review, and the consistent application of criteria-are not easy standards to maintain. Despite the high stakes outcomes, there is little in the way of careful systematic research to ease the concerns of faculty who don't quite trust a process often conducted behind closed doors. Multiple levels of review provide an important check and balance for the often secretive nature of peer review. When provosts review the work of university tenure committees that have reviewed the findings of college-level committees that have reviewed the recommendations of departmental tenure committees; and when journal editors provide independent review of the findings of multiple editorial board reviewers; then a culture of trust develops around a process of inter-coder reliability. Challenges to Effective Peer Review Even so, the system is not perfect. Whether rooted in perception or fact, there are rough edges that deserve mention. The first seems to involve who is doing the reviewing. Who is your peer? Next, faculty may be splintered in their identification of the breadth and substantive components within a university's mission that may be well served by peer review. Do we define our mission as educators narrowly or broadly? The third challenge, especially prevalent in the assessment of teaching, concerns the presence or absence of criterion consistency. Let's begin with who. Some professional schools have long operated autonomously within the tenure and promotion process. In response, provosts and university faculty are beginning to ask whether peer review conducted only at the professional school level discounts the larger values, substantive knowledge, and community standards of the university as a whole. The same thing occurs when distant cousin cohorts of social scientists, critical theorists, and specialized editorial and creative faculty in journalism and mass communication programs question whether the methodological stranger in the office down the hall should be considered a peer when the stakes are high. Can historians or legal scholars adequately evaluate the work of broadcast journalism or quantitative methods faculty? Are faculty drawn from the communication industries and faculty steeped in doctoral scholarship able to serve one another for purposes of tenure or professional development as peers? …