Quantitative assessment of journal quality and consistency: Comment on Winker

Abstract

Professor Kenneth Winker’s (1983) study is commendable for the light it seeks to shed on the differences in the rankings of journals by members of the Academy of Criminal Justice Sciences (ACJS) and the American Society of Criminology (ASC). However, beyond establishing the presence of such differences, his study (or, more properly, DeZee’s (1980) study from which the data were extracted) is severely limited by grievous methodological deficiencies which produced distortions in the data. These were then exacerbated by ‘Winker’s choice of an inappropriate means of analysis. In particular, his conclusions do not adequately represent the rankings within each professional association of either journal quality or consistency, and his conclusions are particularly unreliable regarding the most highly-regarded journals. Winker’s basic method of analysis was drawn from Glen’s (1971) well-known study of journal prestige within sociology, in which the American Sociological Review (ASR) was given a weight of 10. the absence of prestige was given a weight of 0, and respondents were asked to assign weights which they considered appropriate to 62 journals. Since ASR is almost universally recognized by sociologists as the most prestigious of their journals. the result was a prestige weighting of journals on a zero to ten scale. Winker adopted the same approach to assessing evaluations of journals within ACJS and ASC, with the substitution of the presumed most highly regarded criminal justice journal, the Journal of Criminal Law and Criminology ( JCLC). However, Winker’s results were profoundly different from those of Glen, for JCLC was not consistently the highest ranked journal in either association. For example, in terms of quality it was ranked fifth and ninth in ACJS and ASC respectively, indicating that some respondents rated other journals more highly. Consequently, Winker’s respondents weighted journals not on a zero to ten scale, but on a scale from zero to some unspecified figure. It is here that difficulties arise. For a single enthusiastic respondent could have an immense impact upon a journal’s mean weight by giving it a very high weight, e.g., a weight in excess of 100. Indeed, by selecting a weight which was sufficiently great, a single repondent could have guaranteed that his/her favorite journal would receive top billing, i.e., first ranking in Winker’s study. To see how this could have occurred, consider the following perhaps extreme example. If 19 respondents