Bugs in the NRC's Doctoral Program Evaluation Data: From Mites to Hissing Cockroaches

Abstract

n 1995, the National Research Council (NRC) released Research Doctorate Programs in the United States, Continuity and Change. This report was described as containing an extensive, comprehensive, and systematic assessment of the quality, effectiveness (in training scholars), faculty productivity, and Ph.D. production of over 3,600 doctoral programs in 41 disciplines at 274 Universities (Departmental Rankings 1996, 144). The study offers a rich array of objective and subjective data that its authors expected to be useful to policymakers, academic administrators, faculty, and prospective graduate students (NRC 1995). Scholars quickly took advantage of the data to examine sources of high reputational evaluations, or to compare reputational scores with objective measures (Katz and Eagles 1996; Jackman and Siverson 1996; Lowry and Silver 1996; Miller, Tien, and Peebler 1996). Early studies, however, relied on the data as published in the report. Any massive project of this nature is bound to suffer some problems related to data quality or interpretation-perhaps minor, perhaps severe. About a year ago, the NRC made available on CD-ROM the