Reliability and Validity of Student Evaluations: Testing Models Versus Survey Research Models

Abstract

Student evaluations of faculty teaching have four recognized functions. They provide diagnostic feedback for faculty, an evaluative tool for personnel decisions, information for students, and a subject for academic research. Regardless of how the evaluations are used, the user must be concerned about the reliability of the evaluation instrument and the validity of the student responses. Faculty members have a natural selfinterest in the reliability and validity of student evaluations when used as an evaluative tool for personnel decisions. Much of the existing literature on reliability and validity conceptualizes student evaluations as a test, whose reliability is to be estimated and whose validation is to be determined. Student evaluations, however, resemble public opinion surveys more than they do objective tests. Consequently, an alternative approach is to conceptualize student evaluations as survey research rather than as tests. Such a conceptual framework provides new insights into student evaluations and an entirely different dimension to the question of reliability and validity. Personnel systems in American colleges and universities, as in other organizations, must solve the problem of allocating organizational resources to reward and reinforce productive behavior. They must grapple with the universal problem of defining, measuring, and rewarding merit. However, academic personnel systems differ in that they do not share a universally accepted of who is to evaluate merit and how it is to be done. Instead, evaluation in higher education generally uses mixtures of three models for allocating rewards, two of which are commonly found in other organizations. The most common model for evaluating merit is the supervisorsubordinate model, in which the performance of a member is appraised by a supervisor/superior. Although there are a number of different approaches and instruments (i.e., trait-oriented or behaviororiented, comparison or forced choice), the defining characteristic is compatibility with the formal organizational hierarchy. Most textbook treatments of performance appraisals restrict themselves almost exclusively to this model, and a large body of normative literature exists in human resource management and empirical research in organization theory focusing on this model.' All universities use this model to the extent that university administrators are involved in personnel decisions allocating organizational rewards. The second common model for evaluating merit rejects the hierarchical framework in favor of some variation of peer evaluation. More commonly known as the professional model,2 it is based on the premise that the performance of members of certain professions can only be adequately evaluated by other like professionals.3 Peer evaluation, self-governance, and tenure are central to the principles of the American Association of University Professors, and are found to some extent in the accreditation requirements of numerous accrediting bodies.