Validity and generalizability of student ratings from between-classes and within-class data.

Abstract

This article examines the generalizability and validity of student ratings by studying within-class and between-classes correlations of ratings with other variables for regular faculty teaching lecture courses as well as for graduate assistants teaching recitation sections. Results indicate that most ratings are highly generalizable but only some are related to learning and that certain aspects of both generalizability and validity vary with the instructor's role and with the level of data. The implications of these findings for the evaluation of teaching are discussed with reference to two alternative paradigms: construct validity and criterion development. Many prior studies have examined aspects of the validity and generalizability of student ratings of instruction. For example, the validity of student ratings as indices of student learning has been addressed by Remmers, Martin, and Elliott (1949), Elliott (1950), Rodin and Rodin (1972), Frey (1973), Sullivan and Skanes (1974), Doyle and Whitely (1974), and Centra (Note 1). Findings have ranged from high negative to high positive correlations, with the majority indicating a statistically significant but very modest relationship between student ratings and tested student learning. Generalizability has been even more frequently studied. DeWolf (Note 2) lists 76 student characteristics that have been examined as possible correlates of student ratings, and Doyle (1975) cites more than 50 generalizabil ity studies. Again the findings are inconsistent, showing sometimes positive, sometimes negative, and sometimes zero correlations between student ratings and various student characteristics. One important explanation of these inconsistencies may be the failure to study a variety of validity and generalizability variables simultaneously and in a single paradigm. Multivariate strategies reduce the risk of focusing too much attention on iso