Quality of Performance Assessment Instruments for Educators in Higher Education: Implementation of Factor Analysis And Generalizability Theory

Abstract

Assessing the quality of learning in higher education is one of the efforts to ensure its standards. Typically, the assessment of the quality of learning implementation involves observation by multiple raters. This study aims to provide construct validity evidence and estimate the reliability of performance assessment instruments for educators in higher education. 225 second-year and third-year students from the Faculty of Education participated as raters, evaluating the performance of educators in their teaching practices. Forty assessment items were used to evaluate the performance of 19 instructors. Exploratory Factor Analysis (EFA) and Generalizability Theory (G-Theory) were employed to examine the quality of the performance assessment instruments. The EFA analysis resulted in the identification of five factors that contribute to educators' performance in teaching: (1) readiness and planning, (2) pedagogy and professionalism, (3) personality, (4) social relationships within the classroom, and (5) social relationships beyond the classroom, collectively explaining 67.671% of the variance. Of the 40 assessment items, 37 demonstrated construct validity, while three required revisions. These findings indicate the alignment between the instrument's factors and the formulated theory of teaching competence. The reliability of the measurements was estimated using G-Theory in RStudio, yielding a relative G coefficient of 0.88 for three raters. The D-Study results indicated that the instrument could be used to assess performance, with an estimated generalizability coefficient of 0.738, requiring a minimum of five raters for each person (educator) being evaluated. We recommend employing G-Study and D-Study to determine the number of raters involved in performance assessment as a means of cost and time efficiency in the evaluation process.