Validity and Reliability of Portfolio Assessment of Competency in a Baccalaureate Dental Hygiene Program

Abstract

This study examined the validity and reliability of portfolio assessment using Messick's unified framework of construct validity. Theoretical and empirical evidence was sought for six aspects of construct validity. Seven faculty raters evaluated twenty student portfolios using a primary trait analysis scoring rubric. A significant relationship (r = .81-.95; p < .01) between the seven subscales in the scoring rubric demonstrates measurement of a common construct. There was a significant relationship between portfolios and GPA (r = .70; p < .01) and the NBDHE (r = .60; p < .01). The relationship between portfolios and the Central Regional Dental Testing Service (CRDTS) examination was both weak and nonsignificant (r = .19; p > .05). A fully crossed, two-facet generalizability (G) study design was used to examine reliability. ANOVA demonstrated that the greatest source of variance was the scoring rubric itself, accounting for 78 percent of the total variance. The smallest source of variance was the interaction between portfolio and rubric (1.15 percent). Faculty rater variance accounted for only 1.28 percent of total variance. A phi coefficient of .86, analogous to a reliability coefficient in classical test theory, was obtained in the decision study by increasing the subscales to fourteen and decreasing faculty raters to three. In conclusion, the pattern of findings from this study suggests that portfolios can serve as a valid and reliable measure for assessing student competency.