Reliability and Validity of Judgments of Applicant Essays as a Predictor of Academic Success in an Entry-Level Physical Therapy Education Program

Abstract

Currently physical therapy admissions committees may use the essay without scientific justification to help predict an applicant's professional academic success. We retrospectively examined the intertester reliability and predictive validity of judgments made by an admissions committee—on the basis of applicants' essays—about applicants' future academic success in our physical therapy program and compared our physical therapist (PT) and non-PT members of the admissions committee in terms of intertester reliability and predictive validity scores. Judgments were made about 52 students who graduated from our program between 1985 and 1988. Six members of our admissions committee served as examiners of the essays: three were PTs, and three were non-PTs. We used the κ statistic to estimate intertester reliability and predictive validity of the judgments. For intertester reliability, the κ values ranged from 0.19 to 0.25 for non-PTs and from 0.15 to 0.42 for PTs. For predictive validity estimates, κ coefficients ranged from 0.15 to 0.42 for non-PTs and from -0.04 to 0.23 for PTs. On the basis of these estimates, our current essay is neither reliable nor valid when used alone to predict an applicant's potential for future academic success.