The Reliability and Validity of Using Regression Residuals to Measure Institutional Effectiveness in Promoting Degree Completion

Abstract

A relatively simple way of measuring institutional effectiveness in relation to degree completion is to estimate the difference between an actual and predicted graduation rate, but the reliability and validity of this method have not been thoroughly examined. Longitudinal data were obtained from IPEDS for both public and private not-for-profit 4-year institutions (n = 1496). Hierarchical panel regression was used to predict 4- and 6-year graduation rates based on structural, demographic, financial, and contextual attributes. A direct effects model yielded effectiveness scores that were highly correlated between consecutive data years (r = 0.65–0.80), which indicated acceptable to good test–retest reliability. A test of convergent validity indicated that effectiveness scores were positively associated with students’ perceptions of a supportive campus environment (r = 0.32–0.45). A test of discriminant validity revealed relatively small correlations between effectiveness scores and institutional attributes, such as educational expenditures (r = 0.07–0.16). The modeling of interaction effects in relation to institutional type marginally improved the validity of effectiveness scores among public but not private institutions. The results suggest that correct model specification can yield residual scores that reliably and validly measure institutional effectiveness in promoting timely degree completion.