Using multiple program impact analysis to document institutional effectiveness

Abstract

The age of institutional effectiveness has arrived. Accrediting agencies, state agencies, governing boards, communities, students, and parents have increased the pressure on institutions of higher education to document how educational programs contribute to important student outcomes (Ewell, 1987). Does college make a difference in what students learn? How will we know? What evidence will we need to convince our external public that students learn as a result of our efforts? In the past, the assessment movement urged higher-education institutions to document that students achieve important educational outcomes. Today, we must document that these outcomes a r e a result of the institution's educational efforts. This new emphasis is called institutional effectiveness (Astin, 1991; Gardiner, 1989; Nichols, 1989). The assessment of institutional effectiveness requires that we not only document what students have learned but how and under what conditions the learning occurred. To be effective, institutions must link the attainment of student outcomes to the participation of students in specific programmatic efforts while controlling for the students' initial background characteristics (Astin, 1991; Pascarella, 1986). Pascarella (1986) referred to this problem as determining the "net effects" of programmatic efforts. In reviewing the college impact studies summarized by Pascarella and Terenzini (1991), we found relatively few studies that examined the impact of rnore than one program intervention on college student outcomes. Traditionally, individual educational programs have been evaluated in isolation from the larger context of the institution's total efforts. However, students rarely participate in only one educational program. Rather, most students participate in multiple educational programs with common goals such as increased first-year