Performance Measurement for Accountability: Lessons from the School-to-Work Experience

Abstract

One component of the School-to-Work Opportunities Act was the collection of data on program activities and participants. Ms. White and Mr. Medrich look back on the process that was used, explore lessons learned, and offer suggestions for improving such efforts in the future. AS A TIME-LIMITED initiative with a statutorily defined sunset, the School-to-Work Opportunities Act of 1994 (STWOA) provides a unique window through which to view the life span of a performance measurement system -- design through gestation and early implementation to wide-scale operation and, ultimately, closure. In this article we explore some of the important lessons learned the design and implementation of the STWOA performance measurement process and offer recommendations to the policy makers who will help shape future data- collection mandates. Lessons Learned 1. Time and money are needed to build effective performance measurement systems; neither was available in sufficient quantities under the STWOA. While the STWOA explicitly called for the creation of a performance measurement system, it did not target additional resources to support state and local data-collection activities. States and local partnerships were forced to cobble together their processes of data collection with limited resources and, often, with limited expertise. The time line under which the mandated performance measurement system was developed and implemented was ambitious, if not unrealistic. A task force of state school-to-work (STW) directors gave their collective advice on the development of a data-collection instrument, and the resulting Progress Measures Survey was field-tested less than 18 months after the enactment of the STWOA. Even so, more than two years elapsed before a substantial number of local partnerships were in a position to collect any data. The Progress Measures Survey asked local STW partnerships to report quantitative data on a number of items about which information had never before been collected. Moreover, partnerships had to collect data many sources and create mechanisms for documenting participation in a range of school- and work-based learning activities. For many partnerships, the first collection of progress measures data was a learning experience. They discovered which institutions and individuals had what types of data and determined which types of data had to be collected from scratch. 2. Federal, state, and local audiences have different data needs and interests. Federal, state, and local officials think differently about the purposes and processes of data collection. Their differing expectations and perspectives add yet another level of complexity to efforts to build effective systems to measure performance. To produce meaningful data, measurement needs to occur at the level at which goals are set and activities are defined. This was not necessarily the case with STWOA data-collection efforts. Although local partnerships, and ultimately individual schools, were asked to collect data, the Progress Measures Survey reflected federal and state priorities. Local practitioners had little or no input into the indicators used or the questions asked. Table 1 summarizes key differences in perspective that emerged a qualitative study of STW data-collection experiences. The first column identifies specific topics or issues related to the collection and utility of the data requested in the Progress Measures Survey. The remaining columns present federal, state, and local perspectives on each topic. This table illustrates the mismatch of perspectives and the lack of philosophical alignment in five crucial areas. The different needs and wants at each level of governance were based largely on the context and circumstances in which officials and practitioners operate. In the collection of progress measures data, these differences played out as unresolved -- and, perhaps, irreconcilable -- tensions. …