Performance Assessment For California Teachers Research Articles

More Evidence of Low Inter-Rater Reliability of a High-Stakes Performance Assessment of Teacher Candidates

From 2010 to 2015, our school of education used the Performance Assessment for California Teachers (PACT), a summative assessment designed to assess preservice teacher competence. Candidate portfolios were uploaded to an evaluation portal, and trained evaluators assigned a final score of Pass or Fail to the work samples. Three consensus estimates of inter-rater reliability of 181 candidate portfolios that were either double- or triple-scored were computed. Two chance-corrected estimates of inter-rater reliability (Cohen’s kappa and Gwet’s AC1) and percent agreement were computed and calculated within five content areas: elementary math, secondary history/social science, math, science, and English language arts. An initial Pass or Fail score was not more likely to be followed by either a Pass or Fail score given by a subsequent evaluator. Inter-rater reliability was interpreted as being low across all content areas that were examined. None of the percent agreement coefficients attained the minimum standard of 0.700 for consensus agreement. Increasing research access to proprietary double-scored data would lead to an increased understanding of, and perhaps improvement in, teacher performance assessments.

Low Inter-Rater Reliability of a High Stakes Performance Assessment of Teacher Candidates

The Performance Assessment for California Teachers (PACT) is a high stakes summative assessment that was designed to measure pre-service teacher readiness. We examined the inter-rater reliability (IRR) of trained PACT evaluators who rated 19 candidates. As measured by Cohen’s weighted kappa, the overall IRR estimate was 0.17 (poor strength of agreement). IRR estimates ranged from −0.29 (worse than expected by chance) to 0.54 (moderate strength of agreement); all were below the standard of 0.70 for consensus agreement. Follow-up interviews of 10 evaluators revealed possible reasons we observed low IRR, such as departures from established PACT scoring protocol, and lack of, or inconsistent, use of a scoring aid document. Evaluators reported difficulties scoring the materials that candidates submitted, particularly the use of Academic Language. Cognitive Task Analysis (CTA) is suggested as a method to improve IRR in the PACT and other teacher performance assessments such as the edTPA.

Preservice Teachers’ Mathematics Teaching Competence: Comparing Performance on Two Measures

This exploratory study examines the relationship between preservice teachers’ performance on a teaching performance assessment for licensure in elementary mathematics and a measure of knowledge that, in studies of practicing teachers, was found to predict effective mathematics teaching. A sample of 89 preservice teachers completed the Performance Assessment for California Teachers (PACT) and the classroom video analysis (CVA) instrument. Correlation analyses found overall weak associations between measures for the whole group. For groups of high, medium, and low performers on PACT, average scores on the CVA measure and its subscales varied in the predicted direction. However, individuals within the PACT performing groups had a relatively broad range of scores on the CVA, and several preservice teachers who performed poorly on PACT had average or better than average performance on the CVA. Similarly, several of the PACT high performers scored below the whole group CVA average. In addition to suggesting areas for future research, the findings raise questions about assessing preservice teachers’ readiness to teach mathematics and the use of a single measure to make licensing decisions.

What Do We Really Know About the edTPA? Research, PACT, and Packaging a Local Teacher Performance Assessment for National Use

This article calls attention to the overreliance on research about the Performance Assessment for California Teachers (PACT)—often labeled edTPA's predecessor—as justification for the edTPA. The article argues that the distinctions between the assessments are too vast to rely on PACT data to support the edTPA, given the localized nature of PACT and the way in which it is scored.

Politics of policy: Assessing the implementation, impact, and evolution of the Performance Assessment for California Teachers (PACT) and edTPA

Summative performance assessments in teacher education, such as the Performance Assessment for California Teachers (PACT) and the edTPA, have been heralded through polices intended to enhance the quality of the teaching profession and raise its stature among other professions. However, the development and implementation of the PACT, and subsequently the edTPA, have not been without controversy and debate. The purpose of this article is to assess the implementation, impact, and evolution of the PACT and edTPA. To do so, we review the growing body of literature on the impact and implementation of the PACT and critically analyze the state policies surrounding the edTPA. We raise questions about the practical and policy implications of the PACT’s evolution, as a statewide assessment, to the edTPA, a nationally available assessment.

The Need for Instructional Sensitivity and Construct Clarity in PACT

This commentary on the article titled “Examining the Internal Structure Evidence for the Performance Assessment for California Teachers: A Validation Study of the Elementary Literacy Teaching Event for Tier 1 Teacher Licensure” provides an overview of Performance Assessment for California Teachers (PACT), its relationship to edTPA and the construct (domains) measured, a summary of the major findings of the article commented on, a discussion of PACT validity issues and rubrics, and the need for instructionally sensitive and formative assessments. It concludes with recommendations to the State of California that include attention to construct validity, improved rubrics, formative use of the data, and attention to utility as well as validity and reliability.

Integrating a Focus on Academic Language, English Learners, and Mathematics: Teacher Candidates’ Responses on the Performance Assessment for California Teachers (PACT)

Throughout the United States, teacher educators are developing new strategies to improve the preparation of mainstream teachers for linguistic diversity. In this article, we explore teacher candidates’ responses to the Performance Assessment for California Teachers (PACT), a preservice assessment required for credentialing that requires candidates to consider the English language demands of content-area instruction, to enact supports for English learners and others who may struggle with those demands, and to develop ways to promote the academic language development of these students. We report on a study examining written responses on the PACT from 8 elementary teacher candidates who chose mathematics as the focus for their teaching event. We found that the PACT induced teacher candidates to consider language issues and the needs of language learners in a content area often thought to be language-free, yet one in which language in fact plays a crucial role. Such responses provide important information that can be used not only for credentialing decisions but also by teacher educators, teacher-education programs, and teacher candidates themselves in supporting the academic-language development of ELs in content areas such as mathematics.

Performance Assessment of Teaching

Recent challenges concerning the quality of teachers and the programs that prepare them have resulted in responses from professional organizations that have been more widespread and intense than in the past (Robinson, 2011; Wiseman, 2012). The Blue Ribbon Panel on Clinical Preparation (BRP) formed by the National Council for the Accreditation of Teacher Education (NCATE), the merger of NCATE and Teacher Education Accreditation Council (TEAC) into Council for the Accreditation of Educator Preparation (CAEP) to improve teacher education (Cibulka, 2011), and the Teacher Performance Assessment Initiative which emerged from collaboration between the Performance Assessment for California Teachers (PACT), the National Board for Professional Teaching Standards (NBPTS), and American Association of Colleges for Teacher Education (AACTE) member institutions (Robinson, 2011) are examples of ways that professional organizations have responded to the challenges. In general, calls for performance assessments of teaching to provide both formative and summative information about the quality of teachers, teacher candidates, and teacher preparation programs pervade the current discourse on education at every level, including schools and districts, institutions of higher education, state and national policymaking entities, and professional organizations. As a result, a number of performance assessments have emerged that can provide feedback to teacher education programs about the strengths and weaknesses of their candidates as well as evidence for licensure and certification of teachers. For example, Education Teacher Performance Assessment (edTPA) was designed to provide information for program improvement through the use of a common set of performance assessments for preservice teachers and has also been adopted by states and districts for summative evaluation purposes (http://edtpa.aacte.org/). The multiple performance measures in systems like edTPA improve upon value added models (VAM) by providing information on teaching quality, not just teacher quality or teacher effectiveness, and provide information sooner than might be accomplished by use of VAM alone (Newton, 2010). For this theme issue, we invited research and conceptual articles related to teacher performance assessments specifically as they affect teacher education, including validation studies of measures used in the assessments. We posed a series of questions that we considered important for researchers to address related to performance assessments, including the following: What are the relationships between current performance assessments and teacher education, either conceptually or in terms of outcomes? What do we know about the use of performance assessments in the context of teacher education, and what do we need to know? How do performance assessments of teachers in schools map onto assessment of teacher education programs, or how don't they? What differences may exist across novice teachers and expert teachers with respect to performance assessment, and how does that impact their professional learning? What are the implications of the relationships between for-profit companies, university teacher education programs, researchers, states, and professional organizations with respect to performance assessments of teachers? What is the relationship between assessment criteria and the research on teaching and learning, and what are its implications for teacher education? How do states make decisions to move to statewide teacher assessment systems? What potential conflicts of interest may exist among various stakeholders in performance assessment efforts? How do policies related to performance assessment of teachers affect opportunities for teacher learning? What can we learn from the history of past efforts to evaluate beginning teachers? What is different this time? What do we know about the validity and reliability of performance assessments and why does this matter? …

Examining the Internal Structure Evidence for the Performance Assessment for California Teachers

Interpretations for licensure tests involve a series of inferences or a validity argument, leading from the test score to decisions about who is accepted or denied entry into a profession. Utilizing an argument-based framework for validation based on the Standards for Educational and Psychological Testing, we explore the evidence for the ongoing use of the Performance Assessment for California Teachers (PACT) for Tier I licensure decisions. The evidence for a unidimensional and a multidimensional structure based on the instrument’s content are examined with an item response model. Examining operational data ( n = 1,711) from seven California teacher education institutions, we found sufficient internal structure validity evidence to support the continued, but limited, use of this instrument for its intended summative purpose. Evidence for a three-dimensional structure of model fit better explains overall teacher candidate performance on the PACT instrument as it is currently designed.

Performance Assessment for California Teachers and English-Language Arts Candidates in a Rural Border Community

This study explores the appropriateness of the Performance Assessment for California Teachers (PACT) as an instrument of assessing English-language arts (ELA) teacher candidates’ effectiveness in a rural border community. Eight -candidates participated in the study. The findings call into question the adequacy of PACT to assess the candidates’ effectiveness in light of the ways in which the school constrains their abilities from developing culturally relevant practices.

Assessing Pre-service Teachers Prior to Certification: Perspectives on the Performance Assessment for California Teachers (PACT)

This study focuses on pre-service teachers’ perspectives regarding how the process of completing the Performance Assessment for California Teachers (PACT) affected them academically, professionally, and personally. Two cohorts participated in this year-two study. Pre-service teachers’ perspectives were acquired using a survey instrument comprised of open-ended questions. In addition, pre-service teachers’ self-confidence levels pertaining to assessment task components of PACT (i.e., planning, instruction, assessment, reflection, and academic language) were measured prior to the execution of PACT portfolio assessment and these levels were compared to the actual scores on PACT. This study concludes with implications for teacher educators and teacher education programs implementing pre-service teacher assessments; these implications include policy level suggestions as well as a discussion of intended and unintended consequences of the PACT assessment on the pre-service teachers.

Developing and assessing beginning teacher effectiveness: the potential of performance assessments

The Performance Assessment for California Teachers (PACT) is an authentic tool for evaluating prospective teachers by examining their abilities to plan, teach, assess, and reflect on instruction in actual classroom practice. The PACT seeks both to measure and develop teacher effectiveness, and this study of its predictive and consequential validity provides information on how well it achieves these goals. The research finds that teacher candidates’ PACT scores are significant predictors of their later teaching effectiveness as measured by their students’ achievement gains in both English language arts (ELA) and mathematics. Several subscales of the PACT are also influential in predicting later effectiveness: These include planning, assessment, and academic language development in ELA, and assessment and reflection in mathematics. In addition, large majorities of PACT candidates report that they acquired additional knowledge and skills for teaching by virtue of completing the assessment. Candidates’ feelings that they learned from the assessment were the strongest when they also felt well-supported by their program in learning to teach and in completing the assessment process.

A Balancing Act: Dilemmas of Implementing a High-Stakes Performance Assessment

Dilemmas abound in the design, implementation, analysis, and interpretation of the outcomes of a summative performance assessment of teaching when used for licensing, for individual and program evaluation, and for accountability purposes. In this article, we explore the impact of the Performance Assessment for California Teachers (PACT) on the Stanford Teacher Education Program (STEP). Specifically, we explore ways in which this high-stakes assessment influences programmatic priorities and drives structural and curricular decisions that might or might not be at odds with the conceptual frame and stated mission of the program. We highlight two thorny dilemmas that have challenged our program in the implementation of the PACT Teaching Event—a high-stakes performance assessment for teachers—and our attempts to manage these dilemmas “satisficingly.”

Creating “Cultures of Evidence” in Teacher Education: Context, Policy, and Practice in Three High-Data-Use Programs

One of the important rationales for the development and implementation of a rigorous classroom-based measure of preservice teacher quality is that such a tool will provide new sources of data that are highly relevant to the task of improving programs of teacher preparation. However, research on data utilization within organizations from a variety of disciplines makes it clear that even when relevant and useful data are available, they are often not used for decision making. We studied three “high-data-use” programs from among the 32 California institutions implementing the Performance Assessment for California Teachers (PACT) to identify organizational practices associated with the use of PACT data for program improvement. We describe practices within and across these programs that were identified by informants as important to their success in using teacher performance outcome data to make decisions about program renewal and improvement.

Assessment with an “End in View”

This article uses Dewey's (1938) concept of “end in view” to frame one California State University's purposeful action in implementing the Performance Assessment for California Teachers (PACT). The authors provide a chronology of events that reveal the ways in which teacher education faculty were engaged in examining PACT outcomes and candidate performance through informal, “mock scoring” events, and the design of formative, embedded signature assessments, and the resulting changes in the program assessment system, curriculum, and instructional practices. The article closes with concerns and suggestions for sustaining the “end in view” during high-stakes and low-budget circumstances.

Positive effects of brain-based education on teacher performance assessment outcomes

Abstract The primary goal of this project was to improve teacher preparation by helping teachers identify pupils’ strengths, weaknesses, and affinities and by providing appropriate adjustments to instruction when necessary through neurodevelopmental methodology. Using a selected population of teacher candidates who were learning about brain-based education and neurodevelopment (ND) while preparing for the Performance Assessment for California Teachers (PACT), the researchers looked for an opportunity to study the effects of ND methodology on candidates' PACT performance. A grant awarded to elementary, secondary, and special education faculties in an accelerated collaborative teaching (ACT) programme at California State University, Northridge, allowed participants to study certain features of using a specific approach, that is, neurodevelopmental methodology, and the effects of this approach on teaching and learning, including teacher performance assessment results. Fifteen target courses in the accelerated teacher programme were identified in the overall grant project for implementation of neurodevelopmental content; these courses included elementary, secondary, and special education levels and involved, as participants, 98 teacher candidates, 17 university faculty members, 28 university supervisors, and 163 master teachers.

Predicting Performance

The implementation of teaching performance assessments has prompted a range of concerns. Some educators question whether these assessments provide information beyond what university supervisors gain through their formative evaluations and classroom observations of candidates. This research examines the relationship between supervisors’ predictions and candidates’ performance on a summative assessment based on a capstone teaching event, the Performance Assessment for California Teachers. The study, based on records for 337 teacher candidates over a 2-year period, specifically addresses the following questions: To what extent do university supervisors predict candidates’ total scores? On which questions and categories of the assessment do supervisors most accurately predict their candidates’ scores? Do supervisors predict scores more accurately for high- and low-performing candidates? The findings indicate that university supervisors’ perspectives did not always correspond with outcomes on the performance assessment, particularly for high and low performers.

Letter from the Editorial Board

Stanford University and the American Association of Colleges for Teacher Education (AACTE), along with other preparation institutions, are currently piloting a Teacher Performance Assessment (TPA) program. The TPA, based on the successful Performance Assessment for California Teachers (PACT), (AACTE) is being piloted in twenty states. The goal of the TPA consortium is to develop a available (AACTE, 2011) with which to assess performance. This is a significant development in assessment and, as such, is an issue of importance for The High School Journal. There has never before been a nationally available instrument to measure performance apart from the valueadded model, which is a highly contested measure and ultimately dependent on student, rather than teacher, performance. Furthermore, the TPA is being piloted in the context of the teacher accountability era. Some of the discourse surrounding accountability, such as that highlighted in the documentary Waiting for Superman, presents teachers in a less than positive light. In light of these concerns, this editorial will posit critical questions about the TPA that will need to be answered by researchers and highlight reasons to feel optimism and skepticism towards the evaluation model. First, a little more detail about the TPA is in order. According to the TPA consortium website (http://aacte.org) the TPA consists of two components: Embedded Signature Assessments (ESA) and a common portfolio assessment. ESAs vary from to program, are embedded in coursework, and reflect program specific teaching philosophies or goals that contribute to the unique character of graduates. (AACTE, 2011) The larger portion of the assessment is a portfolio, universal to all participating programs, based on teaching events. The portfolio has two components: teaching evidence and commentary. Teaching evidence includes artifacts such as lesson plans, video recorded lessons, and student work. The second component is commentary in which teachers explain the professional judgment underlying the teaching and artifacts. Simply put, teachers provide evidence of what they are doing in the class and explain why they did what they did. As a further note, separate forms are provided for elementary teachers and subject-specific secondary teachers. The move towards a standardized assessment begs several questions. Student high stakes tests have had multiple, and often damaging, unintended consequences; what unintended consequences will result from standard assessments? What is the relationship between standardization for student assessment and the new assessment? If the pilot programs prove measurably successful, how will teachers, education programs, state legislatures, and the public respond? Finally, would a national assessment more closely represent a professionalization of the field of teaching or the slippery slope of national standardization and the marginalization of local decision-making? We encourage educational researchers to begin to consider these questions. Having examined the TPA, the editorial board of The High School Journal looks on the TPA with cautious optimism. In light of the information provided by AACTE, there are a few issues with which we encourage educators to be cautious. The first is our concern about the purposes of the TPA. The purpose of a truly meaningful assessment is not merely to evaluate, but to lead to improvement in practice. This leads to questions of purposes for three constituents. For TPA developers who provide training for school administrators to use the TPA, will the communicated purpose more closely resemble teacher grading or teacher improvement? When school administrators actively use the TPA among the multitude of their other tasks, will the TPA more closely resemble checking off another item on the agenda or a chance to have meaningful pedagogical conversations with their teachers, leading to improved practice? …

Evaluating teacher education outcomes: a study of the Stanford Teacher Education Programme

This paper considers a set of research and assessment strategies used to evaluate programme outcomes in the Stanford Teacher Education Programme (STEP) during a period of programme redesign over 10 years. These strategies include surveys and interviews of students’ perceptions of programme elements and their own preparedness, observations of their practice during and after teacher education, evaluations of their practice on a structured portfolio of practice (the Performance Assessment for California Teachers (PACT)) and analyses of the effects of a sample of graduates of STEP and other programmes on student outcomes, including value‐added measures. While the studies were able to ascertain that the students of STEP graduates had strong value‐added learning gains, the paper concludes that the use of student learning data alone as a measure of teacher effectiveness does not help guide decisions related to programme improvement, and a range of approaches is required. In addition it suggests that there will be continuing concerns about the narrowness of the learning measured by standardised tests, and about the many challenges of collecting and analysing such data in ways that overcome the technical and practical problems associated with their use.

Evidence in Teacher Education

The Performance Assessment for California Teachers (PACT) was developed in response to a California State mandate (SB 2042), requiring teacher preparation programs to use performance assessments as one measure in making credentialing decisions. In this article, results are examined from statewide implementation of the PACT assessments during the first 2 pilot years. Despite the limitation of only 2 years of data, 3 years of implementation experiences have informed participating programs about how they can better support candidate learning and identify areas for examination. In addition, this research suggests that the PACT performance assessment can be used in teacher education as a valid measure of individual teacher competence for the purpose of teacher licensure and as a powerful tool for teacher learning and program improvement.