Anatomy of a physics test: Validation of the physics items on the Texas Assessment of Knowledge and Skills

Jill A Marshall,Jerry O’Connor,Eric A Hagedorn

doi:10.1103/physrevstper.5.010104

Abstract

We report the results of an analysis of the Texas Assessment of Knowledge and Skills (TAKS) designed to determine whether the TAKS is a valid indicator of whether students know and can do physics at the level necessary for success in future coursework, STEM careers, and life in a technological society. We categorized science items from the 2003 and 2004 10th and 11th grade TAKS by content area(s) covered, knowledge and skills required to select the correct answer, and overall quality. We also analyzed a 5000 student sample of item-level results from the 2004 11th grade exam, performing full-information factor analysis, calculating classical test indices, and determining each item's response curve using item response theory. Triangulation of our results revealed strengths and weaknesses of the different methods of analysis. The TAKS was found to be only weakly indicative of physics preparation and we make recommendations for increasing the validity of standardized physics testing.

Highlights

Beginning in 2003, the Texas Assessment of Knowledge and SkillsTAKSfor science has been administered annually to 10th and 11th grade students in Texas
The Texas Education AgencyTEAconsiders this step to be critical at the high school level because of the “advanced level of the content being assessed.”[2] only a limited number of physics experts are included in this review panel and its deliberationsbeyond the released items from the TAKS itselfare not made public
Is the TAKS a valid indicator of who is prepared to “do” physics in future coursework, in the course of employment, and in making the decisions required of citizens in a technological society? Our analyses show that this assessment is valid only in a very limited sense

Summary

Introduction

Beginning in 2003, the Texas Assessment of Knowledge and SkillsTAKSfor science has been administered annually to 10th and 11th grade students in Texas. The science portion of the TAKS is designed to assess a subset of the Texas Essential Knowledge and Skills,[1] the learning standards for Texas students. The process of designing the annual test begins with inviting science educators in Texas to propose items. The test items undergo a content validation review by a panel of recognized science experts. The Texas Education AgencyTEAconsiders this step to be critical at the high school level because of the “advanced level of the content being assessed.”[2] only a limited number of physics experts are included in this review panel and its deliberationsbeyond the released items from the TAKS itselfare not made public

Results

Discussion

Conclusion