Validation Of Cognitive Operations and Processes Across Ability Levels and Individual Test Items

Abstract

Knowledge about cognitive operations and processes (COPs) required for success (1=correct, 0=incorrect) on test items or learning tasks is very important for in-depth understanding of the nature of student performance and the development of valid instruments for its measurement. A key problem in obtaining such knowledge is the validation of hypothesized COPs and their role in the measurement properties of test items. To provide validation feedback for both normally achieving students and students with learning disabilities, it is important to obtain information on the validity of the COPs for students at different ability levels and individual test items (or tasks). To address this issue, the present chapter introduces a method of estimating the probability for correct performance on individual COPs at fixed ability levels thus providing validity information across ability levels and individual test items. When item response theory (IRT) estimates of the item parameters are known (e.g., in a test bank of IRT calibrated items or published research), the proposed validation method does not require information about raw (or ability) scores of examinees. This method is illustrated for algebra test items and reading comprehension test items calibrated in IRT.