DIAGNOSTIC Classification Analysis of Problem-Solving Competence using Process Data: An Item Expansion Method.

Abstract

Process data refer to data recorded in computer-based assessments (CBAs) that reflect respondents' problem-solving processes and provide greater insight into how respondents solve problems, in addition to how well they solve them. Using the rich information contained in process data, this study proposed an item expansion method to analyze action-level process data from the perspective of diagnostic classification in order to comprehensively understand respondents' problem-solving competence. The proposed method cannot only estimate respondents' problem-solving ability along a continuum, but also classify respondents according to their problem-solving skills. To illustrate the application and advantages of the proposed method, a Programme for International Student Assessment (PISA) problem-solving item was used. The results indicated that (a) the estimated latent classes provided more detailed diagnoses of respondents' problem-solving skills than the observed score categories; (b) although only one item was used, the estimated higher-order latent ability reflected the respondents' problem-solving ability more accurately than the unidimensional latent ability estimated from the outcome data; and (c) interactions among problem-solving skills followed the conjunctive condensation rule, which indicated that the specific action sequence appeared only when a respondent mastered all required problem solving skills. In conclusion, the proposed diagnostic classification approach is feasible and promising analyzing process data.