Predictive Feature Generation and Selection Using Process Data from PISA Simulation-Based Environment: An Application of Tree-Based Ensemble Methods

Abstract

As one of the most innovative international large-scale assessments, the Programme for International Student Assessment (PISA) introduced the measurement of problem-solving skills in the 2012 cycle. The items in this new domain were typically designed as scenario-based environments and featured in interactions between students and computers. Process data collected in log files were especially valuable to provide deeper insight into students’ behaviors and allowed tracking their problem-solving strategies. This study illustrates a two-stage approach to generate features from process data and select those that predict student performance using a released problem-solving item “Climate Control” from PISA 2012. The specific research questions we focus on are: (1) how well the features generated from process data can predict test takers’ responses on a certain item, and (2) which features are the most predictive ones. We used a tree-based ensemble method called Random Forest to explore the association between response data as well as to extract features from process data. The eventual goal is to address issues around the complex structure of extracted features and the availability of massive numbers of variables representing different interactions in log-file entries.