The measurement of problem-solving competence using process data

Abstract

<p id=C9>Problem-solving competence is an individual’s capacity to engage in cognitive processing to understand and resolve problem situations where a method of solution is not immediately obvious. The measurement of problem-solving competence requires the use of relatively more complex and real problem situations to induce the presentation of problem-solving behaviors. This brings challenges to both the measurement methods of problem-solving competence and the corresponding data analysis methods. Using virtual assessments to capture the process data in problem-solving and mining the potential information contained therein is a new trend in measuring problem-solving competence in psychometrics. <p id=C10>Firstly, this paper reviews the development of measurement methods from pen-and-paper tests to virtual assessments. Compared with the traditional paper-and-pencil test, modern virtual assessments are not only conducive to simulating real problem situations, improving the ecological validity of the test, but also can record the process data generated by individuals in the process of problem-solving. Process data refers to man-machine or man-human interaction data with timestamps that can reflect the process of individual problem-solving. It records the detailed steps of individual problem solving and reflects the strategy and cognitive process of individual problem-solving. However, it is not easy to adopt effective methods to analyze process data. <p id=C11>Secondly, two methods of analyzing process data are summarized and compared: data mining methods and statistical modeling methods. Data mining is the process of using algorithms to uncover new relationships, trends, and patterns from big data. It is a bottom-up, data-driven research method that focuses on describing and summarizing data. Its advantage is that it can use existing algorithms to analyze a variety of process data at the same time, screen out variables related to individual problem-solving competence, and realize the classification of individual problem-solving competence. But sometimes, different algorithms could get different conclusions based on the same data, which leads to part of the results can not be explained. This method can not construct variables that can reflect the individuals latent trait, either. Statistical modeling method mainly refers to the method of analyzing data by using the idea of artificial modeling. It is a top-down, theory-driven approach. In statistical modeling, function models are generally constructed based on theoretical assumptions, and the observed variables are assumed to be randomly generated by the probability law expressed by the model. For the data recorded by virtual assessments, the existing modeling methods can be divided into three categories: psychometric joint modeling, hidden Markov modeling, and multi-level modeling. The main advantage of statistical modeling is that its results are easy to interpret and conform to the general process of psychological and educational research. Its limitation lies in that the modeling logic has not been unified yet because different types of process data need to be modeled separately. However, by giving full play to the advantages of the two data analysis methods, different problems in psychological and educational assessments can be dealt with. The interpretability of the results is very important in psychological and educational measurements, which determines the dominant role of statistical modeling in process data analysis. <p id=C12>Finally, the possible future research directions are proposed from five aspects: the influence of non-cognitive factors, the use of multimodal data, the measurements of the development of problem-solving competence, the measurements of other higher-order thinking competence, and the definition of the concept and structure of problem-solving competence.