Understanding Test Takers' Choices in a Self-Adapted Test: A Hidden Markov Modeling of Process Data.

Meirav Arieli-Attali,Lu Ou,Vanessa R Simmering

doi:10.3389/fpsyg.2019.00083

Abstract

With the rise of more interactive assessments, such as simulation- and game-based assessment, process data are available to learn about students' cognitive processes as well as motivational aspects. Since process data can be complicated due to interdependencies in time, our traditional psychometric models may not necessarily fit, and we need to look for additional ways to analyze such data. In this study, we draw process data from a study on self-adapted test under different goal conditions (Arieli-Attali, 2016) and use hidden Markov models to learn about test takers' choice making behavior. Self-adapted test is designed to allow test takers to choose the level of difficulty of the items they receive. The data includes test results from two conditions of goal orientation (performance goal and learning goal), as well as confidence ratings on each question. We show that using HMM we can learn about transition probabilities from one state to another as dependent on the goal orientation, the accumulated score and accumulated confidence, and the interactions therein. The implications of such insights are discussed.

Highlights

With the rise of interactive assessment and learning programs, process data become available to infer about students’ cognitive and motivational aspects
We first provide a description and visualization of the data, along with the hidden Markov model (HMM) general results about state classifications and initial state modeling, followed by two sets of our transitions modeling questions: (1) modeling transitions between states in the two goal conditions; (2) modeling transitions based on accumulated correctness and confidence and their interactions
There was not a clear difference between the conditions in the number or proportion of participants choosing to start and stay at the highest difficulty level, substantially more participants chose to start at the lowest difficulty level and stay there in the performance goal condition (33 or 11.54%) than in the learning goal condition (10 or 3.37%)

Summary

Introduction

With the rise of interactive assessment and learning programs, process data become available to infer about students’ cognitive and motivational aspects. In the context of problem solving, detecting strategies may reveal the cognitive processes needed to perform the task, and may even be considered as a factor in ability estimating (DiCerbo and Behrens, 2012; Liu et al, 2018) Interactive assessments such as simulation- and game-based assessments often afford opportunities to make choices about the course of game/simulation (e.g., which variables to try in the simulation, which path to take in the game) that are not directly connected to ability albeit may influence its assessment. Such choices may be a result of or reflect metacognitive or motivational aspects of task performance. Students’ self-estimated knowledge and belief in their ability, students’ tendency toward challenge, or whether students are motivated to do their best or just perform at minimum effort are just a few of the factors that may play a role in choices made in interactive assessment

Methods

Results

Conclusion