How to conceptualize, represent, and analyze log\xa0data from technology-based assessments? A generic framework and an application to questionnaire items

Ulf Kroehne,Frank Goldhammer

doi:10.1007/s41237-018-0063-y

Abstract

Log data from educational assessments attract more and more attention and large-scale assessment programs have started providing log data as scientific use files. Such data generated as a by-product of computer-assisted data collection has been known as paradata in survey research. In this paper, we integrate log data from educational assessments into a taxonomy of paradata. To provide a generic framework for the analysis of log data, finite state machines are suggested. Beyond its computational value, the specific benefit of using finite state machines is achieved by separating platform-specific log events from the definition of indicators by states. Specifically, states represent filtered log data given a theoretical process model, and therefore, encode the information of log files selectively. The approach is empirically illustrated using log data of the context questionnaires of the Programme for International Student Assessment (PISA). We extracted item-level response time components from questionnaire items that were administered as item batteries with multiple questions on one screen and related them to the item responses. Finally, the taxonomy and the finite state machine approach are discussed with respect to the definition of complete log data, the verification of log data and the reproducibility of log data analyses.

Highlights

Educational large-scale assessments are in the middle of introducing computerbased assessment and new methods of data collection
Log data are not entirely new, in particular for questionnaires, as these additional data can be understood as part of the concept of paradata developed in the field of survey research
The taxonomy resulting from the following review represents our attempt to structure different types of paradata according to their use in a small number of categories, which we describe with prototypical examples

Summary

Introduction

Educational large-scale assessments are in the middle of introducing computerbased assessment and new methods of data collection. With this change of test administration mode, the incoming log data attract more attention, for instance, for the investigation of time on task (e.g., Scherer et al 2015), to improve validity and reliability of computer-based administered measures (Ramalingam and Adams 2018), or to compare response sequences (e.g., He and von Davier 2015). We will show that for some indicators the classification of paradata in terms of a taxonomy is not sufficient. If indicators require the combination of multiple paradata points (e.g., multiple timestamps; Zhang and Conrad 2013) or a sequence of multiple log events, the (atomistic) classification of log events falls short

Methods

Results

Discussion

Conclusion