Abstract

The OECD Programme for the International Assessment of Adult Competencies (PIAAC) was the first computer-based large-scale assessment to provide anonymised log file data from the cognitive assessment together with extensive online documentation and a data analysis support tool. The goal of the chapter is to familiarise researchers with how to access, understand, and analyse PIAAC log file data for their research purposes. After providing some conceptual background on the multiple uses of log file data and how to infer states of information processing from log file data, previous research using PIAAC log file data is reviewed. Then, the accessibility, structure, and documentation of the PIAAC log file data are described in detail, as well as how to use the PIAAC LogDataAnalyzer to extract predefined process indicators and how to create new process indicators based on the raw log data export.

Highlights

  • The OECD Programme for the International Assessment of Adult Competencies (PIAAC) was the first computer-based large-scale assessment to provide anonymised log file data from the cognitive assessment together with extensive online documentation and a data analysis support tool

  • The Programme for the International Assessment of Adult Competencies (PIAAC) is an Organisation for Economic Co-operation and Development (OECD) study that assesses and analyses adult skills in the cognitive domains of literacy, numeracy, and problem solving in technology-rich environments (PS-TRE)

  • Log file data are a suitable complement to evidence rules in multiple ways. They can be used to obtain a more fine-grained scoring of the work product, depending on whether interactions contributing to the correct outcome were carried out or not, to inform the coding of missing responses, and to detect suspicious cases showing aberrant response behaviour or data fabrication (Yamamoto and Lennon 2018)

Read more

Summary

10.1.1 Conceptual Remarks

The reasons for using log file data in educational assessment can be diverse and driven by substantive research questions and technical measurement issues. Log file data are a suitable complement to evidence rules in multiple ways They can be used to obtain a more fine-grained (partial credit) scoring of the work product, depending on whether interactions contributing to the correct outcome were carried out or not (e.g. problem solving in PISA 2012; OECD 2013a), to inform the coding of missing responses (e.g. responses in PIAAC without any interaction and a time on task less than 5 s were coded as ‘Not reached/not attempted’; OECD 2013b), and to detect suspicious cases showing aberrant response behaviour (van der Linden and Guo 2008) or data fabrication (Yamamoto and Lennon 2018). This information can be fed back to the individual testtaker via prompts (Buerger et al 2019), so that he or she can adapt, or to the proctor via a dashboard, so that he or she can intervene if needed (Wise et al 2019)

10.1.2 Methodological Remarks
10.2 Review of Research Work Using PIAAC Log File Data
10.2.1 Studies of Time Components Across Competence Domains
10.2.2 Studies of the Domain of PS-TRE
10.3 The Released PIAAC Log File Data
10.3.1 Overview of PIAAC Test Design
10.3.2 File Structure and Accessibility
10.3.3 Documentation of Items and Log Events
10.4 Preprocessing and Analysing PIAAC Log File Data
10.4.1 Aggregated Variables Provided by the PIAAC Log Data Analyzer
10.4.2 User-Defined Aggregated Variables
Findings
10.5 Conclusions
Full Text
Published version (Free)

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call