Examining the Validity of Adaptive Comparative Judgment for Peer Evaluation in a Design Thinking Course

Nathan Mentzer,Wonki Lee,Scott Ronald Bartholomew

doi:10.3389/feduc.2021.772832

Nathan Mentzer, Wonki Lee + Show 1 more

Open Access

https://doi.org/10.3389/feduc.2021.772832

Copy DOI

Abstract

Adaptive comparative judgment (ACJ) is a holistic judgment approach used to evaluate the quality of something (e.g., student work) in which individuals are presented with pairs of work and select the better item from each pair. This approach has demonstrated high levels of reliability with less bias than other approaches, hence providing accurate values in summative and formative assessment in educational settings. Though ACJ itself has demonstrated significantly high reliability levels, relatively few studies have investigated the validity of peer-evaluated ACJ in the context of design thinking. This study explored peer-evaluation, facilitated through ACJ, in terms of construct validity and criterion validity (concurrent validity and predictive validity) in the context of a design thinking course. Using ACJ, undergraduate students (n= 597) who took a design thinking course during Spring 2019 were invited to evaluate design point-of-view (POV) statements written by their peers. As a result of this ACJ exercise, each POV statement attained a specific parameter value, which reflects the quality of POV statements. In order to examine the construct validity, researchers conducted a content analysis, comparing the contents of the 10 POV statements with highest scores (parameter values) and the 10 POV statements with the lowest scores (parameter values)—as derived from the ACJ session. For the criterion validity, we studied the relationship between peer-evaluated ACJ and grader’s rubric-based grading. To study the concurrent validity, we investigated the correlation between peer-evaluated ACJ parameter values and grades assigned by course instructors for the same POV writing task. Then, predictive validity was studied by exploring if peer-evaluated ACJ of POV statements were predictive of students’ grades on the final project. Results showed that the contents of the statements with the highest parameter values were of better quality compared to the statements with the lowest parameter values. Therefore, peer-evaluated ACJ showed construct validity. Also, though peer-evaluated ACJ did not show concurrent validity, it did show moderate predictive validity.

Highlights

Design is believed to be the core of technology and engineering, which promotes experiential learning towards the development of a robust understanding (Dym et al, 2005; Atman et al, 2008)
Our research questions guiding the inquiry were: 1) What is the construct validity of ACJ? Does peer-reviewed ACJ reflect general criteria of good POV statements? 2) What is the criterion validity of ACJ? By doing so, this study aimed to validate peer-evaluated ACJ in the design thinking education context
This study analyzed ten high parameter value statements and ten low parameter value statements based on the criteria of “good” POV statements (Interaction Design Foundation, 2020; Rikke Friis and Teo Yu, 2020) to examine the construct validity of ACJ

Summary

Introduction

Design is believed to be the core of technology and engineering, which promotes experiential learning towards the development of a robust understanding (Dym et al, 2005; Atman et al, 2008). Design situates learning in real life contexts, involving ambiguity and multiple potentially viable solutions (Lammi and Becker, 2013), and promotes the development of students to adapt rapidly to diverse, complicated, and changing requirements (Dym et al, 2005; Lammi and Becker, 2013). Design thinking in the context of technology and engineering settings follows five stages (Erickson et al, 2005; Lindberg et al, 2010): Empathy, define, ideate, prototype, and test. In the design thinking process, defining the problem is a critical step to capturing what the students are attempting to accomplish through the design. The Point-Of-View (POV) statement (Figure 1), which includes three parts (user, need, insight), is one element of problem definition; this artifact often arises during the define stage and serves as a guideline during the entire design process (Sohaib et al, 2019)

Objectives

Results

Discussion

Conclusion