Supporting Storytelling With Evidence in Holistic Review Processes: A Participatory Design Approach

Abstract

Review processes involve complex and often subjective decision-making tasks in which individual reviewers must read and rate submissions, such as a college application, along many relevant dimensions and typically with a rubric in mind. A common part of the work is committee review, where individual reviewers meet to discuss the merits of a particular submission in order to recommend an accept or reject decision. Prior work indicates that visualization and sensemaking support may be beneficial in such processes where reviewers must present the "story" of the applicant under question. We conducted a series of participatory design workshops with reviewers in the domain of holistic college admissions to better understand the challenges and opportunities regarding storytelling. Based on these workshops, we contribute a characterization for how reviewers in this domain construct visual stories, we provide guidance for designing for evidence capture and storytelling, and we draw parallels and distinctions between this domain and other reviewing domains.