Predicting essay quality from search and writing behavior

Pertti Vakkari,Michael Völske,Matthias Hagen,Benno Stein,Martin Potthast

doi:10.1002/asi.24451

Abstract

AbstractFew studies have investigated how search behavior affects complex writing tasks. We analyze a dataset of 150 long essays whose authors searched the ClueWeb09 corpus for source material, while all querying, clicking, and writing activity was meticulously recorded. We model the effect of search and writing behavior on essay quality using path analysis. Since the boil‐down and build‐up writing strategies identified in previous research have been found to affect search behavior, we model each writing strategy separately. Our analysis shows that the search process contributes significantly to essay quality through both direct and mediated effects, while the author's writing strategy moderates this relationship. Our models explain 25–35% of the variation in essay quality through rather simple search and writing process characteristics alone, a fact that has implications on how search engines could personalize result pages for writing tasks. Authors' writing strategies and associated searching patterns differ, producing differences in essay quality. In a nutshell: essay quality improves if search and writing strategies harmonize—build‐up writers benefit from focused, in‐depth querying, while boil‐down writers fare better with a broader and shallower querying strategy.

Highlights

Users of information systems seek support in various daily activities, and while search processes, that is, querying and assessment of search results, likely affect the outcome of the tasks behind the search, only few studies have explored these associations
Our study is one of the first attempts at modeling task outcome by search process variables taking into account the use of information found for the task
Its results significantly extend our understanding of how search process and writing effort contribute to the quality of essay text and how writing strategies moderate the contribution of these variables

Summary

Introduction

Users of information systems seek support in various daily activities, and while search processes, that is, querying and assessment of search results, likely affect the outcome of the tasks behind the search, only few studies have explored these associations. Performance is assessed by the output of the search, rather than by its outcome, that is, the resulting benefits to the task at hand (Belkin, 2010; Järvelin et al, 2015; Vakkari, 2003). Since the utility of search results depends on how much they contributes to the outcome of an underlying task, the use of information retrieved is a link between search process and task outcome (Järvelin et al, 2015; Vakkari, Völske, Potthast, Hagen, & Stein, 2019). We analyze to what extent search process—i.e., querying, result examination, and information selection in documents—and text

Objectives

Results

Conclusion