Designing Academic Writing Analytics for Civil Law Student Self-Assessment

Simon Knight,Simon Buckingham Shum,Philippa Ryan,Xiaolong Wang,Ágnes Sándor

doi:10.1007/s40593-016-0121-0

Simon Knight, Simon Buckingham Shum + Show 3 more

Open Access

https://doi.org/10.1007/s40593-016-0121-0

Copy DOI

Abstract

Research into the teaching and assessment of student writing shows that many students find academic writing a challenge to learn, with legal writing no exception. Improving the availability and quality of timely formative feedback is an important aim. However, the time-consuming nature of assessing writing makes it impractical for instructors to provide rapid, detailed feedback on hundreds of draft texts which might be improved prior to submission. This paper describes the design of a natural language processing (NLP) tool to provide such support. We report progress in the development of a web application called AWA (Academic Writing Analytics), which has been piloted in a Civil Law degree. We describe: the underlying NLP platform and the participatory design process through which the law academic and analytics team tested and refined an existing rhetorical parser for the discipline; the user interface design and evaluation process; and feedback from students, which was broadly positive, but also identifies important issues to address. We discuss how our approach is positioned in relation to concerns regarding automated essay grading, and ways in which AWA might provide more actionable feedback to students. We conclude by considering how this design process addresses the challenge of making explicit to learners and educators the underlying mode of action in analytic devices such as our rhetorical parser, which we term algorithmic accountability.

Highlights

IntroductionAnalytical writing is a key skill in learning, in higher education contexts, and for employment in most knowledge-intensive professions (National Commission On Writing 2003; OECD and Statistics Canada 2010)
Writing as a Key Disciplinary SkillCritical, analytical writing is a key skill in learning, in higher education contexts, and for employment in most knowledge-intensive professions (National Commission On Writing 2003; OECD and Statistics Canada 2010)
In the remainder of this paper, we briefly introduce the Academic Writing Analytics (AWA) web application, describing its architecture and user interface

Summary

Introduction

Analytical writing is a key skill in learning, in higher education contexts, and for employment in most knowledge-intensive professions (National Commission On Writing 2003; OECD and Statistics Canada 2010). A 1992 US report, commonly known as the MacCrate report (The Task Force on Law Schools and the Profession: Narrowing the Gap 1992), notes that it is key for lawyers to learn effective communication methods (including analytical writing), there is a disconnect between the practice, and legaleducation of, lawyers with too little focus on this communication in legal training. In the context described in this work, across the qualifications offered by the University of Technology Sydney (UTS) Law Faculty, critical analysis and evaluation, research skills (to find, synthesize and evaluate relevant information), and communication and collaboration (using English effectively to inform, analyse, report and persuade in an appropriate – often written – medium), are all highlighted as core graduate attributes. REFERENCING: Written expression, footnotes and bibliography in accordance with Australian guide to legal citation (AGLC) 3rd edition

Objectives

Methods

Results

Discussion

Conclusion