Investigating Attorney Trust in Machine-enabled Legal Research: A Mixed Methods Approach

Abstract

ROSS is a new legal research tool leveraging artificial intelligence. This tool is somewhat unique both in terms of its interface and underlying functionality. ROSS’s deviation from more traditional systems may have an impact on the adoption of this new technology. The researchers in the current study conducted a mixed methods evaluation of ROSS to assess user experience and likelihood of adoption with a particular emphasis on evaluating user trust in the technology. The researchers conducted simulation interviews with eleven experienced bankruptcy attorneys. In addition, a human-computer trust survey was administered to assess user trust in the technology at various time points throughout the study. This quantitative survey measured both cognitive and affect-based trust and revealed a substantial increase in trust with exposure to the system. Participants were particularly impressed with attributes of the system that are believed to form the cognitive basis of trust, such as the tool’s reliability and understandability. Despite these positive findings, our qualitative assessment of experiential trust was mixed and most users concluded that they would not use ROSS as their primary research tool. This decision may be attributed to a lack of faith in the technology and concerns related to ROSS’s usability. The simulations were conducted between February and April of 2017. It should be noted that ROSS has been updated since this time and certain findings may no longer be representative of the current platform.