Framework for SQL Error Message Design: A Data-Driven Approach

Abstract

Software developers use a significant amount of time reading and interpreting error messages. However, error messages have often been based on either anecdotal evidence or expert opinion, disregarding novices, who arguably are the ones who benefit the most from effective error messages. Furthermore, the usability aspects of Structured Query Language (SQL) error messages have not received much scientific attention. In this mixed-methods study, we coded a total of 128 error messages from eight database management systems (DBMS), and using data from 311 participants, analysed 4,796 queries using regression analysis to find out if and how acknowledged error message qualities explain SQL syntax error fixing success rates. Additionally, we performed a conventional content analysis on 1,505 suggestions on how to improve SQL error messages, and based on the analysis, formulated a framework consisting of nine guidelines for SQL error message design. The results indicate that general error message qualities do not necessarily explain query fixing success in the context of SQL syntax errors and that even some novel NewSQL systems fail to account for basic error message design guidelines. The error message design framework, and examples of its practical applications shown in this study, are applicable in educational contexts, as well as by DBMS vendors in understanding novice perspectives in error message design.