Evidence Engineering

Abstract

The costs of computing have decreased a billion times over half a century. The focus of software engineering has consequently transformed from trying to squeeze as much as possible from every compute cycle and of every bit of memory, to improving developer productivity, and, as of late, to engineering user experiences and behaviors. As computing becomes a commodity, software is omnipresent in all parts of life and, therefore, it either helps end users make decisions or it makes decisions for them. Because most users are not able to understand software systems or articulating their needs, software systems have both to collect massive amounts of operational data related to user activities and to analyze and use that data to provide user experiences that lead to desired outcomes, e.g., increasing sales revenue or the quality of software (if the user happens to be a software developer). It no longer suffices to deliver software that requires, for example, an entry field for a specific piece of data. Instead, the software has to ensure that users can and will enter the relevant data or it has to obtain the data by observing user behavior. Moreover, the software has to ensure that the resulting data reflects the intended quantities, and that the quality of that data is sufficient to make important decisions either automatically of with human support. Such software is engineered to provide accurate and actionable evidence and, therefore, it requires novel approaches to design, implement, test, and operate it. The criteria for success demand much more than performing a well-defined task according to specification. Software has to provide evidence that is both accurate and also leads to the intended user behavior. In contexts where the desired user behaviors are relatively well defined, some existing software systems achieve these goals through detailed measurement of behavior and massive use of AB testing (in which two samples of users provided slightly different versions of software in order to estimate the effect these differences have on user activity). It is not clear if and how these approaches could generalize to the setting where the desired behaviors are less clearly defined or vary among users. As operation and measurement are becoming increasingly a part of software development, the separation between the software tools and end-user software are increasingly blurred. Similarly, the measurement associated with testing and use of software is increasingly becoming an integral part of the software delivered to users. Software engineering needs to catch up with these realities by adjusting the topics of its study. Software construction, development, build, delivery, and operation will become increasingly critical tools and an integral part of the software system. Simply concerning ourselves with architectures and languages to support scalable computation and storage will not be enough. Software systems will have to produce compelling evidence, not simply store or push bits around. Software engineering will, therefore, need to become evidence engineering.