Test Reporting at a Large-Scale Austrian Logistics Organization: Lessons Learned and Improvement

Abstract

Context and Background. Software testing and test automation are important activities in software development where frequent requirements changes and the fast delivery of software increments are supported by traditional and agile development processes. Test reports are often used as “proof of evidence” for executed software tests. However, the practical impact of test reports, such as decision making and quality assessment, requires structured information which might not be available in sufficient quality. Goal. In this paper we (a) report on needs of test reports of different stakeholders at a large-scale Austrian logistics organization, (b) develop candidate improvement actions based on the state of the practice, and (c) conceptually evaluate selected improvement actions. Method. We used surveys and interviews to elicit needs and expected capabilities for test reporting and developed candidate improvement. We used expert discussions prioritize improvement actions in the organization context for further implementation. Results. Based on 23 recommended improvement actions, 14 were initially selected for implementation. Most of these accepted improvement action focus on regular test status reports and visualization aspects of test reports. Conclusion. Although test reporting is systematically applied in development processes, there is still some potential to improve test reports to gain (additional) benefits for related stakeholder.