Taming complexity while gaining efficiency: Requirements for the next generation of test automation tools

Abstract

The current product development environment for mechatronic systems is characterized by tight budgets, reduced development times and immense complexity across the industries - aerospace, automotive, commercial vehicles, etc. A huge emphasis is being given to increase the efficiency in product testing and to make the process more productive in increasing quality, while being cost effective. Companies are seeking software testing tools that offer a comprehensive solution that helps in achieving this goal. Automated software testing for both hardware and the software components is one of the ways the industries are trying to gain efficiency in testing. Given the fact that current mechatronic systems are quite complex, have large numbers and types of IO channels, and are implementing distributed real-time control with multiple communication protocols, a test automation tool that incorporates and tests all of the system functionality is highly desirable. In particular, due to the focus on software quality, as more and more functions are added to the network of Electronic Control Units (ECU), a greater emphasis is laid on the requirements for a comprehensive automated testing tool. The efficiency in development and planned increase in reuse of test assets is an important consideration for efficiency gain. Bringing clarity to the test description would improve readability and maintainability of these test assets. This paper shows the lessons learned from a similar evolution in Model-Based Design, moving from handwritten code to model-based graphical executable representation, can be applied to the testing arena to develop better and more efficient tests. To make the testing process efficient, the test tool should be able to not just execute tests, but organize the test results and present them in a manner easily understood by the end user. The user should be easily able to select tests to be executed and visualize execution results of those tests without need of external data analysis and reporting tools. A user should also be able to gather data related to test execution for reporting on overall software quality metrics. In the case of real-time embedded systems, the traditional approach for software testing of test execution, data capture, analysis and reporting creates a lot of inefficiency and repeated testing. A new approach to run test evaluation in parallel to test execution, using the so-called real-time testing solution, eliminates these inefficiencies and results in significant gains in testing time and improving utilization of the test infrastructure. Globalized, distributed development environments may lead to organizations adopting a variety of tool chains and test platforms. Therefore, to gain efficiency, it is desired that the test automation solution should offer easy integration with other third-party test platforms and be independent of the hardware to protect the investment in the long run. Basing such integration interface on established standards gives the user freedom to make choices. Some standards, such as HIL API from ASAM, are being proposed for test systems in the automotive industry. The paper reviews applicability and benefits of such standards. Various industry standards, such as automotive functional safety standard ISO 26262, IEC 61508, IEC 62304 and DO-178C, have addressed requirements for software testing and test tools. It is therefore critical that the selected test automation tool provide support for the prescribed methods and support the standard compliant development process. Tool qualification is a critical topic addressed by these standards and is discussed in this paper. Further, increasingly important requirements such as integration with the overall software development process, management of tests and test data, test reusability, hardware platform independence, test development and execution efficiency, etc. are discussed.