Optimizing Regression Testing of Software for the Consolidated Automated Support System

Abstract

The use of Commercial-Off-The-Shelf (COTS) operating systems in newer generations of Automatic Test Equipment (ATE) has introduced challenges that did not exist with legacy ATE. Unfortunately, COTS instruments and ATE operating systems do not have well documented test sequence execution time. COTS operating systems also require frequent updates due to cyber security concerns, optimization, and obsolescence. These updates, in turn, can affect a Test Program Sets' (TPSs) test sequence execution time and in the worst cases, generate errors. During initial TPS development, the test engineer accounts for any instrument and operating system latency during the TPS integration phase. Because of changes in this latency, the TPS will need to be re-certified whenever a new operating system update is installed prior to releasing it to fleet. This requires maintainers to ensure the integrity of the TPS with extensive regression testing and performing re-integration. For the US Navy's Consolidated Automated Support System (CASS) family of testers, which supports over 2000 unique avionics components, this is a very expensive and labor-intensive effort. Due to the complexity of the TPSs, a highly skilled engineering team is needed to correct test failures that occur during regression testing. As legacy CASS approaches sundown, and is replaced by newer versions of CASS, this regression testing effort will increase significantly. A newer, more automated, and less labor intensive process for regression testing needs to be developed. This paper will highlight the statistical analysis of TPS log data from the CASS family of testers and focus on the test sequence execution time in order to reduce cycle time for regression testing of new software releases to the fleet. Driven by the conclusions of the analysis, an automated tool will be developed to allow software engineers to adjust timing in the test executive in order to minimize the labor hours needed for testing. By reducing the labor needed to certify TPSs, maintenance costs can be optimized to better serve the fleet and Depot customers.