Space Shuttle Flight Software Research Articles

Software Testing and Reliability Strategies

Software testing strategies are developed based on fault removal—deterministic and random. Based on literature search, it is appears that this is the first research that has evaluated these strategies. Deterministic fault removal is determined by the ranking of deviations of the time of fault occurrence from statistical control chart upper limit. Random removal of faults is selected by random number generation. Counterintuitively, random fault removal proved to be the better strategy based on improvement in reliability metrics across a series of tests. Contrariwise, deterministic fault removal was better when the cost effectiveness of fault removal was considered. Fault and failure data from three diverse projects were used to make the strategy assessment: NASA Space Shuttle flight software, Japanese University application, and a database application.

Updated Software Reliability Metrics

The Institute of Electrical and Electronics Engineers’ Standard Dictionary of Measures of the Software Aspects of Dependability (982.1) has been scheduled for updating. This standard was issued several years ago. Since then new software reliability metrics have been developed and evaluated. In addition, modifications of metrics in the standard have developed and evaluated. The objective of this paper is to describe, evaluate, and apply the new and modified metrics, using failure data from several releases of the NASA Space Shuttle flight software. Recognizing that users of standards have other applications, the methodology, equations, and prediction plots are explained so that reliability engineers can apply the metrics to their applications. The metrics are assessed from two standpoints: 1) identify metrics that support a specified purpose (e.g., demonstrate reliability growth) and 2) use these metrics to identify software releases that, based on reliability predictions, are ready to deploy and identify which software requires additional testing. Prediction accuracy is computed for all metrics and the metrics are compared based on the results.

Software Sequential Testing Model

A risk-driven reliability model and testing process is developed that borrows concepts from classical sequential testing methodology which is used for hardware. The model is adapted to software. Both consumer and producer risk are considered, reflecting the fact that the consumer (e.g., customer) and producer (e.g., contractor) have different perspectives concerning what they consider to be tolerable risks of software failure. Similarly, there is also a differentiation based on what the consumer and producer consider to be acceptable reliability. Test rules are specified for determining at each decision point in testing whether the software and the model prediction accuracy are acceptable. In addition, the test rules serve as stopping criteria for testing. Both empirical and predicted quantities are assessed. Based on experience in using the model, lessons learned are provided with the objective of improving the model and process for future applications. This model and test scenario is applied to a real application involving the NASA Space Shuttle flight software. The model and test scenario can be tailored to commercial applications, as well.

Experience report on using object‐oriented design for software maintenance

AbstractWe experimented with modifying the existing object‐oriented (OO) design and C++ code of a software reliability model. Our purpose was to assess the efficacy of OO methods for performing maintenance on mathematical software, using a real‐world system (NASA Space Shuttle flight software) to illustrate the approach. In this process, we used variants of UML diagrams to modify our design. We found that although a top‐down approach to software maintenance is normally a good idea, it was still necessary to modify the design once the realities of what could be accomplished in the C++ code came to light. As reliability and maintenance are intimately related, we developed reliability risk analysis to show how maintenance changes to our design and code could be used to measure risk. Another maintenance enhancement to the design and code is the use of reliability parameter analysis to assess, in the advance of prediction, the reliability of a set of software releases. We believe this is the first evaluation of software maintenance using OO methods. Copyright © 2007 John Wiley & Sons, Ltd.

RISK-DRIVEN SOFTWARE TESTING AND RELIABILITY

We develop a risk-driven reliability model and testing process that borrows concepts from classical sequential testing methodology that is used for hardware, with adaptation to software. Both consumer and producer risk are considered, reflecting the fact the consumer (e.g., customer) and producer (e.g., contractor) have different perspectives concerning what they consider to be tolerable risks of software failure. Similarly, there is also a differentiation based on what the consumer and producer consider to be acceptable reliability. Using the consumer-producer framework, we develop a model and process for executing sequential tests, based on software risk and reliability and model risk and reliability prediction accuracy. Test rules are specified for determining at each decision point in testing whether the software and the model prediction accuracy are acceptable. In addition, the test rules serve as stopping criteria for testing. Both empirical and predicted quantities are assessed. The test rules are integrated with several levels of criticality of software (i.e., the higher the criticality, the more stringent the tests). Based on our experience, we provide lessons learned with the objective of improving the model and process for future applications. We start the analysis by developing a model and process template based on the Poisson distribution of failures. This hypothetical example allows us to debug the model and process before it is applied to a real application involving the NASA Space Shuttle flight software.

Software metrics model for integrating quality control and prediction

A model is developed that is used to validate and apply metrics for quality control and quality prediction, with the objective of using metrics as early indicators of software quality problems. Metrics and quality factor data from the Space Shuttle flight software are used as an example. Our approach is to integrate quality control and prediction in a single model and to validate metrics with respect to a quality factor. Boolean discriminant functions (BDFs) were developed for use in the quality control and quality prediction process. BDFs provide good accuracy for classifying low quality software because they include additional information for discriminating quality: critical values. Critical values are threshold values of metrics that are used to either accept or reject modules when the modules are inspected during the quality control process. A series of nonparametric statistical methods is also used in the method presented. It is important to perform a marginal analysis when making a decision about how many metrics to use in the quality control and prediction process. We found that certain metrics are dominant in their effects on classifying quality and that additional metrics are not needed to accurately classify quality. This effect is called dominance. Related to the property of dominance is the property of concordance, which is the degree to which a set of metrics produces the same result in classifying software quality. A high value of concordance implies that additional metrics will not make a significant contribution to accurately classifying quality; hence, these metrics are redundant.

Software metrics model for quality control

A model is developed for validating and applying metrics for quality control, using the Space Shuttle flight software as an example. We validate metrics with respect to a quality factor in accordance with the metrics validation methodology previously developed. Boolean discriminant functions (BDFs) are developed for use in the quality control process. These functions make fewer mistakes in classifying software that is low quality than is the case when linear vectors of metrics are used because the BDFs include additional information for discriminating quality: critical values. Critical values are threshold values of metrics that are used to either accept or reject modules when the modules are inspected during the quality control process. A series of nonparametric statistical methods is used to: 1) identify a set of candidate metrics for further analysis; 2) identify the critical values of the metrics, and 3) find the optimal function of metrics and critical values. A marginal analysis should be performed when making a decision about how many metrics to use in a quality control process. Certain metrics are dominant in their effects on classifying quality and additional metrics are not needed to accurately classify quality. This effect is called dominance. Related to the property of dominance is the property of concordance, which is the degree to which a set of metrics produces the same result in classifying software quality. A high value of concordance implies that additional metrics will not make a significant contribution to accurately classifying quality; hence, these metrics are redundant.

Measuring and evaluating maintenance process using reliability, risk, and test metrics

In analyzing the stability of a software maintenance process, it is important that it is not treated in isolation from the reliability and risk of deploying the software that result from applying the process. Furthermore, we need to consider the efficiency of the test effort that is a part of the process and a determinate of reliability and risk of deployment. The relationship between product quality and process capability and maturity has been recognized as a major issue in software engineering based on the premise that improvements in the process will lead to higher-quality products. To this end, we have been investigating an important facet of process capability-stability-as defined and evaluated by trend, change and shape metrics, across releases and within a release. Our integration of product and process measurement serves the dual purpose of using metrics to assess and predict reliability and risk and to evaluate process stability. We use the NASA Space Shuttle flight software to illustrate our approach.

Measuring and evaluating maintenance process using reliability, risk, and test metrics

In analyzing the stability of a software maintenance process, it is important that it is not treated in isolation from the reliability and risk of deploying the software that result from applying the process. Furthermore, we need to consider the efficiency of the test effort that is a part of the process and a determinate of reliability and risk of deployment. The relationship between product quality and process capability and maturity has been recognized as a major issue in software engineering based on the premise that improvements in the process will lead to higher-quality products. To this end, we have been investigating an important facet of process capability-stability-as defined and evaluated by trend, change and shape metrics, across releases and within a release. Our integration of product and process measurement serves the dual purpose of using metrics to assess and predict reliability and risk and to evaluate process stability. We use the NASA Space Shuttle flight software to illustrate our approach.

Controlling and predicting the quality of space shuttle software using metrics

Software quality metrics have potential for helping to assure the quality of software on large projects such as the Space Shuttle flight software. It is feasible to validate metrics for controlling and predicting software quality during design by validating metrics against a quality factor. Quality factors, like reliability, are of more interest to customers than metrics, like complexity. However quality factors cannot be collected until late in a project. Therefore the need arises to validate metrics, which developers can collect early in a project, against a quality factor. We investigate the feasibility of validating metrics for controlling and predicting quality on the Space Shuttle. The key to the approach is the use of validated metrics for early identification and resolution of quality problems.

Validating metrics for ensuring Space Shuttle flight software quality

In this article, we cover the validation of software quality metrics for the Space Shuttle. Experiments with Space Shuttle flight software show that the Boolean OR discriminator function can successfully validate metrics for controlling and predicting quality. Further, we found that statement count and node count are the metrics most closely associated with the discrepancy reports count, and that with data smoothing their critical values can be used as predictors of software quality. We are continuing our research and comparing validated metrics with actual results in an attempt to determine whether the use of additional metrics provides sufficiently greater discriminative power to justify the increased inspection costs. >

Structured IV & V for the space shuttle flight software

No abstract available.