Validation of Assurance Case for Dynamic Systems

Abstract

Aviation safety has accrued and applied decades of understanding on known risks and effective mitigations. That knowledge captured in compliance standards - can be tested for predictable outcomes. Autonomy that involves learning systems tend to be dynamic and may continuously be adapting to their environment. Such continuous adaptation has inherent unknown risks depending upon the guardrails imposed on the learning systems. Design assurance and use of traditional standards are inadequate for these dynamic systems.Assurance Cases are used to present an argument for the assurance of systems. Dynamic systems require that assurance cases be continuously validated. One method of validation is using real time collection of Safety Performance Indicators (SPIs) which are crafted during the development of the system. This paper presents the need for SPIs and methods for creating and nurturing the SPIs to help all stakeholders. This method shadows regulations and allows risk-based approvals that may be applied for both conventional and for novel technology.Aviation is facing enormous growth in autonomous technology, reuse of components of unknown pedigree, and new aircraft designs that do not fit into Type classification. Mitigations are more heavily connected to operations, training, and other components of the ecosystem itself. The challenge is to make an assurance case for vehicles within the ecosystem. The automobile industry, which is similarly challenged by dynamically changing autonomous systems, is finding some possible solutions to build safer systems.UL 4600, a Standard for Safety for the Evaluation of Autonomous Products applies to fully autonomous road vehicles. The goal-based, technology-neutral features of UL 4600 have been extended to apply to aviation. So applied, the assurance process is adaptable to innovation and discovery while encouraging the current practices of standards compliance and taking a System of Systems (SoS) view. It proposes an assurance case that is an organized argument that a system is acceptable for its intended use with respect to specified concerns (such as safety, security, correctness). This paper gives guidance for validation of an assurance case through monitoring SPI within the operational context. The method by monitoring safety performance indices in the operational environment provides continued validation even as the ecosystem, components and controls change.For approval of novel systems including UAS and AAM, with features that do not lend themselves to traditional compliance methods, regulators have embraced the Safety Continuum perspective, which focuses on safety performance achieving expected outcomes. The performance-based assurance methods can be used with initially wider performance margins for certification of novel products, components of unknown pedigree, and autonomous vehicles. As the performance range is better known the margins can be decreased.Further, this paper recognizes that a one-time initial approval/acceptance is not adequate for learning systems and novel features. The continued validation through performance supports fast-paced development and product evolution. The initial assurance case for a product can limit risk through a closed environment until the margin for some unknowns is validated. for example, if the performance of collision avoidance function using new technology is not known, larger alert limits may be implemented until more confidence is gained after validating the assurance case via SPIs.The approach of monitored SPI throughout the life of the product is now feasible with the aid of big data processing. The aviation industry is already using similar methods for identifying maintenance problems. As systems grow more autonomous, more machine-to-machine exchanges are involved, making it easy to extend the monitoring and prediction practices to SPI.The method also allows for variants and derivatives of the baseline to have their own assurance case within the context of the baseline argument. The key is replacing design approval with through-life assurance that connects continuous operational safety into both the design and airworthiness determinations. The determination is predicated on the monitored SPIs and predicted performance of the product remaining consistent with the assurance argument predictions. This enables even complex automated products to be audited for airworthiness with an evolving ecosystem based on monitored and predictive data.Another advantage of the performance-based assurance case is the public comprehensibility of safety. With SPIs and predictions of performance the automobile segment has paved the way for public scrutiny of automated vehicles. The use of SPIs in aeronautical product assurance will facilitate transparency. This could be accomplished through appropriate dashboards to aid public perception and explain events and precautions taken during the evolution toward more autonomous aviation vehicles. This could reflect a stepwise evolution of complexity.This paper explores how the aviation industry can apply performance-based assurance case methods to assure new and novel as well as systems of unknown pedigree. The same framework could then be extended to autonomous systems and new types of aircraft which do not fit the current Type classification. One of the major benefits of this technology agnostic method are faster risk-based approvals of novel technology within a Safety Continuum.