Metrics for evaluating the performance of complex engineering system health monitoring models

Abstract

Recent efforts to apply prognostics and health management (PHM) practices towards the risk management of complex engineering systems (CES) traditionally structured using probabilistic risk assessments (PRA) provide more opportunities for modeling and dynamically monitoring system-level heath. However, it is unclear what metrics should be used to assess and compare the performance of these models. PHM performance metrics center around prediction accuracy without considering operational realities of the system, while risk management models are not easily compared. A new set of model metrics is needed for this evolving space.This paper identifies and defines multi-dimensional metrics to assess the performance, effectiveness and efficiency of system-level health monitoring models based on alignment with specific phases of model design and operation. Using high-interest model aspects and metrics from PHM and PRA, a taxonomy of measurable characteristics is presented for systematically comparing model designs. The verification process for this taxonomy is described as well as an illustrative example for utilizing the metrics in model design decision. The comprehensive set of metrics presented in this work enables rigorous and scientific comparisons of developed and proposed system models, enabling both PRA and PHM communities to analytically assess system-level health monitoring models beyond accuracy-driven measurements.