Probabilistic Verification Research Articles

Safe Networked Robotics With Probabilistic Verification

Safe Networked Robotics With Probabilistic Verification

Process Algebraic Approach for Probabilistic Verification of Safety and Security Requirements of Smart IoT (Internet of Things) Systems in Digital Twin

Process algebra can be considered one of the most practical formal methods for modeling Smart IoT Systems in Digital Twin, since each IoT device in the systems can be considered as a process. Further, some of the algebras are applied to predict the behavior of the systems. For example, PALOMA (Process Algebra for Located Markovian Agents) and PACSR (Probabilistic Algebra of Communicating Shared Resources) process algebras are designed to predict the behavior of IoT Systems with probability on choice operations. However, there is a lack of analytical methods in the algebras to predict the nondeterministic behavior of the systems. Further, there is no control mechanism to handle undesirable nondeterministic behavior of the systems. In order to overcome these limitations, this paper proposes a new process algebra, called dTP-Calculus, which can be used (1) to specify the nondeterministic behavior of the systems with static probability, (2) verify the safety and security requirements of the nondeterministic behavior with probability requirements, and (3) control undesirable nondeterministic behavior with dynamic probability. To demonstrate the feasibility and practicality of the approach, the SAVE (Specification, Analysis, Verification, Evaluation) tool has been developed on the ADOxx Meta-Modeling Platform and applied to a SEMS (Smart Emergency Medical Service) example. In addition, a miniature digital twin system for the SEMS example was constructed and applied to the SAVE tool as a proof of concept for Digital Twin. It shows that the approach with dTP-Calculus on the tool can be very efficient and effective for Smart IoT Systems in Digital Twin.

Impacts of precipitation uncertainty on hydrological ensemble simulations over the Ganjiang River basin

Study regionThe Ganjiang River basin (113°−117°E, 24°−30°N; 83,374 km2) is a large watershed with complex topography in the Poyang Lake basin in Jiangxi province, China. Study focusThis study evaluates three quantitative precipitation estimates (QPEs) over the Ganjiang River basin, namely the China Gauged-Based Daily Precipitation Analysis (CGDPA) data, the Precipitation Estimation from Remotely Sensed Information using Artificial Neural Networks-Climate Data Record (PERSIANN-CDR) data, and the Asian Precipitation - Highly-Resolved Observational Data Integration Towards Evaluation of Water Resources (APHRODITE). The study also investigates the impacts of precipitation uncertainty on hydrological ensemble streamflow simulations using the three QPEs as precipitation inputs of the Variable Infiltration Capacity (VIC) hydrological model. New hydrological insights for the regionAPHRODITE underestimates precipitation compared to CGDPA, while PERSIANN-CDR shows greater spatiotemporal variability. The ensemble mean streamflow demonstrates greater improvement compared to the results obtained from a single parameter set. Among the three QPEs, the simulations forced by CGDPA show the best deterministic and probabilistic verification scores, followed by APHRODITE. PERSIANN-CDR tends to underestimate evaporation and leads to the lowest score of ensemble streamflow simulations, but shows advantages in simulating extremely low streamflow. The study highlights that high-density gauge-based QPEs remain the most accurate source of precipitation inputs for reliable hydrological simulations, while satellite-gauge merged QPEs can provide valuable inputs for hydrological simulations over the basins where meteorological stations are scarce.

Automatic Synthesis of FSMs for Enforcing Non-functional Requirements on MPSoCs Using Multi-objective Evolutionary Algorithms

Embedded system applications often require guarantees regarding non-functional properties when executed on a given MPSoC platform. Examples of such requirements include real-time, energy, or safety properties on corresponding programs. One option to implement the enforcement of such requirements is by a reactive control loop, where an enforcer decides based on a system response (feedback) how to control the system, e.g., by adapting the number of cores allocated to a program or by scaling the voltage/frequency mode of involved processors. Typically, a violation of a requirement must either never happen in case of strict enforcement, or only happen temporally (in case of so-called loose enforcement). However, it is a challenge to design enforcers for which it is possible to give formal guarantees with respect to requirements, especially in the presence of typically largely varying environmental input (workload) per execution. Technically, an enforcement strategy can be formally modeled by a finite state machine (FSM) and the uncertain environment determining the workload by a discrete-time Markov chain. It has been shown in previous work that this formalization allows the formal verification of temporal properties (verification goals) regarding the fulfillment of requirements for a given enforcement strategy. In this article, we consider the so-far-unsolved problem of design space exploration and automatic synthesis of enforcement automata that maximize a number of deterministic and probabilistic verification goals formulated on a given set of non-functional requirements. For the design space exploration (DSE), an approach based on multi-objective evolutionary algorithms is proposed in which enforcement automata are encoded as genes of states and state transition conditions. For each individual, the verification goals are evaluated using probabilistic model checking. At the end, the DSE returns a set of efficient FSMs in terms of probabilities of meeting given requirements. As experimental results, we present three use cases while considering requirements on latency and energy consumption.

Physically based stochastic perturbations improve a high‐resolution forecast of convection

AbstractA physically based stochastic perturbation (PSP) scheme has been implemented in the convection‐permitting ICON‐D2 ensemble prediction system at Deutscher Wetterdienst (DWD) and run for a three‐month trial experiment in summer 2021. The scheme mimics the impact of boundary‐layer turbulence on the smallest resolved scales and impacts convective precipitation in particular. A weather‐regime‐dependent systematic evaluation shows that PSP efficiently increases ensemble spread of precipitation in weak synoptic forcing, while producing realistic convective structures. During strong forcing, the effect of the scheme is negligible, as expected by design. A probabilistic verification shows improvements in the forecast skill of other variables as well, especially the spread‐to‐skill ratio, but identifies starting points for further improvements of the method.

Information-Based Probabilistic Verification Scores for Two-Dimensional Ensemble Forecast Data: A Madden–Julian Oscillation Index Example

Abstract Probabilistic forecasting is a common activity in many fields of the Earth sciences. Assessing the quality of probabilistic forecasts—probabilistic forecast verification—is therefore an essential task in these activities. Numerous methods and metrics have been proposed for this purpose; however, the probabilistic verification of vector variables of ensemble forecasts has received less attention than others. Here we introduce a new approach that is applicable for verifying ensemble forecasts of continuous, scalar, and two-dimensional vector data. The proposed method uses a fixed-radius near-neighbors search to compute two information-based scores, the ignorance score (the logarithmic score) and the information gain, which quantifies the skill gain from the reference forecast. Basic characteristics of the proposed scores were examined using idealized Monte Carlo simulations. The results indicated that both the continuous ranked probability score (CRPS) and the proposed score with a relatively small ensemble size (<25) are not proper in terms of the forecast dispersion. The proposed verification method was successfully used to verify the Madden–Julian oscillation index, which is a two-dimensional quantity. The proposed method is expected to advance probabilistic ensemble forecasts in various fields. Significance Statement In the Earth sciences, stochastic future states are estimated by solving a large number of forecasts (called ensemble forecasts) based on physical equations with slightly different initial conditions and stochastic parameters. The verification of probabilistic forecasts is an essential part of forecasting and modeling activity in the Earth sciences. However, there is no information-based probabilistic verification score applicable for vector variables of ensemble forecasts. The purpose of this study is to introduce a novel method for verifying scalar and two-dimensional vector variables of ensemble forecasts. The proposed method offers a new approach to probabilistic verification and is expected to advance probabilistic ensemble forecasts in various fields.

How to assess climate change impact models: uncertainty analysis of streamflow statistics via approximate Bayesian computation (ABC)

ABSTRACT Climate change impact models (CCIMs) suffer from inherent bias, uncertainty, and asynchronous observations in the baseline period. To overcome these challenges, this study introduces a methodology to assess CCIMs in the baseline period using the uncertainty analysis of streamflow statistics via the approximate Bayesian computation (ABC) post-processor, which infers the residual error model parameters based on summary statistics (signatures). As an illustrative case study, we analyzed the climate change projections of the fifth assessment report of the United Nations intergovernmental panel on climate change (AR5 - IPCC) of the monthly streamflow in the upper Oria catchment (Spain) with deterministic and probabilistic verification frameworks to assess the ABC post-processor outputs. In addition, the ABC post-processor is evaluated against the ensemble (reference method). The results show that the ABC post-processor outperformed the ensemble method in all verification metrics, and the ensemble method has reasonable reliability but exhibited poor sharpness. We suggest that the ensemble method should be complemented with the ABC post-processor for climate change impact studies.

Probabilistic verification of diagnosability for a certain class of timed stochastic systems

This paper addresses diagnosability for a class of timed stochastic discrete event systems that are modeled with labeled continuous time Markov models. A large variety of fault patterns is considered and the paper aims to define and propose conditions for weak notions of diagnosability of fault patterns in a time stochastic framework. The proposed approach is based on a state isolation perspective. Logical and probabilistic verifiers that result by first taking the product of the system with the fault pattern, followed by the parallel composition of the resulting structure with its logical observer are designed in this perspective. The advantage of this schema is to separate explicitly the system from the pattern we are interested in, as well as from the sensoring and observational aspects. Consequently, the verifiers allow the explicit tracking of the information about the first occurrence of a fault pattern and the preservation of not only the logical information provided by the labeling function but also the timing aspects imposed by the dynamics of the system.

Improving the predictability of the Qendresa Medicane by the assimilation of conventional and atmospheric motion vector observations. Storm-scale analysis and short-range forecast

Abstract. The coastal population in the western Mediterranean Basin is frequently affected by high-impact weather events that produce huge economic and human losses. Among the wide spectrum of maritime severe weather events, tropical-like Mediterranean cyclones (a.k.a. medicanes) draw particular attention, specially due to their poor predictability. The accurate prediction of this kind of event still remains a key challenge to the weather forecast community, mainly because of (i) errors in the initial conditions, (ii) lack of accuracy of modeling micro-scale physics processes and (iii) chaotic behavior inherent to numerical weather prediction models. The 7 November 2014 Qendresa Medicane, that took place over the Sicilian channel affecting the islands of Lampedusa, Pantelleria and Malta, was selected for this study because of its extremely low predictability behavior in terms of its track and intensity. To enhance the prediction of Qendresa, a high-resolution (4 km) ensemble-based data assimilation technique, known as ensemble Kalman filter (EnKF), is used. In this study, both in situ conventional and satellite-derived observations are assimilated with the main objective of improving Qendresa's model initial conditions and thus its subsequent forecast. The performance of the EnKF system and its impact on the Qendresa forecast are quantitatively assessed using different deterministic and probabilistic verification methods. A discussion in terms of the relevant physical mechanisms adjusted by the EnKF is also provided. Results reveal that the assimilation of both conventional and satellite-derived observations improves the short-range forecasts of the trajectory and intensity of Qendresa. In this context, the relevance of assimilating satellite-derived observations to improve the pre-convective estimation of Qendresa's upper-level dynamics is shown, which is key to obtain a realistic track and intensity forecast of this event.

On the verification of ensemble precipitation forecasts over the Godavari River basin

The ensemble forecasts from Numerical Weather Prediction (NWP) models are considered as essential elements for operational hydrologic forecasting. However, these forecasts are burdened with significant inherent biases and cannot be used directly for hydrologic forecasting. Therefore, the present study examines the capability of ensemble precipitation forecasts obtained from two NWP models i.e., a 20-member ensemble from the National Center for Environmental Prediction (NCEP) and a 50-member ensemble from the European center for Medium-term Weather Forecasts (ECMWF). The skill of 70-member multi-model grand ensemble (MME) generated from the NCEP and ECMWF forecasts is also verified in this study. Furthermore, post-processing of the raw ensemble members was carried out using two methods (Quantile Mapping (QM) and Quantile Regression Forests (QRF)) in order to minimize the inherent biases. The analysis was performed over subbasins of an Indian river basin namely the Godavari River Basin. Here, a set of deterministic and probabilistic verification measures (forecast error box plots, correlation coefficient, Relative Mean Error, mean Continuous Ranked Probability Score, spread-skill relationship, rank histograms, reliability diagram, Area under Relative Operating Characteristic Curve) were employed to assess various quality attributes of the forecasts. The findings from the study suggests that QRF post-processed NCEP and MME forecasts are comparatively better than other forecasts in terms of all employed verification measures over all subbasins. The QRF post-processed forecasts were found to be superior than QM post-processed and raw forecasts over all subbasins. The adopted post-processing methods resulted in improved forecasts at shorter lead times. However, the skill of both raw and post-processed forecasts declines at higher lead times over all subbasins.

Analog Ensemble Forecasting System for Low-Visibility Conditions over the Main Airports of Morocco

Low-visibility conditions (LVC) are a common cause of air traffic, road, and sailing fatalities. Forecasting those conditions is an arduous challenge for weather forecasters all over the world. In this work, a new decision support system is developed based on an analog ensemble (AnEn) method to predict LVC over 15 airports of Morocco for 24 forecast hours. Hourly forecasts from the AROME model of eight predictors were used to select the skillful analogs from 2016 to 2018. The verified hourly observations were used as members of the ensemble. The developed ensemble prediction system (EPS) was assessed over 1 year (2019) as a single-value forecast and as a probabilistic forecast. Results analysis shows that AnEn outperforms persistence and its best performances are perceived generally during night and early-morning lead times. From continuous verification analysis, AnEn forecasting errors are found to be location- and lead-time-dependent and become higher for low-visibility cases. AnEn draws an averaged Centered Root Mean Square Error of about 1500 m for all visibilities, 2000 m for fog and 1500 m for mist. As an EPS, AnEn is under-dispersive for all lead times and draws a positive bias for fog and mist events. For probabilistic verification analysis, AnEn visibility forecasts are converted to binary occurrences depending on a set of thresholds from 200 m to 6000 m by a step of 200 m. It is found that the averaged Heidke Skill Score for AnEn is 0.65 for all thresholds. However, AnEn performance generally becomes weaker for fog or mist events prediction.

Practical concepts for the use of probabilistic methods in the structural analysis and reassessment of existing bridges - presentation of latest research and implementation

In structural engineering and bridge construction, probabilistic calculations are only performed in a few exceptional cases. The ability to directly use information from the existing structure is an important advantage of probabilistic methods. Geometric data or the position and quantity of the installed reinforcing steel and tendons can be measured. Results from monitoring of the traffic can be used to generate structure-specific load models. Many clients are still very sceptical about the results of probabilistic calculations, even though there are several examples of successful application. The use of probabilistic verification formats is explicitly permitted by the German guideline for the assessment of existing bridges (German: Nachrechnungsrichtlinie). Since a high degree of special knowledge is required for the application of probabilistic calculations, it is difficult to establish the potential of this verification method in practice. It is necessary to enable the practical engineer to utilize the advantages of probabilistic calculations and to include the actual structural conditions into the calculation model, but without the need for all the special knowledge. Structure-specific partial safety factors include measured data and can be used for the well-known and in the codes established semi-probabilistic design concept. The authors present examples for the successful application of probabilistic methods and of data measurement during the reassessment of two existing bridges in Germany. The capabilities of full-probabilistic calculations for the reassessment of existing structures are described and concepts for the calculation of structure-specific partial safety factors are shown.

Towards an Extension of the Model Conditional Processor: Predictive Uncertainty Quantification of Monthly Streamflow via Gaussian Mixture Models and Clusters

This research develops an extension of the Model Conditional Processor (MCP), which merges clusters with Gaussian mixture models to offer an alternative solution to manage heteroscedastic errors. The new method is called the Gaussian mixture clustering post-processor (GMCP). The results of the proposed post-processor were compared to the traditional MCP and MCP using a truncated Normal distribution (MCPt) by applying multiple deterministic and probabilistic verification indices. This research also assesses the GMCP’s capacity to estimate the predictive uncertainty of the monthly streamflow under different climate conditions in the “Second Workshop on Model Parameter Estimation Experiment” (MOPEX) catchments distributed in the SE part of the USA. The results indicate that all three post-processors showed promising results. However, the GMCP post-processor has shown significant potential in generating more reliable, sharp, and accurate monthly streamflow predictions than the MCP and MCPt methods, especially in dry catchments. Moreover, the MCP and MCPt provided similar performances for monthly streamflow and better performances in wet catchments than in dry catchments. The GMCP constitutes a promising solution to handle heteroscedastic errors in monthly streamflow, therefore moving towards a more realistic monthly hydrological prediction to support effective decision-making in planning and managing water resources.

The Use of Statistic Complexity for Security and Performance Analysis in Autonomic Component Ensembles

The paper proposes a new technique for detecting malware threats in autonomic component ensembles. The technique is based on the statistic complexity metrics, which relate objects to random variables and (unlike other complexity measures considering objects as individual symbol strings) are ensemble based. This transforms the classic problem of assessing the complexity of an object into the realm of statistics. The proposed technique requires implementation of the process X (which generates ‘healthy’ flows containing no malware threats) and objects generated by the actual (possible infected) process Y. The component flows files are used as objects of the processes X and Y. The result of the proposed procedure gives us the distribution of probabilities of malware infection among autonomic components. The possibility to use the results obtained to perform quantitative probabilistic verification and analysis of ASEs using the probabilistic model checking tool PRISM is demonstrated.

On the modeling and verification of the spread of fake news, algebraically

Abstract In recent years, one of the major issues concerning the study of online social network phenomena is related to the diffusion of misinformation, commonly known as fake news. In the literature, several mathematical tools are used to investigate both the identifying attributes of fake news and their spreading model. This work concentrates on the latter aspect and follows the lines of research inspired by the analysis of epidemic models. Its main contribution is the definition of a modeling framework based on process algebra for the explicit, formal representation of typical behavioral patterns of agents in social networks, and the use of a probabilistic verification framework based on modal logics and model checking for the quantitative analysis of the fake news spread.

Modelling and verifying BDI agents with bigraphs

• We show BDI agent programming languages can be encoded in Milner's Bigraphs. • We prove the encoding of a superset of AgentSpeak, e.g. CAN language, is faithful. • We provide a verified executable semantics for CAN language programs. • We verify CAN language program properties through model checking. • This is a foundation to enable advanced reasoning, e.g. probabilistic verification. The Belief-Desire-Intention (BDI) architecture is a popular framework for rational agents; existing verification approaches either directly encode simplified (e.g. lacking features like failure recovery) BDI languages into existing verification frameworks (e.g. Promela), or reason about specific BDI language implementations . We take an alternative approach and employ Milner's bigraphs as a modelling framework for a fully featured BDI language, the Conceptual Agent Notation (CAN)—a superset of AgentSpeak featuring declarative goals, concurrency, and failure recovery. We provide an encoding of the syntax and semantics of Can agents, and give a rigorous proof that the encoding is faithful. Verification is based on the use of mainstream software tools including BigraphER, and a small case study verifying several properties of Unmanned Aerial Vehicles (UAVs) illustrates the framework in action. The executable framework is a foundational step that will enable more advanced reasoning such as plan preference, intention priorities and trade-offs, and interactions with an environment under uncertainty.

An Efficient Privacy and Integrity-Preserving Range Query Scheme over Unsorted Data in IoT Two-tiered Wireless Sensor Networks

In most existing privacy-preserving range-query schemes, sorting data items is extremely critical for enabling integrity verification over query results. In two-tiered WSNs, the data is collected by several sensors and send to a central node called the storage node. If data is not sorted, it is hard for the sink to verify query result integrity without sending all data items to the sink. To preserve data privacy on the storage node, the sensors must do sorting, and data is encrypted before it’s sent to the storage nodes since storage nodes are more vulnerable to adversary attacks than the sensor nodes. However, sensors are resource-constrained, yet the sorting process is resource-consuming. The higher the number of data items collected, the larger the number of resources required to sort the elements. In this paper, on top of proposing a scheme that assumes that the data is sorted, we also propose a scheme we call Probabilistic Verification Neighborhood chaining (PvNC) for both sorted and unsorted data that still preserves data and range-query privacy and to a great extent guarantee data integrity. Experimental results show that sensor nodes use less power using our PvNC for unsorted data than the existing schemes that require data to be sorted.

Task-Aware Verifiable RNN-Based Policies for Partially Observable Markov Decision Processes

Partially observable Markov decision processes (POMDPs) are models for sequential decision-making under uncertainty and incomplete information. Machine learning methods typically train recurrent neural networks (RNN) as effective representations of POMDP policies that can efficiently process sequential data. However, it is hard to verify whether the POMDP driven by such RNN-based policies satisfies safety constraints, for instance, given by temporal logic specifications. We propose a novel method that combines techniques from machine learning with the field of formal methods: training an RNN-based policy and then automatically extracting a so-called finite-state controller (FSC) from the RNN. Such FSCs offer a convenient way to verify temporal logic constraints. Implemented on a POMDP, they induce a Markov chain, and probabilistic verification methods can efficiently check whether this induced Markov chain satisfies a temporal logic specification. Using such methods, if the Markov chain does not satisfy the specification, a byproduct of verification is diagnostic information about the states in the POMDP that are critical for the specification. The method exploits this diagnostic information to either adjust the complexity of the extracted FSC or improve the policy by performing focused retraining of the RNN. The method synthesizes policies that satisfy temporal logic specifications for POMDPs with up to millions of states, which are three orders of magnitude larger than comparable approaches.

Probabilistic Verification of Concurrent Autonomous Systems

Probabilistic Verification of Concurrent Autonomous Systems

ProbNV: probabilistic verification of network control planes

ProbNV is a new framework for probabilistic network control plane verification that strikes a balance between generality and scalability. ProbNV is general enough to encode a wide range of features from the most common protocols (eBGP and OSPF) and yet scalable enough to handle challenging properties, such as probabilistic all-failures analysis of medium-sized networks with 100-200 devices. When there are a small, bounded number of failures, networks with up to 500 devices may be verified in seconds. ProbNV operates by translating raw CISCO configurations into a probabilistic and functional programming language designed for network verification. This language comes equipped with a novel type system that characterizes the sort of representation to be used for each data structure: concrete for the usual representation of values; symbolic for a BDD-based representation of sets of values; and multi-value for an MTBDD-based representation of values that depend upon symbolics. Careful use of these varying representations speeds execution of symbolic simulation of network models. The MTBDD-based representations are also used to calculate probabilistic properties of network models once symbolic simulation is complete. We implement the language and evaluate its performance on benchmarks constructed from real network topologies and synthesized routing policies.