Verification Framework Research Articles

Reproducibility of density functional approximations: How new functionals should be reported.

Density functional theory is the workhorse of chemistry and materials science, and novel density functional approximations are published every year. To become available in program packages, the novel density functional approximations (DFAs) need to be (re)implemented. However, according to our experience as developers of Libxc [Lehtola et al., SoftwareX 7, 1 (2018)], a constant problem in this task is verification due to the lack of reliable reference data. As we discuss in this work, this lack has led to several non-equivalent implementations of functionals such as Becke-Perdew 1986, Perdew-Wang 1991, Perdew-Burke-Ernzerhof, and Becke's three-parameter hybrid functional with Lee-Yang-Parr correlation across various program packages, yielding different total energies. Through careful verification, we have also found many issues with incorrect functional forms in recent DFAs. The goal of this work is to ensure the reproducibility of DFAs. DFAs must be verifiable in order to prevent the reappearance of the above-mentioned errors and incompatibilities. A common framework for verification and testing is, therefore, needed. We suggest several ways in which reference energies can be produced with free and open source software, either with non-self-consistent calculations with tabulated atomic densities or via self-consistent calculations with various program packages. The employed numerical parameters-especially the quadrature grid-need to be converged to guarantee a ≲0.1 μEh precision in the total energy, which is nowadays routinely achievable in fully numerical calculations. Moreover, as such sub-μEh level agreement can only be achieved when fully equivalent implementations of the DFA are used, the source code of the reference implementation should also be made available in any publication describing a new DFA.

FrameD: Framework for DNA-based Data Storage Design, Verification, and Validation.

DNA-based data storage is a quickly growing field that hopes to harness the massive theoretical information density of DNA molecules to produce a competitive next-generation storage medium suitable for archival data. In recent years, many DNA-based storage system designs have been proposed. Given that no common infrastructure exists for simulating these storage systems, comparing many different designs along with many different error models is increasingly difficult. To address this challenge we introduce FrameD, a simulation infrastructure for DNA storage systems that leverages the underlying modularity of DNA storage system designs to provide a framework to express different designs while being able to reuse common components. We demonstrate the utility of FrameD and the need for a common simulation platform using a case study. Our case study compares designs that utilize strand copies differently, some that align strand copies using Multiple Sequence Alignment (MSA) algorithms and others that do not. We found that the choice to include MSA in the pipeline is dependent on the error rate and the type of errors being injected and is not always beneficial. In addition to supporting a wide range of designs, FrameD provides the user with transparent parallelism to deal with a large number of reads from sequencing and the need for many fault injection iterations. We believe that FrameD fills a void in the tools publicly available to the DNA storage community by providing a modular and extensible framework with support for massive parallelism. As a result, it will help accelerate the design process of future DNA-based storage systems. The source code for FrameD along with the data generated during the demonstration of FrameD is available in a public Github repository at https://github.com/dna-storage/framed (10.5281/zenodo.7757762).

Assessing network-based earthquake early warning systems in low-seismicity areas

Earthquake early warning (EEW) technology, designed to alert the public of earthquake risks after initial P-wave detection but before the onset of strong tremors, has developed rapidly. Methodologies from various fields are combined in EEW systems to estimate earthquake locations, magnitudes, and expected intensities based on the initial P-wave data. These systems operate automatically because prompt responses are required. However, as no common evaluation framework for EEW system verification exists, potentially divergent evaluations from reviewers or countries could ensue. Moreover, evaluating EEW systems is more complicated when the target area does not experience frequent earthquakes. We aimed to establish a guidance review process for low-seismicity areas to ensure reliable and stable integrated EEW system operation. We incorporated management aspects through actual system operator surveillance and designed an EEW assessment process based on feedback from our surveys. Using this approach, we created a comprehensive and well-informed evaluation process that considers the diverse perspectives of experts involved in EEWs. Our proposed assessment method allows for a uniform and consistent evaluation process, regardless of changes in the methods or technologies used by EEW systems. The method aims to guide EEW system assessments in low-seismicity areas.

Effects analysis and probability forecast (EAPF) of real-time management on urban flooding: A novel bidirectional verification framework

Government departments usually prepare and implement contingency plans to address frequent urban flooding caused by short-term heavy rainfall. Previous studies focused on the evaluation of the static impact of the policies on urban floods, while there is a lack of research on the effect of off-design conditions, real-time feedback and treatments of the flood events on urban flood mitigation, which is detrimental to the optimization of management strategies of the cities. To quantify the effects of real-time management on flood mitigation in Fuzhou City, China, this study proposed a framework (EAPF) for evaluation and risk prediction. First, we collected data on the locations, rainfall intensity, inundation time, and the triggers of the waterlogging events from 2017 to 2021. Second, based on the vigilance analyses, a structural equation model (SEM) was constructed to quantitatively evaluate the mitigation effects of management on waterlogging. Finally, a probability prediction model of dynamic drainage capacity was proposed for flood simulation caused by the rainwater grate blockage. The results indicate that the environmental factors were the decisive triggers affecting the severity of waterlogging, and increasing the frequency of management events effectively reduced the probability of blocking. The correlation between the number of management events and blocking flood events was −0.42, while a decrease in vigilance increased the possibility of flooding caused by overdue treatment. The proposed hydrological waterlogging model, which considered blockages, exhibited a Nash–Sutcliffe efficiency (NSE) coefficient exceeding 0.9 under deterministic conditions. The probability prediction model verified the mitigating effect of management on the blockages and urban flooding, and its results were consistent with those of the SEM. Our study contributes to improving the reliability of waterlogging prediction and optimizing the management flow in the developing cities.

Full-Life Cycle Intent-Driven Network Verification: Challenges and Approaches

With the human friendly declarative intent policy expression, intent-driven network can make network management and configuration autonomous without human intervention. However, the availability and dependability of these refined policies from the expressed intents should be well ensured by full-life cycle verification. Moreover, intent-driven network verification is still in its initial stage, and there is a lack of full-life cycle end-to-end verification framework. As a result, in this article, we present and review existing verification techniques, and classify them according to objective, purpose, and feedback. Furthermore, we describe intent verification as a technology that provides assurance during the intent form conversion process and propose a novel full-life cycle verification framework that expands on the concept of traditional network verification. Finally, we verify the feasibility and validity of the presented verification framework in the case of an access control policy for different network functions with multi conflict intents.

Nonparametric Multivariate Probability Density Forecast in Smart Grids With Deep Learning

This paper proposes a nonparametric multivariate density forecast model based on deep learning. It not only offers the whole marginal distribution of each random variable in forecasting targets, but also reveals the future correlation between them. Differing from existing multivariate density forecast models, the proposed method requires no a priori hypotheses on the forecasted joint probability distribution of forecasting targets. In addition, based on the universal approximation capability of neural networks, the real joint cumulative distribution functions of forecasting targets are well-approximated by a special positive-weighted deep neural network in the proposed method. Numerical tests from different scenarios were implemented under a comprehensive verification framework for evaluation, including the very short-term forecast of the wind speed, wind power, and the day-ahead forecast of the aggregated electricity load. Testing results corroborate the superiority of the proposed method over current multivariate density forecast models considering the accordance with reality, prediction interval width, and correlations between different random variables.

Biases and Skill of Four Two-Moment Bulk Microphysics Schemes in Convection-Allowing Forecasts for the 2018 Hazardous Weather Testbed Spring Forecasting Experiment Period

Abstract A proof-of-concept systematic evaluation of convective hazards is applied to short-term (1–6 h) forecasts using the Morrison, National Severe Storms Laboratory (NSSL), Predicted Particle Properties (P3), and Thompson two-moment microphysics schemes for the 2018 NOAA Hazardous Weather Testbed Spring Forecasting Experiment (HWT SFE) period (hereafter “MORR,” “NSSL,” “P3,” and “THOM” experiments, respectively). Four convective line cases are highlighted to elaborate on relative experiment biases/skill. Composite reflectivity and 1-h accumulated precipitation are examined to determine storm coverage/precipitation biases/skill utilizing point-based verification with a neighborhood. Simulated 1–6-km updraft helicity and observed 3–6-km azimuthal shear and MESH are examined to consider simulated rotation and hail core prediction with object-based scores. Over the full season, MORR displays little overall storm coverage bias relative to NSSL, P3, and THOM underprediction. The equitable threat score (ETS) and fractions skill score (FSS) of P3 are lower than the other experiments. P3 and THOM underpredict convective regions with intense reflectivity relative to MORR and NSSL overprediction. All experiments underpredict precipitation amounts. P3 light precipitation FSS is lower than other experiments. Rotation object verification exhibits sensitivity to microphysics experiments, as microphysics has an indirect influence on storm dynamics. While P3 has the largest hail object underprediction, all experiments grossly overpredict the number of hail objects in convective line cases despite forecast objects defined with the same product (MESH) and threshold as observations. The importance of microphysics ice parameterization and ongoing scheme updates highlight the need to apply this verification framework to optimal/updated schemes before optimizing ensemble design.

Trustworthy Runtime Verification via Bisimulation (Experience Report)

When runtime verification is used to monitor safety-critical systems, it is essential that monitoring code behaves correctly. The Copilot runtime verification framework pursues this goal by automatically generating C monitor programs from a high-level DSL embedded in Haskell. In safety-critical domains, every piece of deployed code must be accompanied by an assurance argument that is convincing to human auditors. However, it is difficult for auditors to determine with confidence that a compiled monitor cannot crash and implements the behavior required by the Copilot semantics. In this paper we describe CopilotVerifier, which runs alongside the Copilot compiler, generating a proof of correctness for the compiled output. The proof establishes that a given Copilot monitor and its compiled form produce equivalent outputs on equivalent inputs, and that they either crash in identical circumstances or cannot crash. The proof takes the form of a bisimulation broken down into a set of verification conditions. We leverage two pieces of SMT-backed technology: the Crucible symbolic execution library for LLVM and the What4 solver interface library. Our results demonstrate that dramatically increased compiler assurance can be achieved at moderate cost by building on existing tools. This paves the way to our ultimate goal of generating formal assurance arguments that are convincing to human auditors.

Different approaches for stress index calculation in elbows of a fusion plant piping system

In the framework of the final verification for the mechanical assessment of a nuclear piping system, critical cross sections, which turned out to belong to an elbow, have been first selected: a study has been subsequently developed, focused on the evaluation of stress index and safety margins for primary stress, starting from what is set out by ASME B31.3 for Process Piping and by RCC-MRx design Code for mechanical components of nuclear installations. The aim was to move to the critical evaluation of the application of the approach “by rules” when finite elements analyses provide both results in terms of internal forces to obtain the nominal stress, and the detailed numerical evaluation of the local stress state and stress concentration. The results show the differences in stress computation that can be obtained by adopting the finite element modeling or by straight application of codes rules and allows to identify the different levels of conservatism and to choose the suitable criteria to be used in an elbow stress assessment.

Mapping historical forest biomass for stock-change assessments at parcel to landscape scales

Understanding historical forest dynamics, specifically changes in forest biomass and carbon stocks, has become critical for assessing current forest climate benefits and projecting future benefits under various policy, regulatory, and stewardship scenarios. Carbon accounting frameworks based exclusively on national forest inventories are limited to broad-scale estimates, but model-based approaches that combine these inventories with remotely sensed data can yield contiguous fine-resolution maps of forest biomass and carbon stocks across landscapes over time. Here we describe a fundamental step in building a map-based stock-change framework: mapping historical forest biomass at fine temporal and spatial resolution (annual, 30 m) across all of New York State (USA) from 1990 to 2019, using freely available data and open-source tools. Using Landsat imagery, US Forest Service Forest Inventory and Analysis (FIA) data, and off-the-shelf LiDAR collections we developed three modeling approaches for mapping historical forest aboveground biomass (AGB): training on FIA plot-level AGB estimates (direct), training on LiDAR-derived AGB maps (indirect), and an ensemble averaging predictions from the direct and indirect models. Model prediction surfaces (maps) were tested against FIA estimates at multiple scales. All three approaches produced viable outputs, yet tradeoffs were evident in terms of model complexity, map accuracy, saturation, and fine-scale pattern representation. The resulting map products can help identify where, when, and how forest carbon stocks are changing as a result of both anthropogenic and natural drivers alike. These products can thus serve as inputs to a wide range of applications including stock-change assessments, monitoring reporting and verification frameworks, and prioritizing parcels for protection or enrollment in improved management programs.

Hybrid Optimized Verification Methodology using Deep Reinforcement Neural Network

Universal Verification Methodology (UVM) caters to an essential role in verifying the different categories of circuits ranging from small-scale chips to complex system-on-chip architectures. Constrained random simulations are an indispensable part of UVM and are often used for design verification. However, the effort and time spent manually updating and analyzing the design input constraints result in high time complexity, which typically impacts the coverage goal and fault verification ratio. To overcome this problem, this paper proposes a novel hybrid optimized verification framework that combines Reinforcement Learning (RL) and Deep Neural Networks (DNN) for automatically optimizing the input constraints, accelerating faster verification with a high coverage ratio. The proposed algorithm uses reinforcement learning to generate all possible vector sequences needed for testing the target devices and corresponding outputs of the target devices and potential design errors. Furthermore, the framework intends to use high-speed deep-feedforward neural networks to automate and optimize the constraints during runtime. The proposed framework was developed using Python interfaced with the TCL environment. Extensive experimentation was carried out using several circuits, including multi-core designs, and performance parameters such as coverage accuracy, speed, and computational complexity were calculated and analyzed. The experiment demonstrated the proposed framework remarkable results, showing its superior performance in faster coverage and fewer misclassification errors. Furthermore, the proposed framework is compared with existing verification frameworks and other classical learning models. Good results demonstrate that the proposed framework increases the 4.5x speed for verifying multi-core designs and the 99% accuracy of detection and coverage.

A Highly Compatible Verification Framework with Minimal Upgrades to Secure an Existing Edge Network

Edge networks are providing services for an increasing number of companies, and they can be used for communication between edge devices and edge gateways. However, the performance of edge devices varies greatly, and it is not easy to upgrade low-performance edge devices. Therefore, cyber attackers can use the vulnerability of edge devices to implement advanced persistent threat attacks. This article proposes a network verification framework for edge networks that can minimize the upgrades needed to strengthen edge network security. First, the communication parties use the data transmitted by the given edge network. Our method uses our proposed PacketVerifier to attach verification information to the packet after it is sent and to verify and restore the packet before it reaches the receiver. Second, due to the performance requirements of edge networks, we design a new data processing structure, namely, a sliding window double ring, to improve the performance of strict sequential protocols in parallel validation. Finally, experimental simulations show that our parallel processing algorithm has good performance in terms of network bandwidth compared with two existing packet processing algorithms. Furthermore, the proposed packet with verification information is compatible with the existing network topology, which helps PacketVerifier establish trustworthy transmission in a zero-trust environment.

Runtime Verification for Programmable Switches

We introduce a runtime verification framework for programmable switches that complements static analysis. To evaluate our approach, we design and develop, a runtime verification system that automatically detects, localizes, and patches software bugs in P4 programs. Bugs are reported via a violation of pre-specified expected behavior that is captured by . is based on machine learning-guided fuzzing that tests P4 switch non-intrusively, i.e., without modifying the P4 program for detecting runtime bugs. This enables an automated and real-time localization and patching of bugs. We used a prototype to detect and patch existing bugs in various publicly available P4 application programs deployed on two different switch platforms, namely, behavioral model (bmv2) and Tofino. Our evaluation shows that significantly outperforms bug detection baselines while generating fewer packets and patches bugs in large P4 programs, e.g., switch.p4 without triggering any regressions.

How to assess climate change impact models: uncertainty analysis of streamflow statistics via approximate Bayesian computation (ABC)

ABSTRACT Climate change impact models (CCIMs) suffer from inherent bias, uncertainty, and asynchronous observations in the baseline period. To overcome these challenges, this study introduces a methodology to assess CCIMs in the baseline period using the uncertainty analysis of streamflow statistics via the approximate Bayesian computation (ABC) post-processor, which infers the residual error model parameters based on summary statistics (signatures). As an illustrative case study, we analyzed the climate change projections of the fifth assessment report of the United Nations intergovernmental panel on climate change (AR5 - IPCC) of the monthly streamflow in the upper Oria catchment (Spain) with deterministic and probabilistic verification frameworks to assess the ABC post-processor outputs. In addition, the ABC post-processor is evaluated against the ensemble (reference method). The results show that the ABC post-processor outperformed the ensemble method in all verification metrics, and the ensemble method has reasonable reliability but exhibited poor sharpness. We suggest that the ensemble method should be complemented with the ABC post-processor for climate change impact studies.

SSProve: A Foundational Framework for Modular Cryptographic Proofs in Coq

State-separating proofs (SSP) is a recent methodology for structuring game-based cryptographic proofs in a modular way, by using algebraic laws to exploit the modular structure of composed protocols. While promising, this methodology was previously not fully formalized and came with little tool support. We address this by introducing SSProve, the first general verification framework for machine-checked state-separating proofs. SSProve combines high-level modular proofs about composed protocols, as proposed in SSP, with a probabilistic relational program logic for formalizing the lower-level details, which together enable constructing machine-checked cryptographic proofs in the Coq proof assistant. Moreover, SSProve is itself fully formalized in Coq, including the algebraic laws of SSP, the soundness of the program logic, and the connection between these two verification styles. To illustrate SSProve, we use it to mechanize the simple security proofs of ElGamal and pseudo-random-function–based encryption. We also validate the SSProve approach by conducting two more substantial case studies: First, we mechanize an SSP security proof of the key encapsulation mechanism–data encryption mechanism (KEM-DEM) public key encryption scheme, which led to the discovery of an error in the original paper proof that has since been fixed. Second, we use SSProve to formally prove security of the sigma-protocol zero-knowledge construction, and we moreover construct a commitment scheme from a sigma-protocol to compare with a similar development in CryptHOL. We instantiate the security proof for sigma-protocols to give concrete security bounds for Schnorr’s sigma-protocol.

A Novel Finger Vein Verification Framework Based on Siamese Network and Gabor Residual Block

A Novel Finger Vein Verification Framework Based on Siamese Network and Gabor Residual Block

CRBF: Cross-Referencing Bloom-Filter-Based Data Integrity Verification Framework for Object-Based Big Data Transfer Systems

Various components are involved in the end-to-end path of data transfer. Protecting data integrity from failures in these intermediate components is a key feature of big data transfer tools. Although most of these components provide some degree of data integrity, they are either too expensive or inefficient in recovering corrupted data. This problem highlights the need for application-level end-to-end integrity verification during data transfer. However, the computational, memory, and storage overhead of big data transfer tools can be a significant bottleneck for ensuring data integrity due to the large size of the data. This paper proposes a novel framework for data integrity verification in big data transfer systems using a cross-referencing Bloom filter. This framework has three advantages over state-of-the-art data integrity techniques: lower computation and memory overhead and zero false-positive errors for a limited number of elements. This study evaluates the computation, memory, recovery time, and false-positive overhead for the proposed framework and compares them with state-of-the-art solutions. The evaluation results indicate that the proposed framework is efficient in detecting and recovering from integrity errors while eliminating false positives in the Bloom filter data structure. In addition, we observe negligible computation, memory, and recovery overheads for all workloads.

Correct-by-Design Interacting Smart Contracts and a Systematic Approach for Verifying ERC20 and ERC721 Contracts With VeriSolid

Blockchain-based smart contracts enable the creation of decentralized applications, which often handle assets of considerable value. While the underlying platforms guarantee the correctness of smart-contract execution, they cannot ensure that the code of a contract is correct. Today, as evidenced by a number of recent security breaches, developers still have a hard time making contracts that work properly.Even though these incidents often exploit contract interaction, prior work on smart-contract verification, vulnerability discovery, and secure development typically considers only individual contracts in isolation. To address this gap, we introduce the <small>VeriSolid</small> framework for the formal verification of contracts that are specified using a abstract state machine based model with rigorous operational semantics. Our model-based approach allows developers to reason about and verify the behavior of a set of interacting contracts at a high level of abstraction. <small>VeriSolid</small> allows the generation of Solidity code that is functionally and behaviorally equivalent to verified models, which enables the creation of correct-by-design smart contracts. We additionally introduce a graphical notation (called <i>deployment diagrams</i>) for specifying possible interactions between contract types. Based on this notation, we present a framework for the automated verification, generation, and deployment of contracts that conform to a deployment diagram. To demonstrate the applicability of <small>VeriSolid</small>, we translate existing Ethereum Improvement Proposal (EIP) specifications to temporal properties for two of the most popular contract interfaces: ERC20 and ERC721. We also show you how to write code for the ERC20 and ERC721 interfaces in a way that is safe, and we do this by using <small>VeriSolid</small>. We evaluate our framework on 726 contracts that are currently deployed on the Ethereum blockchain, which include 267 ERC20 and 459 ERC721 contracts. Our experiments indicate that 18&#x0025; of ERC20 contracts and 4&#x0025; of ERC721 contracts fail to satisfy the EIP specifications.

Verification and Validation Test Framework Using a Model‐Based Systems Engineering Approach

AbstractThis paper describes a test framework for verification and validation (V&amp;V) planning and execution using a model‐based system engineering (MBSE) approach that is suitable for large‐scale cyber‐physical systems. The test framework (TF) is defined using the systems modeling language (SysML) and an MBSE tool and describes and links to external test plans, test procedures, and other relevant test documents. This test framework approach includes the following considerations: level of abstraction of systems for testing, multiple types of testing across the development lifecycle, design vs. system V&amp;V approaches, verification vs. validation, and the use of test patterns, instances, inheritance, and libraries. The libraries contain distinct types of tests to maximize model element reuse. An example is provided that illustrates the application of this V&amp;V Test Framework (V&amp;VTF) to a lubrication system. This method demonstrates that the proposed V&amp;VTF is desirable as part of Systems Engineering metamorphosis to support an organization's digital transformation.

Reachability Analysis of Neural Network Control Systems

Neural network controllers (NNCs) have shown great promise in autonomous and cyber-physical systems. Despite the various verification approaches for neural networks, the safety analysis of NNCs remains an open problem. Existing verification approaches for neural network control systems (NNCSs) either can only work on a limited type of activation functions, or result in non-trivial over-approximation errors with time evolving. This paper proposes a verification framework for NNCS based on Lipschitzian optimisation, called DeepNNC. We first prove the Lipschitz continuity of closed-loop NNCSs by unrolling and eliminating the loops. We then reveal the working principles of applying Lipschitzian optimisation on NNCS verification and illustrate it by verifying an adaptive cruise control model. Compared to state-of-the-art verification approaches, DeepNNC shows superior performance in terms of efficiency and accuracy over a wide range of NNCs. We also provide a case study to demonstrate the capability of DeepNNC to handle a real-world, practical, and complex system. Our tool DeepNNC is available at https://github.com/TrustAI/DeepNNC.