Verification Challenges Research Articles

Functional Verification of Large-integers Circuits using a Cosimulation-based Approach

Cryptography and computational algebra designs are complex systems based on modular arithmetic and build on multi-level modules where bit-width is generally larger than 64-bit. Because of their particularity, such designs pose a real challenge for verification, in part because large-integer’s functions are not supported in actual hardware description languages (HDLs), therefore limiting the HDL testbench utility. In another hand, high-level verification approach proved its efficiency in the last decade over HDL testbench technique by raising the latter at a higher abstraction level. In this work, we propose a high-level platform to verify such designs, by leveraging the capabilities of a popular tool (Matlab/Simulink) to meet the requirements of a cycle accurate verification without bit-size restrictions and in multi-level inside the design architecture. The proposed high-level platform is augmented by an assertion-based verification to complete the verification coverage. The platform experimental results of the testcase provided good evidence of its performance and re-usability.

CIVL solutions to verifythis 2016 challenges

The VerifyThis 2016 program verification competition was held in April, 2016. The competition presented three verification challenges: the first dealt with matrix multiplication, including the correctness of Strassen's algorithm; the second dealt with Morris' tree traversal algorithm, and the third required verification of a multithreaded tree barrier. In this paper I present solutions using the CIVL (Concurrency Intermediate Verification Language) verifier. CIVL is able to verify each program within small but non-trivial bounds. The solutions are relatively simple, and are presented in full.

Exploring implementation practices in results-based financing: the case of the verification in Benin

BackgroundResults-based financing (RBF) has been introduced in many countries across Africa and a growing literature is building around the assessment of their impact. These studies are usually quantitative and often silent on the paths and processes through which results are achieved and on the wider health system effects of RBF. To address this gap, our study aims at exploring the implementation of an RBF pilot in Benin, focusing on the verification of results.MethodsThe study is based on action research carried out by authors involved in the pilot as part of the agency supporting the RBF implementation in Benin. While our participant observation and operational collaboration with project’s stakeholders informed the study, the analysis is mostly based on quantitative and qualitative secondary data, collected throughout the project’s implementation and documentation processes. Data include project documents, reports and budgets, RBF data on service outputs and on the outcome of the verification, daily activity timesheets of the technical assistants in the districts, as well as focus groups with Community-based Organizations and informal interviews with technical assistants and district medical officers.ResultsOur analysis focuses on the actual practices of quantitative, qualitative and community verification. Results show that the verification processes are complex, costly and time-consuming, and in practice they end up differing from what designed originally. We explore the consequences of this on the operation of the scheme, on its potential to generate the envisaged change. We find, for example, that the time taken up by verification procedures limits the time available for data analysis and feedback to facility staff, thus limiting the potential to improve service delivery. Verification challenges also result in delays in bonus payment, which delink effort and reward. Additionally, the limited integration of the verification activities of district teams with their routine tasks causes a further verticalization of the health system.ConclusionsOur results highlight the potential disconnect between the theory of change behind RBF and the actual scheme’s implementation. The implications are relevant at methodological level, stressing the importance of analyzing implementation processes to fully understand results, as well as at operational level, pointing to the need to carefully adapt the design of RBF schemes (including verification and other key functions) to the context and to allow room to iteratively modify it during implementation. They also question whether the rationale for thorough and costly verification is justified, or rather adaptations are possible.

Service Provider DevOps

Although there is consensus that software defined networking and network functions virtualization overhaul service provisioning and deployment, the community still lacks a definite answer on how carrier-grade operations praxis needs to evolve. This article presents what lies beyond the first evolutionary steps in network management, identifies the challenges in service verification, observability, and troubleshooting, and explains how to address them using our Service Provider DevOps (SP-DevOps) framework. We compendiously cover the entire process from design goals to tool realization and employ an elastic version of an industry-standard use case to show how on-the-fly verification, software-defined monitoring, and automated troubleshooting of services reduce the cost of fault management actions. We assess SP-DevOps with respect to key attributes of software-defined telecommunication infrastructures both qualitatively and quantitatively, and demonstrate that SP-DevOps paves the way toward carrier-grade operations and management in the network virtualization era.

Score-Aging Calibration for Speaker Verification

The gradual changes that occur in the human voice due to aging create challenges for speaker verification. This study presents an approach to calibrating the output scores of a speaker verification system using the time interval between comparison samples as additional information. Several functions are proposed for the incorporation of this time information, which is viewed as aging information, in a conventional linear score calibration transformation. Experiments are presented on data with short-term aging intervals ranging between 2 months and 3 years, and long-term aging intervals of up to 30 years. The aging calibration proposal is shown to offset the decreased discrimination and calibration performance for both short- and long-term intervals, and to extrapolate well to unseen aging intervals. Relative reductions in $C_{\ell \ell {\rm r}}$ (cost of log-likelihood ratio) of 1–4% and 10–43% are obtained at short- and long-term intervals, respectively. Assuming that a system has knowledge of the time interval between samples under comparison, this approach represents a straightforward means of compensating for the detrimental impact of aging on speaker verification performance.

Biomarker measurements: how far have we come and where are we heading?

Biomarker measurements: how far have we come and where are we heading?

Extreme Event Verification for Probabilistic Downscaling

AbstractExtreme events are important to many studying regional climate impacts but provide a challenge for many “deterministic” downscaling methodologies. The University of Wisconsin Probabilistic Downscaling (UWPD) dataset applies a “probabilistic” approach to downscaling that may be advantageous in a number of situations, including realistic representation of extreme events. The probabilistic approach to downscaling, however, presents some unique challenges for verification, especially when comparing a full probability density function with a single observed value for each day. Furthermore, because of the wide range of specific climatic information needed in climate impacts assessment, any single verification metric will be useful to only a limited set of practitioners. The intent of this study, then, is (i) to identify verification metrics appropriate for probabilistic downscaling of climate data; (ii) to apply, within the UWPD, those metrics to a suite of extreme event statistics that may be of use in climate impacts assessments; and (iii) in applying these metrics, to demonstrate the utility of a probabilistic approach to downscaling climate data, especially for representing extreme events.

Analyzing a wind turbine system: From simulation to formal verification

Many industrial systems are hybrid by nature, most often being made of a discrete controller that senses and regulates the execution of a plant characterized by continuous dynamics. Examples of such systems include wind turbines that convert wind energy into electrical energy. Designing industrial control systems is challenging, due to the mixed nature of requirements (functional, timing, etc.) as well as due to the complexity stemming from the interaction of the controller with the plant. Model-based techniques help in tackling the design challenges, whereas methods such as simulation with tools like MATLAB/Simulink can be employed for analysis. Although practical, these methods alone cannot ensure full predictability, due to the fact that they cannot guarantee system properties for all possible executions of the system model. In order to ensure that the system will behave as expected under any operational circumstance, formal verification and validation procedures need to be added to the actual development process. In this paper, we propose an extension of the iFEST (industrial Framework for Embedded Systems Tools) process and platform for embedded systems design with model-based testing using MaTeLo, and model checking time-dependent requirements with the Uppaal tool, as means of increasing the confidence in the system's behavior. To show the feasibility of the techniques on industrially-sized systems, we analyze a wind turbine industrial prototype model against functional and timing requirements. We capture the execution semantics of the plant and controller components of the wind turbine via logical clocks and constraints expressed in the clock constraint specification language (CCSL) of UML MARTE, after which we construct real-time models amenable to model checking, by mapping the timed behavior (expressed in CCSL) of the real-time components of the wind turbine, onto timed automata. Our work is a first application on an industrial wind turbine system of complementary methods for formal analysis, that is, model-based testing, and model checking a mathematically tractable system abstraction based on data obtained by simulating the system with MATLAB/Simulink. We also discuss relevant modeling and verification challenges encountered during our experiences with the wind turbine system. • Integrated framework for applying model-based tools to designing industrial systems. • Formal analysis of a wind turbine industrial system against various requirements. • A CCSL-based execution semantics of the plant and controller components of the wind turbine.

Spatial Evaluation and Verification of Earthquake Simulators

In this paper, we address the problem of verifying earthquake simulators with observed data. Earthquake simulators are a class of computational simulations which attempt to mirror the topological complexity of fault systems on which earthquakes occur. In addition, the physics of friction and elastic interactions between fault elements are included in these simulations. Simulation parameters are adjusted so that natural earthquake sequences are matched in their scaling properties. Physically based earthquake simulators can generate many thousands of years of simulated seismicity, allowing for a robust capture of the statistical properties of large, damaging earthquakes that have long recurrence time scales. Verification of simulations against current observed earthquake seismicity is necessary, and following past simulator and forecast model verification methods, we approach the challenges in spatial forecast verification to simulators; namely, that simulator outputs are confined to the modeled faults, while observed earthquake epicenters often occur off of known faults. We present two methods for addressing this discrepancy: a simplistic approach whereby observed earthquakes are shifted to the nearest fault element and a smoothing method based on the power laws of the epidemic-type aftershock (ETAS) model, which distributes the seismicity of each simulated earthquake over the entire test region at a decaying rate with epicentral distance. To test these methods, a receiver operating characteristic plot was produced by comparing the rate maps to observed m > 6.0 earthquakes in California since 1980. We found that the nearest-neighbor mapping produced poor forecasts, while the ETAS power-law method produced rate maps that agreed reasonably well with observations.

Warhead verification as inverse problem: Applications of neutron spectrum unfolding from organic-scintillator measurements

Verification of future warhead-dismantlement treaties will require detection of certain warhead attributes without the disclosure of sensitive design information, and this presents an unusual measurement challenge. Neutron spectroscopy—commonly eschewed as an ill-posed inverse problem—may hold special advantages for warhead verification by virtue of its insensitivity to certain neutron-source parameters like plutonium isotopics. In this article, we investigate the usefulness of unfolded neutron spectra obtained from organic-scintillator data for verifying a particular treaty-relevant warhead attribute: the presence of high-explosive and neutron-reflecting materials. Toward this end, several improvements on current unfolding capabilities are demonstrated: deuterated detectors are shown to have superior response-matrix condition to that of standard hydrogen-base scintintillators; a novel data-discretization scheme is proposed which removes important detector nonlinearities; and a technique is described for re-parameterizing the unfolding problem in order to constrain the parameter space of solutions sought, sidestepping the inverse problem altogether. These improvements are demonstrated with trial measurements and verified using accelerator-based time-of-flight calculation of reference spectra. Then, a demonstration is presented in which the elemental compositions of low-Z neutron-attenuating materials are estimated to within 10%. These techniques could have direct application in verifying the presence of high-explosive materials in a neutron-emitting test item, as well as other for treaty verification challenges.

Verified peephole optimizations for CompCert

Transformations over assembly code are common in many compilers. These transformations are also some of the most bug-dense compiler components. Such bugs could be elim- inated by formally verifying the compiler, but state-of-the- art formally verified compilers like CompCert do not sup- port assembly-level program transformations. This paper presents Peek, a framework for expressing, verifying, and running meaning-preserving assembly-level program trans- formations in CompCert. Peek contributes four new com- ponents: a lower level semantics for CompCert x86 syntax, a liveness analysis, a library for expressing and verifying peephole optimizations, and a verified peephole optimiza- tion pass built into CompCert. Each of these is accompanied by a correctness proof in Coq against realistic assumptions about the calling convention and the system memory alloca- tor. Verifying peephole optimizations in Peek requires prov- ing only a set of local properties, which we have proved are sufficient to ensure global transformation correctness. We have proven these local properties for 28 peephole transfor- mations from the literature. We discuss the development of our new assembly semantics, liveness analysis, representa- tion of program transformations, and execution engine; de- scribe the verification challenges of each component; and detail techniques we applied to mitigate the proof burden.

Average Stand Age from Forest Inventory Plots Does Not Describe Historical Fire Regimes in Ponderosa Pine and Mixed-Conifer Forests of Western North America.

Quantifying historical fire regimes provides important information for managing contemporary forests. Historical fire frequency and severity can be estimated using several methods; each method has strengths and weaknesses and presents challenges for interpretation and verification. Recent efforts to quantify the timing of historical high-severity fire events in forests of western North America have assumed that the “stand age” variable from the US Forest Service Forest Inventory and Analysis (FIA) program reflects the timing of historical high-severity (i.e. stand-replacing) fire in ponderosa pine and mixed-conifer forests. To test this assumption, we re-analyze the dataset used in a previous analysis, and compare information from fire history records with information from co-located FIA plots. We demonstrate that 1) the FIA stand age variable does not reflect the large range of individual tree ages in the FIA plots: older trees comprised more than 10% of pre-stand age basal area in 58% of plots analyzed and more than 30% of pre-stand age basal area in 32% of plots, and 2) recruitment events are not necessarily related to high-severity fire occurrence. Because the FIA stand age variable is estimated from a sample of tree ages within the tree size class containing a plurality of canopy trees in the plot, it does not necessarily include the oldest trees, especially in uneven-aged stands. Thus, the FIA stand age variable does not indicate whether the trees in the predominant size class established in response to severe fire, or established during the absence of fire. FIA stand age was not designed to measure the time since a stand-replacing disturbance. Quantification of historical “mixed-severity” fire regimes must be explicit about the spatial scale of high-severity fire effects, which is not possible using FIA stand age data.

Safeguards Considerations for Thorium Fuel Cycles

By around 2025, thorium-based fuel cycles are likely to be deployed internationally. States such as China and India are pursuing research, development, and deployment pathways toward a number of commercial-scale thorium fuel cycles, and they are already building test reactors and the associated fuel cycle infrastructure. In the future, the potential exists for these emerging programs to sell, export, and deploy thorium fuel cycle technology in other states. Without technically adequate international safeguards protocols and measures in place, any future potential clandestine misuse of these fuel cycles could go undetected, compromising the deterrent value of these protocols and measures. The development of safeguards approaches for thorium-based fuel cycles is therefore a matter of some urgency. Yet, the focus of the international safeguards community remains mainly on safeguarding conventional 235U- and 239Pu-based fuel cycles while the safeguards challenges of thorium-uranium fuel cycles remain largely uninvestigated. This raises the following question: Is the International Atomic Energy Agency and international safeguards system ready for thorium fuel cycles? Furthermore, is the safeguards technology of today sufficiently mature to meet the verification challenges posed by thorium-based fuel cycles? In defining these and other related research questions, the objectives of this paper are to identify key safeguards considerations for thorium-based fuel cycles and to call for an early dialogue between the international safeguards and the nuclear fuel cycle communities to prepare for the potential safeguards challenges associated with these fuel cycles. In this paper, it is concluded that directed research and development programs are required to meet the identified safeguards challenges and to take timely action in preparation for the international deployment of thorium fuel cycles.

Summary of the 2014 Sandia Verification and Validation Challenge Workshop

A discussion of the five responses to the 2014 Sandia Verification and Validation (V&V) Challenge Problem, presented within this special issue, is provided hereafter. Overviews of the challenge problem workshop, workshop participants, and the problem statement are also included. Brief summations of teams' responses to the challenge problem are provided. Issues that arose throughout the responses that are deemed applicable to the general verification, validation, and uncertainty quantification (VVUQ) community are the main focal point of this paper. The discussion is oriented and organized into big picture comparison of data and model usage, VVUQ activities, and differentiating conceptual themes behind the teams' VVUQ strategies. Significant differences are noted in the teams' approaches toward all VVUQ activities, and those deemed most relevant are discussed. Beyond the specific details of VVUQ implementations, thematic concepts are found to create differences among the approaches; some of the major themes are discussed. Finally, an encapsulation of the key contributions, the lessons learned, and advice for the future are presented.

The 2014 Sandia Verification and Validation Challenge: Problem Statement

This paper describes the challenge problem associated with the 2014 Sandia Verification and Validation (V&V) Challenge Workshop. The problem was developed to highlight core issues in V&V of engineering models. It is intended as an analog to projects currently underway at the Sandia National Laboratories—in other words, a realistic case study in applying V&V methods and integrating information from experimental data and simulations to support decisions. The problem statement includes the data, model, and directions for participants in the challenge. In addition, the workings of the provided code and the “truth model” used to create the data are revealed. The code, data, and truth model are available in this paper.

Introduction: The 2014 Sandia Verification and Validation Challenge Workshop

Introduction: The 2014 Sandia Verification and Validation Challenge Workshop

Sandia Verification and Validation Challenge Problem: A PCMM-Based Approach to Assessing Prediction Credibility

The process of establishing credibility in computational model predictions via verification and validation (V&V) encompasses a wide range of activities. Those activities are focused on collecting evidence that the model is adequate for the intended application and that the errors and uncertainties are quantified. In this work, we use the predictive capability maturity model (PCMM) as an organizing framework for evidence collection activities and summarizing our credibility assessment. We discuss our approaches to sensitivity analysis, model calibration, model validation, and uncertainty quantification and how they support our assessments in the solution verification, model validation, and uncertainty quantification elements of the PCMM. For completeness, we also include some limited assessment discussion for the remaining PCMM elements. Because the computational cost of performing V&V and the ensuing predictive calculations is substantial, we include discussion of our approach to addressing computational resource considerations, primarily through the use of response surface surrogates and multiple mesh fidelities.

Verifying Custom Synchronization Constructs Using Higher-Order Separation Logic

Synchronization constructs lie at the heart of any reliable concurrent program. Many such constructs are standard (e.g., locks, queues, stacks, and hash-tables). However, many concurrent applications require custom synchronization constructs with special-purpose behavior. These constructs present a significant challenge for verification. Like standard constructs, they rely on subtle racy behavior, but unlike standard constructs, they may not have well-understood abstract interfaces. As they are custom built, such constructs are also far more likely to be unreliable. This article examines the formal specification and verification of custom synchronization constructs. Our target is a library of channels used in automated parallelization to enforce sequential behavior between program statements. Our high-level specification captures the conditions necessary for correct execution; these conditions reflect program dependencies necessary to ensure sequential behavior. We connect the high-level specification with the low-level library implementation to prove that a client’s requirements are satisfied. Significantly, we can reason about program and library correctness without breaking abstraction boundaries. To achieve this, we use a program logic called iCAP (impredicative Concurrent Abstract Predicates) based on separation logic. iCAP supports both high-level abstraction and low-level reasoning about races. We use this to show that our high-level channel specification abstracts three different, increasingly complex low-level implementations of the library. iCAP’s support for higher-order reasoning lets us prove that sequential dependencies are respected, while iCAP’s next-generation semantic model lets us avoid ugly problems with cyclic dependencies.

Community-Reviewed Biological Network Models for Toxicology and Drug Discovery Applications.

Biological network models offer a framework for understanding disease by describing the relationships between the mechanisms involved in the regulation of biological processes. Crowdsourcing can efficiently gather feedback from a wide audience with varying expertise. In the Network Verification Challenge, scientists verified and enhanced a set of 46 biological networks relevant to lung and chronic obstructive pulmonary disease. The networks were built using Biological Expression Language and contain detailed information for each node and edge, including supporting evidence from the literature. Network scoring of public transcriptomics data inferred perturbation of a subset of mechanisms and networks that matched the measured outcomes. These results, based on a computable network approach, can be used to identify novel mechanisms activated in disease, quantitatively compare different treatments and time points, and allow for assessment of data with low signal. These networks are periodically verified by the crowd to maintain an up-to-date suite of networks for toxicology and drug discovery applications.

A position-aware Merkle tree for dynamic cloud data integrity verification

In the cloud storage framework, once clients remotely store their data on cloud storage providers, they will lose the physical control over their outsourced data. The risk of unauthorized access to the data increases dramatically. One of the most serious problems in cloud storage is to ensure the correctness of the outsourced data. Specifically, we need to protect these data from unauthorized operations; we also need to detect and recover users' data after unexpected changes. In this paper, we propose a publicly verifiable scheme to protect the integrity of cloud data and support dynamic maintenance, which is based on a position-aware Merkle tree. We adopt a 3-tuple to define the node of the new Merkle tree, which records the position of the corresponding node, so that users can verify the consistency of the challenge-response blocks by computing the root value directly without retrieving the whole Merkle tree. In our scheme, the storage complexity at the client side is O(1); the computation complexity at the client side is $$O(\log n)$$O(logn); the computation cost at the server side is $$O(\log n)$$O(logn) and the communication overhead is $$O(\log n)$$O(logn). Our method supports unlimited verification challenges as well.