Java PathFinder Research Articles

Dynamic Partial Order Reduction for Checking Correctness against Transaction Isolation Levels

Modern applications, such as social networking systems and e-commerce platforms are centered around using large-scale databases for storing and retrieving data. Accesses to the database are typically enclosed in transactions that allow computations on shared data to be isolated from other concurrent computations and resilient to failures. Modern databases trade isolation for performance. The weaker the isolation level is, the more behaviors a database is allowed to exhibit and it is up to the developer to ensure that their application can tolerate those behaviors.In this work, we propose stateless model checking algorithms for studying correctness of such applications that rely on dynamic partial order reduction. These algorithms work for a number of widely-used weak isolation levels, including Read Committed, Causal Consistency, Snapshot Isolation and Serializability. We show that they are complete, sound and optimal, and run with polynomial memory consumption in all cases. We report on an implementation of these algorithms in the context of Java Pathfinder applied to a number of challenging applications drawn from the literature of distributed systems and databases.

Using Theia Trace Viewer to Visualize JPF Traces

Java Pathfinder (JPF) is currently the most feature-rich platform for an in-depth analysis of programs that run on the Java Virtual Machine (JVM). However, using JPF and interpreting its results is a challenge. Dependencies on specific Java versions inhibit the adoption of JPF, and its textual output makes it hard to understand its analysis results. To this end, we present the Theia Trace Viewer (TTV) for JPF traces, which features a web-based graphical user interface. It does not only make it easier to get an overview of JPF traces, but it can visualize traces without requiring an installation on the client side, thus making JPF more accessible at the technical level as well. We evaluated TTV with several groups of users, showing the potential of our approach.

The Java Pathfinder Workshop 2022

Java Pathfinder (JPF) was originally developed as an explicit state software model checker and subsequently evolved into an extensible Java bytecode analysis framework that has been successfully used to implement techniques such as symbolic and concolic execution, compositional verification, parallel execution, incremental program analysis, and many more. Apart from its original domain of concurrent Java programs, JPF now supports the verification of new domains such as UMLs, numeric programs, graphical user interfaces, and Android applications.

Checking Just Pairs of Threads for Efficient and Scalable Incremental Verification of Multithreaded Programs

Many techniques of automated verification target multithreaded programs, because subtle interactions between threads may trigger concurrency errors such as deadlocks and data races. However, techniques and tools involving systematic exploration of the whole space of possible thread interleavings do not scale to large software systems, despite various clever algorithmic optimizations. A viable approach is to use incremental verification techniques that, in each run, focus just on the recently modified code and the relatively small number of affected execution traces, and therefore can provide results (bug reports) very quickly. In this paper we present a new algorithm for incremental verification of multithreaded programs based on the pairwise approach, whose key idea is systematic exploration of all possible thread interleavings just for specific relevant pairs of threads. We implemented the algorithm with Java Pathfinder as the backend verification tool, and evaluated it on several multithreaded Java programs. Results show that our incremental algorithm (1) can find errors very fast, (2) greatly reduces time needed for complete safety verification, and (3) it can find the same errors as full verification of the whole state space.

Jpf-logic

We present jpf-logic, an extension of the model checker Java PathFinder (JPF). Our extension jpf-logic provides a framework to check properties expressed in temporal logics such as linear temporal logic (LTL) and computation tree logic (CTL). To support a logic in our framework, we (1) implement a parser for the logic, (2) develop a hierarchy of classes that represent the abstract syntax of the logic and implement a transformation from parse trees of formulas to the corresponding abstract syntax trees, and (3) implement a model checking algorithm that takes as input an abstract syntax tree of a formula and a partial transition system. The latter represents a model of the Java application. All three components have been implemented for CTL. The first two have been implemented for LTL.

Bug detection in Java code: An extensive evaluation of static analysis tools using Juliet Test Suites

AbstractPrevious studies have demonstrated the usefulness of employing automated static analysis tools (ASAT) and techniques to detect security bugs in software systems. However, these studies are usually focused on analyzing the effectiveness of the tools using open‐source tools based on C/C++ source code. The choice for making an appropriate decision on the most suitable tool for bug detection in Java code software remains a relatively unexplored domain. To address this deficiency, this study empirically evaluates eight widely used ASATs, namely, Findbug, PMD, YASCA, LAPSE+, JLint, Bandera, ESC/Java, and Java Pathfinder using the Juliet Test Suite (Test Suite v1.2). Additionally, we assessed the performance of the detection capabilities for the aforementioned bug detection tools using robust performance measures such as precision, recall, Youden index, and the OWASP web benchmark evaluation (WBE). The experimental results show that the tools obtain precision values ranging from 83% to 90.7% based on the studied datasets. Specifically, the Java Pathfinder achieves the best precision score of 90.7%, followed by YASCA and Bandera with a precision score of 88.7% and 83%, respectively. Similarly, Bandera, ESC/Java, and Java Pathfinder obtain a Youden index of 0.8, which indicates the effectiveness of the tools in detecting security bugs in Java source code.

Test Case Generation using Symbolic Execution

Testing is a well-known technique for identifying errors in software programs. Testing can be done in two ways: Static analysis and Dynamic analysis. Symbolic execution plays a vital role in static analysis for test case generation and to find the unreachable path with minimum test cases. Unreachable path is a part of a program which can never be executed i.e., the symbolic execution doesn’t continue for that path and the current execution stops there. It generates a test suite for loop-free programs that is achieved by path coverage. In the best case program loops implies increase in the number of paths exponentially and in the worst case the program will not terminate. The functions of symbolic execution are test input generation, unreachable path detection, finding bugs in software programs, debugging. In this paper, we focus on dead code detection and test input generation using symbolic execution. Our execution for Java programs uses Java Path Finder (JPF) model tester. Our analysis shows that the symbolic execution method can be used to reduce symbolic execution time and to find out the unreachable path with less number of test cases.

SPOT: Testing Stream Processing Programs with Symbolic Execution and Stream Synthesizing

Adoption of distributed stream processing (DSP) systems such as Apache Flink in real-time big data processing is increasing. However, DSP programs are prone to be buggy, especially when one programmer neglects some DSP features (e.g., source data reordering), which motivates development of approaches for testing and verification. In this paper, we focus on the test data generation problem for DSP programs. Currently, there is a lack of an approach that generates test data for DSP programs with both high path coverage and covering different stream reordering situations. We present a novel solution, SPOT (i.e., Stream Processing Program Test), to achieve these two goals simultaneously. At first, SPOT generates a set of individual test data representing each path of one DSP program through symbolic execution. Then, SPOT composes these independent data into various time series data (a.k.a, stream) in diverse reordering. Finally, we can perform a test by feeding the DSP program with these streams continuously. To automatically support symbolic analysis, we also developed JPF-Flink, a JPF (i.e., Java Pathfinder) extension to coordinate the execution of Flink programs. We present four case studies to illustrate that: (1) SPOT can support symbolic analysis for the commonly used DSP operators; (2) test data generated by SPOT can more efficiently achieve high JDU (i.e., Joint Dataflow and UDF) path coverage than two recent DSP testing approaches; (3) test data generated by SPOT can more easily trigger software failure when comparing with those two DSP testing approaches; and (4) the data randomly generated by those two test techniques are highly skewed in terms of stream reordering, which is measured by the entropy metric. In comparison, it is even for test data from SPOT.

Documentation‐based functional constraint generation for library methods

SummaryAlthough software libraries promote code reuse and facilitate software development, they increase the complexity of programme analysis tasks. To effectively analyse programmes built on top of software libraries, it is essential to have specifications for the library methods that can be easily processed by analysis tools. However, the availability of such specifications is seriously limited at the moment. Manually writing the specifications can be prohibitively expensive and error‐prone, while existing automated approaches to inferring the specifications seldom produce results that are strong enough to be used in programme analysis. In this work, we propose the Doc2smt approach to generating strong functional constraints in SMT for library methods based on their documentations. Doc2smt first applies natural language processing (NLP) techniques and a set of rules to translate a method's natural language documentation into a large number of candidate constraint clauses in OCL. Then, it utilizes a manually enhanced domain model to identify OCL candidate constraint clauses that comply with the problem domain in static validation, translates well‐formed OCL constraints into the SMT‐LIB format, and checks whether each SMB‐LIB constraint rightly abstracts the functionalities of the method under consideration via testing in dynamic validation. In the end, it reports the first functional constraint that survives both validations to the user as the result. We have implemented the approach into a supporting tool with the same name. In experiments conducted on 451 methods from the Java Collections Framework and the Java IO library, Doc2smt generated correct constraints for 309 methods, with the average generation time for each correct constraint being merely 2.7 min. We have also applied the generated constraints to facilitate symbolic‐execution‐based test generation with the Symbolic Java PathFinder (SPF) tool. For 24 utility methods manipulating Java container and IO objects, SPF with access to the generated constraints produced 51.2 times more test cases than SPF without the access.

Finding Polluter Tests Using Java PathFinder

Tests that modify (i.e., "pollute") the state shared among tests in a test suite are called \polluter tests". Finding these tests is im- portant because they could result in di erent test outcomes based on the order of the tests in the test suite. Prior work has proposed the PolDet technique for nding polluter tests in runs of JUnit tests on a regular Java Virtual Machine (JVM). Given that Java PathFinder (JPF) provides desirable infrastructure support, such as systematically exploring thread schedules, it is a worthwhile attempt to re-implement techniques such as PolDet in JPF. We present a new implementation of PolDet for nding polluter tests in runs of JUnit tests in JPF. We customize the existing state comparison in JPF to support the so-called \common-root iso- morphism" required by PolDet. We find that our implementation is simple, requiring only -200 lines of code, demonstrating that JPF is a sophisticated infrastructure for rapid exploration of re-search ideas on software testing. We evaluate our implementation on 187 test classes from 13 Java projects and nd 26 polluter tests. Our results show that the runtime overhead of PolDet@JPF com- pared to base JPF is relatively low, on average 1.43x. However, our experiments also show some potential challenges with JPF.

A Double-Level Model Checking Approach for an Agent-Based Autonomous Vehicle and Road Junction Regulations

Usually, the design of an Autonomous Vehicle (AV) does not take into account traffic rules and so the adoption of these rules can bring some challenges, e.g., how to come up with a Digital Highway Code which captures the proper behaviour of an AV against the traffic rules and at the same time minimises changes to the existing Highway Code? Here, we formally model and implement three Road Junction rules (from the UK Highway Code). We use timed automata to model the system and the MCAPL (Model Checking Agent Programming Language) framework to implement an agent and its environment. We also assess the behaviour of our agent according to the Road Junction rules using a double-level Model Checking technique, i.e., UPPAAL at the design level and AJPF (Agent Java PathFinder) at the development level. We have formally verified 30 properties (18 with UPPAAL and 12 with AJPF), where these properties describe the agent’s behaviour against the three Road Junction rules using a simulated traffic scenario, including artefacts like traffic signs and road users. In addition, our approach aims to extract the best from the double-level verification, i.e., using time constraints in UPPAAL timed automata to determine thresholds for the AVs actions and tracing the agent’s behaviour by using MCAPL, in a way that one can tell when and how a given Road Junction rule was selected by the agent. This work provides a proof-of-concept for the formal verification of AV behaviour with respect to traffic rules.

Toward a Holistic Approach to Verification and Validation of Autonomous Cognitive Systems

When applying formal verification to a system that interacts with the real world, we must use a model of the environment. This model represents an abstraction of the actual environment, so it is necessarily incomplete and hence presents an issue for system verification. If the actual environment matches the model, then the verification is correct; however, if the environment falls outside the abstraction captured by the model, then we cannot guarantee that the system is well behaved. A solution to this problem consists in exploiting the model of the environment used for statically verifying the system’s behaviour and, if the verification succeeds, using it also for validating the model against the real environment via runtime verification. The article discusses this approach and demonstrates its feasibility by presenting its implementation on top of a framework integrating the Agent Java PathFinder model checker. A high-level Domain Specific Language is used to model the environment in a user-friendly way; the latter is then compiled to trace expressions for both static formal verification and runtime verification. To evaluate our approach, we apply it to two different case studies: an autonomous cruise control system and a simulation of the Mars Curiosity rover.

Thread Scheduling Sequence Generation Based on All Synchronization Pair Coverage Criteria

Testing multi-thread programs becomes extremely difficult because thread interleavings are uncertain, which may cause a program getting different results in each execution. Thus, Thread Scheduling Sequence (TSS) is a crucial factor in multi-thread program testing. A good TSS can obtain better testing efficiency and save the testing cost especially with the increase of thread numbers. Focusing on the above problem, in this paper, we discuss a kind of approach that can efficiently generate TSS based on the concurrent coverage criteria. First, we give a definition of Synchronization Pair (SP) as well as all Synchronization Pairs Coverage (ASPC) criterion. Then, we introduce the Synchronization Pair Thread Graph (SPTG) to describe the relationships between SPs and threads. Moreover, this paper presents a TSS generation method based on the ASPC according to SPTG. Finally, TSSs automatic generation experiments are conducted on six multi-thread programs in Java Library with the help of Java Path Finder (JPF) tool. The experimental results illustrate that our method not only generates TSSs to cover all SPs but also requires less state number, transition number as well as TSS number when satisfying ASPC, compared with other three widely used TSS generation methods. As a result, it is clear that the efficiency of TSS generation is obviously improved.

A Sound Dynamic Partial Order Reduction Engine for Java Pathfinder

When model checking a multi-threaded program, it is often nec- essary to enumerate the possible ordering of concurrent events to evaluate the behavior of the program. However, enumerating every possible order of events quickly leads to state-space explo- sion. Dynamic Partial Order Reduction (DPOR) is a method to dynamically determine a subset of schedules that need to be evaluated to observe all the relevant behavior of a program. A sound implementation of DPOR in Java Path nder (JPF) can be tricky without incurring unacceptable amounts of overhead, be- cause JPF does not support subdividing existing transitions and conservatively inserting choice generators to end transitions at each possible scheduling point causes JPF to save a large amount of state. We present an extension to JPF, which is an efficient implementation of DPOR that attempts to minimize spacial com- plexity. It handles the directing of the search and uses a simple interface to allow the user to de ne the set of events to operate on and to determine which of those events are dependent. It keeps its own internal representation of all possible scheduling points without inserting choice generators at each point. It then restarts portions of the search, if necessary, to insert only the needed choice generators.

Design, implementation, and application of GPU-based Java bytecode interpreters

We present the design and implementation of GVM, the first system for executing Java bytecode entirely on GPUs. GVM is ideal for applications that execute a large number of short-living tasks, which share a significant fraction of their codebase and have similar execution time. GVM uses novel algorithms, scheduling, and data layout techniques to adapt to the massively parallel programming and execution model of GPUs. We apply GVM to generate and execute tests for Java projects. First, we implement a sequence-based test generation on top of GVM and design novel algorithms to avoid redundant test sequences. Second, we use GVM to execute randomly generated test cases. We evaluate GVM by comparing it with two existing Java bytecode interpreters (Oracle JVM and Java Pathfinder), as well as with the Oracle JVM with just-in-time (JIT) compiler, which has been engineered and optimized for over twenty years. Our evaluation shows that sequence-based test generation on GVM outperforms both Java Pathfinder and Oracle JVM interpreter. Additionally, our results show that GVM performs as well as running our parallel sequence-based test generation algorithm using JVM with JIT with many CPU threads. Furthermore, our evaluation on several classes from open-source projects shows that executing randomly generated tests on GVM outperforms sequential execution on JVM interpreter and JVM with JIT.

EdSketch: execution-driven sketching for Java

Sketching is a synthesis approach that allows users to provide high-level insights into a synthesis problem and let synthesis tools complete low-level details. Users write sketches—partial programs that have “holes” and provide test assertions as the correctness criteria. The sketching techniques fill the holes with code fragments such that the complete program satisfies all test assertions. Traditional techniques translate the sketching problem to propositional satisfiability formulas and leverage SAT solvers to generate programs with the desired functionality. While effective for a range of small well-defined domains, such translation-based approaches have a key limitation when applying to real applications: They require either translating all relevant libraries that are invoked directly or indirectly by the given sketch or creating models of those libraries, which requires much manual effort. This paper introduces execution-driven sketching, a novel approach for synthesizing Java programs with on-demand candidate generation. The key novelty of our work is to leverage runtime behavior to prune a large amount of search space. EdSketch explores the actual program behaviors in the presence of libraries and sketches small parts of real-world applications, which may use complex constructs of modern languages, such as reflection, native calls and File I/O. We further leverage a set of pruning strategies based on Java syntax to expedite the synthesis process. EdSketch embodies our approach in two forms: a stateful search based on the Java PathFinder model checker; and a stateless search based on re-execution inspired by the VeriSoft model checker. Experimental results show that EdSketch can complete some sketches that contain complex constructs in the presence of libraries, recursive procedures and advanced features like reflection. Without translating to SAT, EdSketch ’s performance compares well with the SAT-based Sketch system for a range of small but complex data structure subjects.

Towards Model Checking Android Applications

As feature-rich Android applications (apps for short) are increasingly popularized in security-sensitive scenarios, methods to verify their security properties are highly desirable. Existing approaches on verifying Android apps often have limited effectiveness. For instance, static analysis often suffers from a high false-positive rate, whereas approaches based on dynamic testing are limited in coverage. In this work, we propose an alternative approach, which is to apply the software model checking technique to verify Android apps. We have built a general framework named DroidPF upon Java PathFinder (JPF), towards model checking Android apps. In the framework, we craft an executable mock-up Android OS which enables JPF to dynamically explore the concrete state spaces of the tested apps; we construct programs to generate user interaction and environmental input so as to drive the dynamic execution of the apps; and we introduce Android specific reduction techniques to help alleviate the state space explosion. DroidPF focuses on common security vulnerabilities in Android apps including sensitive data leakage involving a non-trivial flow- and context-sensitive taint-style analysis. DroidPF has been evaluated with 131 apps, which include real-world apps, third-party libraries, malware samples and benchmarks for evaluating app analysis techniques like ours. DroidPF precisely identifies nearly all of the previously known security issues and nine previously unreported vulnerabilities/bugs.

The MCAPL Framework including the Agent Infrastructure Layer an Agent Java Pathfinder

The MCAPL Framework including the Agent Infrastructure Layer an Agent Java Pathfinder

Concurrency testing using symbolic path finder

Concurrent programs have specific features such as italic communication, synchronization and nondeterministic behavior that make the testing activity complex. The objective is to find various types of concurrent defects. In this paper, we have used a model checking tool called Symbolic Path Finder (SPF) which is the upgradation of Java Path Finder (JPF) for concurrent testing. SPF is used for generating the test cases to check concurrent defects such as deadlock, race condition etc. SPF generates symbolic execution tree of the given code which is used as an input for test case generation. The execution is done for finding the test cases in concurrent program where number of threads is operating together with the concurrent defects. The test cases show the type of concurrent defects in the respective line number of the source code.

Fast detection of concurrency errors by state space traversal with randomization and early backtracking

State space traversal is a very popular approach to detect concurrency errors and test concurrent programs. However, it is not practically feasible for complex programs with many thread interleavings and a large state space. Many techniques explore only a part of the state space in order to find errors quickly—building upon the observation that errors can often be found in a particular small part of the state space. Great improvements in performance have been achieved also through randomization. In the context of this research direction, we present the DFS-RB algorithm that augments the standard algorithm for depth-first traversal with early backtracking. Specifically, it is possible to backtrack early from a state before all outgoing transitions have been explored. The DFS-RB algorithm is non-deterministic—it uses random numbers, together with values of several parameters, to determine when and how early backtracking takes place in the search. To evaluate DFS-RB, we performed a large experimental study with our prototype implementation in Java Pathfinder on several Java programs. The results show that DFS-RB achieves better performance in terms of speed and error detection than many state-of-the-art techniques for many benchmarks in our set. Nevertheless, it is difficult to find a single configuration of DFS-RB that works well for many different benchmarks. We designed a ranking algorithm whose purpose is to identify configurations that yield overall consistently good performance with a small variation.