Manipulation Of Data Structures Research Articles

Solving constrained Horn clauses modulo algebraic data types and recursive functions

This work addresses the problem of verifying imperative programs that manipulate data structures, e.g., Rust programs. Data structures are usually modeled by Algebraic Data Types (ADTs) in verification conditions. Inductive invariants of such programs often require recursively defined functions (RDFs) to represent abstractions of data structures. From the logic perspective, this reduces to solving Constrained Horn Clauses (CHCs) modulo both ADT and RDF. The underlying logic with RDFs is undecidable. Thus, even verifying a candidate inductive invariant is undecidable. Similarly, IC3-based algorithms for solving CHCs lose their progress guarantee: they may not find counterexamples when the program is unsafe. We propose a novel IC3-inspired algorithm Racer for solving CHCs modulo ADT and RDF (i.e., automatically synthesizing inductive invariants, as opposed to only verifying them as is done in deductive verification). Racer ensures progress despite the undecidability of the underlying theory, and is guaranteed to terminate with a counterexample for unsafe programs. It works with a general class of RDFs over ADTs called catamorphisms. The key idea is to represent catamorphisms as both CHCs, via relationification , and RDFs, using novel abstractions . Encoding catamorphisms as CHCs allows learning inductive properties of catamorphisms, as well as preserving unsatisfiabilty of the original CHCs despite the use of RDF abstractions, whereas encoding catamorphisms as RDFs allows unfolding the recursive definition, and relying on it in solutions. Abstractions ensure that the underlying theory remains decidable. We implement our approach in Z3 and show that it works well in practice.

PyKVFinder: an efficient and integrable Python package for biomolecular cavity detection and characterization in data science

BackgroundBiomolecular interactions that modulate biological processes occur mainly in cavities throughout the surface of biomolecular structures. In the data science era, structural biology has benefited from the increasing availability of biostructural data due to advances in structural determination and computational methods. In this scenario, data-intensive cavity analysis demands efficient scripting routines built on easily manipulated data structures. To fulfill this need, we developed pyKVFinder, a Python package to detect and characterize cavities in biomolecular structures for data science and automated pipelines.ResultspyKVFinder efficiently detects cavities in biomolecular structures and computes their volume, area, depth and hydropathy, storing these cavity properties in NumPy arrays. Benefited from Python ecosystem interoperability and data structures, pyKVFinder can be integrated with third-party scientific packages and libraries for mathematical calculations, machine learning and 3D visualization in automated workflows. As proof of pyKVFinder’s capabilities, we successfully identified and compared ADRP substrate-binding site of SARS-CoV-2 and a set of homologous proteins with pyKVFinder, showing its integrability with data science packages such as matplotlib, NGL Viewer, SciPy and Jupyter notebook.ConclusionsWe introduce an efficient, highly versatile and easily integrable software for detecting and characterizing biomolecular cavities in data science applications and automated protocols. pyKVFinder facilitates biostructural data analysis with scripting routines in the Python ecosystem and can be building blocks for data science and drug design applications.

Toward an Objective Measure of Developers’ Cognitive Activities

Understanding how developers carry out different computer science activities with objective measures can help to improve productivity and guide the use and development of supporting tools in software engineering. In this article, we present two controlled experiments involving 112 students to explore multiple computing activities (code comprehension, code review, and data structure manipulations) using three different objective measures including neuroimaging (functional near-infrared spectroscopy (fNIRS) and functional magnetic resonance imaging (fMRI)) and eye tracking. By examining code review and prose review using fMRI, we find that the neural representations of programming languages vs. natural languages are distinct. We can classify which task a participant is undertaking based solely on brain activity, and those task distinctions are modulated by expertise. We leverage insights from the psychological notion of spatial ability to decode the neural representations of several fundamental data structures and their manipulations using fMRI, fNIRS, and eye tracking. We examine list, array, tree, and mental rotation tasks and find that data structure and spatial operations use the same focal regions of the brain but to different degrees: they are related but distinct neural tasks. We demonstrate best practices and describe the implication and tradeoffs between fMRI, fNIRS, eye tracking, and self-reporting for software engineering research.

Neural-Guided Inductive Synthesis of Functional Programs on List Manipulation by Offline Supervised Learning

Synthesizing intended programs from user-specified input-output examples, also known as Programming by Examples (PBE), is a challenging problem in program synthesis, and has been applied to a wide range of domains. A key challenge in PBE is to efficiently discover a user-intended program in the search space that can be exponentially large. In this work, we propose a method for automatic synthesis of functional programs on list manipulation, by using offline-trained Recurrent Neural Network (RNN) models to guide the program search. We adopt miniKanren, an embedded domain-specific language for flexible relational programming, as an underlying top-down deductive search engine of candidate programs that are consistent with input-output examples. Our approach targets an easy and effective integration of deep learning techniques in making better PBE systems and combines two technical ideas on generating diverse training dataset and designing rich feature embeddings of probable subproblems for synthesis generated by deductive search. The offline-learned model is then used in PBE to guide the top-down deductive search with specific strategies. To practically manipulate data structures of lists, our method synthesizes functional programs with popular higher-order combinators including $\texttt {map}$ , $\texttt {foldl}$ and $\texttt {foldr}$ . We have implemented our method and evaluated it with challenging program synthesis tasks on list manipulation. The experiments show promising results on the performance of our method compared to related state-of-the-art inductive synthesizers.

Resonator Networks, 1: An Efficient Solution for Factoring High-Dimensional, Distributed Representations of Data Structures.

The ability to encode and manipulate data structures with distributed neural representations could qualitatively enhance the capabilities of traditional neural networks by supporting rule-based symbolic reasoning, a central property of cognition. Here we show how this may be accomplished within the framework of Vector Symbolic Architectures (VSAs) (Plate, 1991; Gayler, 1998; Kanerva, 1996), whereby data structures are encoded by combining high-dimensional vectors with operations that together form an algebra on the space of distributed representations. In particular, we propose an efficient solution to a hard combinatorial search problem that arises when decoding elements of a VSA data structure: the factorization of products of multiple codevectors. Our proposed algorithm, called a resonator network, is a new type of recurrent neural network that interleaves VSA multiplication operations and pattern completion. We show in two examples-parsing of a tree-like data structure and parsing of a visual scene-how the factorization problem arises and how the resonator network can solve it. More broadly, resonator networks open the possibility of applying VSAs to myriad artificial intelligence problems in real-world domains. The companion article in this issue (Kent, Frady, Sommer, & Olshausen, 2020) presents a rigorous analysis and evaluation of the performance of resonator networks, showing it outperforms alternative approaches.

Test input generation using separati logic

Unit testing is the practice of testing individual units of a program, where a unit can be a method or a group of methods. Unit testing is important for software development, and automated test case generation is important for reducing the cost and improving the quality of unit testing. When an input of a un is a dynamically allocated data structure, such as list and tree, it has unbounded domain, and strict requirements over the shape or size. This makes test case generation notoriously hard. In this talk, we present Java StarFinder (JSF), a JPF extension for testing units that manipulate data structures. JSF is a symbolic execution engine that uses separation logi specification to capture the constraints over the input, and performs context-sensitive lazy initialization to construct valid test cases.

Abstract Representation of Binders in OCaml using the Bindlib Library

The Bindlib library for OCaml provides a set of tools for the manipulation of data structures with variable binding. It is very well suited for the representation of abstract syntax trees, and has already been used for the implementation of half a dozen languages and proof assistants (including a new version of the logical framework Dedukti). Bindlib is optimised for fast substitution, and it supports variable renaming. Since the representation of binders is based on higher-order abstract syntax, variable capture cannot arise during substitution. As a consequence, variable names are not updated at substitution time. They can however be explicitly recomputed to avoid "visual capture" (i.e., distinct variables with the same apparent name) when a data structure is displayed.

Feedback control can make data structure layout randomization more cost-effective under zero-day attacks

In the wake of the research community gaining deep understanding about control-hijacking attacks, data-oriented attacks have emerged. Among data-oriented attacks, data structure manipulation attack (DSMA) is a major category. Pioneering research was conducted and shows that DSMA is able to circumvent the most effective defenses against control-hijacking attacks — DEP, ASLR and CFI. Up to this day, only two defense techniques have demonstrated their effectiveness: Data Flow Integrity (DFI) and Data Structure Layout Randomization (DSLR). However, DFI has high performance overhead, and dynamic DSLR has two main limitations. L-1: Randomizing a large set of data structures will significantly affect the performance. L-2: To be practical, only a fixed sub-set of data structures are randomized. In the case that the data structures targeted by an attack are not covered, dynamic DSLR is essentially noneffective.To address these two limitations, we propose a novel technique, feedback-control-based adaptive DSLR and build a system named SALADSPlus. SALADSPlus seeks to optimize the trade-off between security and cost through feedback control. Using a novel feedback-control-based adaptive algorithm extended from the Upper Confidence Bound (UCB) algorithm, the defender (controller) uses the feedbacks (cost-effectiveness) from previous randomization cycles to adaptively choose the set of data structures to randomize (the next action). Different from dynamic DSLR, the set of randomized data structures are adaptively changed based on the feedbacks. To obtain the feedbacks, SALADSPlus inserts canary in each data structure at the time of compilation. We have implemented SALADSPlus based on gcc-4.5.0. Experimental results show that the runtime overheads are 1.8%, 3.7%, and 5.3% when the randomization cycles are selected as 10s, 5s, and 1s respectively.

Natural synthesis of provably-correct data-structure manipulations

This paper presents natural synthesis, which generalizes the proof-theoretic synthesis technique to support very expressive logic theories. This approach leverages the natural proof methodology and reduces an intractable, unbounded-size synthesis problem to a tractable, bounded-size synthesis problem, which is amenable to be handled by modern inductive synthesis engines. The synthesized program admits a natural proof and is a provably-correct solution to the original synthesis problem. We explore the natural synthesis approach in the domain of imperative data-structure manipulations and present a novel syntax-guided synthesizer based on natural synthesis. The input to our system is a program template together with a rich functional specification that the synthesized program must meet. Our system automatically produces a program implementation along with necessary proof artifacts, namely loop invariants and ranking functions, and guarantees the total correctness with a natural proof. Experiments show that our natural synthesizer can efficiently produce provably-correct implementations for sorted lists and binary search trees. To our knowledge, this is the first system that can automatically synthesize these programs, their functional correctness and their termination in tandem from bare-bones control flow skeletons.

Automatic induction proofs of data-structures in imperative programs

We consider the problem of automated reasoning about dynamically manipulated data structures. Essential properties are encoded as predicates whose definitions are formalized via user-defined recursive rules. Traditionally, proving relationships between such properties is limited to the unfold-and-match (U+M) paradigm which employs systematic transformation steps of folding/unfolding the rules. A proof, using U+M, succeeds when we find a sequence of transformations that produces a final formula which is obviously provable by simply matching terms. Our contribution here is the addition of the fundamental principle of induction to this automated process. We first show that some proof obligations that are dynamically generated in the process can be used as induction hypotheses in the future, and then we show how to use these hypotheses in an induction step which generates a new proof obligation aside from those obtained by using the fold/unfold operations. While the adding of induction is an obvious need in general, no automated method has managed to include this in a systematic and general way. The main reason for this is the problem of avoiding circular reasoning. We overcome this with a novel checking condition. In summary, our contribution is a proof method which – beyond U+M – performs automatic formula re-writing by treating previously encountered obligations in each proof path as possible induction hypotheses. In the practical evaluation part of this paper, we show how the commonly used technique of using unproven lemmas can be avoided, using realistic benchmarks. This not only removes the current burden of coming up with the appropriate lemmas, but also significantly boosts up the verification process, since lemma applications, coupled with unfolding, often induce a large search space. In the end, our method can automatically reason about a new class of formulas arising from practical program verification.

Natural proofs for data structure manipulation in C using separation logic

The natural proof technique for heap verification developed by Qiu et al. [32] provides a platform for powerful sound reasoning for specifications written in a dialect of separation logic called Dryad. Natural proofs are proof tactics that enable automated reasoning exploiting recursion, mimicking common patterns found in human proofs. However, these proofs are known to work only for a simple toy language [32]. In this work, we develop a framework called VCDryad that extends the Vcc framework [9] to provide an automated deductive framework against separation logic specifications for C programs based on natural proofs. We develop several new techniques to build this framework, including (a) a novel tool architecture that allows encoding natural proofs at a higher level in order to use the existing Vcc framework (including its intricate memory model, the underlying type-checker, and the SMT-based verification infrastructure), and (b) a synthesis of ghost-code annotations that captures natural proof tactics, in essence forcing Vcc to find natural proofs using primarily decidable theories. We evaluate our tool extensively, on more than 150 programs, ranging from code manipulating standard data structures, well-known open source library routines (Glib, OpenBSD), Linux kernel routines, customized OS data structures, etc. We show that all these C programs can be fully automatically verified using natural proofs (given pre/post conditions and loop invariants) without any user-provided proof tactics . VCDryad is perhaps the first deductive verification framework for heap-manipulating programs in a real language that can prove such a wide variety of programs automatically.

Dependent vector types for data structuring in multirate Faust

Faust is a functional programming language dedicated to the specification of executable monorate synchronous musical applications. To extend Faust capabilities to important domains such as FFT-based spectral processing, we introduce here a multirate extension of the core Faust language. The novel idea is to link rate changes to data structure manipulation operations. Creating a vector-valued output signal divides the rate of input signals by the vector size, while serializing vectors multiplies rates accordingly. As duals to vectors, we also introduce record-like data structures, which are used to gather data but do not change signal rates. This interplay between data structures and rates is made possible in the language static semantics by the introduction of dependent types. We present a typing semantics, a denotational semantics and correctness theorems that show that this data structuring/multirate extension preserves the language synchronous characteristics. This new design is under implementation in the Faust compiler.

A fresh look at programming with names and binders

A wide range of computer programs, including compilers and theorem provers, manipulate data structures that involve names and binding. However, the design of programming idioms which allow performing these manipulations in a safe and natural style has, to a large extent, remained elusive. In this paper, we present a novel approach to the problem. Our proposal can be viewed either as a programming language design or as a library: in fact, it is currently implemented within Agda. It provides a safe and expressive means of programming with names and binders. It is abstract enough to support multiple concrete implementations: we present one in nominal style and one in de Bruijn style. We use logical relations to prove that "well-typed programs do not mix names with different scope". We exhibit an adequate encoding of Pitts-style nominal terms into our system.

Pointer Logic Dealing with Uncertain Equality of Pointers

A pointer logic is designed for a C-like programming language PointerC.The pointer logic is an extension of Hoare logic,and it uses the idea of precise alias analysis in pointer program verification to support safety verification of programs in which equality of pointers is well-regulated.This paper presents an extension to the pointer logic by introducing a set of uncertain-equality pointer access path sets,so as to reason in the extended pointer logic about properties of programs which manipulate data structures like directed graph in which equality of pointers is uncertain.

Sinfonia

We propose a new paradigm for building scalable distributed systems. Our approach does not require dealing with message-passing protocols, a major complication in existing distributed systems. Instead, developers just design and manipulate data structures within our service called Sinfonia. Sinfonia keeps data for applications on a set of memory nodes, each exporting a linear address space. At the core of Sinfonia is a new minitransaction primitive that enables efficient and consistent access to data, while hiding the complexities that arise from concurrency and failures. Using Sinfonia, we implemented two very different and complex applications in a few months: a cluster file system and a group communication service. Our implementations perform well and scale to hundreds of machines.

Nonblocking k-Compare-Single-Swap

The current literature offers two extremes of nonblocking software synchronization support for concurrent data structure design: intricate designs of specific structures based on single-location operations such as compare-and-swap (CAS), and general-purpose multilocation transactional memory implementations. While the former are sometimes efficient, they are invariably hard to extend and generalize. The latter are flexible and general, but costly. This paper aims at a middle ground: reasonably efficient multilocation operations that are general enough to reduce the design difficulties of algorithms based on CAS alone. We present an obstruction-free implementation of an atomic k -location-compare single-location-swap (KCSS) operation. KCSS allows for simple nonblocking manipulation of linked data structures by overcoming the key algorithmic difficulty in their design: making sure that while a pointer is being manipulated, neighboring parts of the data structure remain unchanged. Our algorithm is efficient in the common uncontended case: A successful k-location KCSS operation requires only two CAS operations, two stores, and 2k noncached loads when there is no contention. We therefore believe our results lend themselves to efficient and flexible nonblocking manipulation of list-based data structures in today’s architectures.

Automatic software verification for robotics

This thesis investigates the application of novel software verification techniques to robotic systems. Control software lies at the core of these systems, and its complexity poses several challenges to the verification task, among which the intensive use of numeric values and constraints, and the manipulation of data structures. This thesis puts forward a Model Checking procedure specifically targeted for control software that manipulates numeric values, arithmetic constraints, and data structures such as arrays and matrices. It also demonstrates the effectiveness of this procedure in practice, when applied to robotic systems.

Sinfonia

We propose a new paradigm for building scalable distributed systems. Our approach does not require dealing with message-passing protocols -- a major complication in existing distributed systems. Instead, developers just design and manipulate data structures within our service called Sinfonia. Sinfonia keeps data for applications on a set of memory nodes, each exporting a linear address space. At the core of Sinfonia is a novel minitransaction primitive that enables efficient and consistent access to data, while hiding the complexities that arise from concurrency and failures. Using Sinfonia, we implemented two very different and complex applications in a few months: a cluster file system and a group communication service. Our implementations perform well and scale to hundreds of machines.

SpectrinoSoftware: Spectra Visualization and Preparation forR

spectrino is a spectra preparation software utility for the R language and environment for statistical computing. It is an operating-system specific tool, for use under Microsoft Windows, with specialized visualization, organization and preprocessing features for spectra. The software accepts spectral data from analytical instruments and then prepares a data structure to be introduced in R. spectrino has a rich set of features to create data structures and visually manipulate/compare spectra. The application is accessible by a library of functions from within R. These commands allow for the creation and manipulation of data structures in spectrino and the selective extraction of spectral data. Before exporting, the spectra are preprocessed according the requirements of consecutive discriminant analysis. This preprocessing is adjustable by a series of options.

Deciding Boolean Algebra with Presburger Arithmetic

We describe an algorithm for deciding the first-order multisorted theory BAPA, which combines Boolean algebras of sets of uninterpreted elements (BA) and Presburger arithmetic operations (PA). BAPA can express the relationship between integer variables and cardinalities of unbounded finite sets, and it supports arbitrary quantification over sets and integers. Our motivation for BAPA is deciding verification conditions that arise in the static analysis of data structure consistency properties. Data structures often use an integer variable to keep track of the number of elements they store; an invariant of such a data structure is that the value of the integer variable is equal to the number of elements stored in the data structure. When the data structure content is represented by a set, the resulting constraints can be captured in BAPA. BAPA formulas with quantifier alternations arise when verifying programs with annotations containing quantifiers or when proving simulation relation conditions for refinement and equivalence of program fragments. Furthermore, BAPA constraints can be used for proving the termination of programs that manipulate data structures, as well as in constraint database query evaluation and loop invariant inference. We give a formal description of an algorithm for deciding BAPA. We analyze our algorithm and show that it has optimal alternating time complexity and that the complexity of BAPA matches the complexity of PA. Because it works by a reduction to PA, our algorithm yields the decidability of a combination of sets of uninterpreted elements with any decidable extension of PA. When restricted to BA formulas, the algorithm can be used to decide BA in optimal alternating time. Furthermore, the algorithm can eliminate individual quantifiers from a formula with free variables and therefore perform projection onto a desirable set of variables. We have implemented our algorithm and used it to discharge verification conditions in the Jahob system for data structure consistency checking of Java programs; our experience suggest that a straightforward implementation of the algorithm is effective on nontrivial formulas as long as the number of set variables is small. We also report on a new algorithm for solving the quantifier-free fragment of BAPA.