Parser Generator Research Articles

A derivative-based parser generator for visibly Pushdown grammars

In this paper, we present a derivative-based, functional recognizer and parser generator for visibly pushdown grammars. The generated parser accepts ambiguous grammars and produces a parse forest containing all valid parse trees for an input string in linear time. Each parse tree in the forest can then be extracted also in linear time. Besides the parser generator, to allow more flexible forms of the visibly pushdown grammars, we also present a translator that converts a tagged CFG to a visibly pushdown grammar in a sound way, and the parse trees of the tagged CFG are further produced by running the semantic actions embedded in the parse trees of the translated visibly pushdown grammar. The performance of the parser is compared with a popular parsing tool ANTLR and other popular hand-crafted parsers. The correctness of the core parsing algorithm is formally verified in the proof assistant Coq.

A compiler for multi-key homomorphic signatures for Turing machines

At SCN 2018, Fiore and Pagnin proposed a generic compiler (called “Matrioska”) allowing to transform sufficiently expressive single-key homomorphic signatures (SKHSs) into multi-key homomorphic signatures (MKHSs) under falsifiable assumptions in the standard model. Matrioska is designed for homomorphic signatures that support programs represented as circuits. The MKHS schemes obtained through Matrioska support the evaluation and verification of arbitrary circuits over data signed from multiple users, but they require the underlying SKHS scheme to work with circuits whose size is exponential in the number of users, and thus can only support a constant number of users.In this work, we propose a new generic compiler to convert an SKHS scheme into an MKHS scheme. Our compiler is a generalization of Matrioska for homomorphic signatures that support programs in any model of computation. When instantiated with SKHS for circuits, we recover the Matrioska compiler of Fiore and Pagnin. As an additional contribution, we show how to instantiate our generic compiler in the Turing Machines (TM) model and argue that this instantiation allows to overcome some limitations of Matrioska:•First, the MKHS we obtain require the underlying SKHS to support TMs whose size depends only linearly in the number of users.•Second, when instantiated with an SKHS with succinctness poly(λ) and fast enough verification time, e.g., S⋅log⁡T+n⋅poly(λ) or T+n⋅poly(λ) (where T, S, and n are the running time, description size, and input length of the program to verify, respectively), our compiler yields an MKHS in which the time complexity of both the prover and the verifier remains poly(λ) even if executed on programs with inputs from poly(λ) users. While we leave constructing an SKHS with these efficiency properties as an open problem, we make one step towards this goal by proposing an SKHS scheme with verification time poly(λ)⋅T under falsifiable assumptions in the standard model.

Compiler Design for Distributed Quantum Computing

In distributed quantum computing architectures, with the network and communications functionalities provided by the Quantum Internet, remote quantum processing units (QPUs) can communicate and cooperate for executing computational tasks that single NISQ devices cannot handle by themselves. To this aim, distributed quantum computing requires a new generation of quantum compilers, for mapping any quantum algorithm to any distributed quantum computing architecture. With this perspective, in this paper, we first discuss the main challenges arising with compiler design for distributed quantum computing. Then, we analytically derive an upper bound of the overhead induced by quantum compilation for distributed quantum computing. The derived bound accounts for the overhead induced by the underlying computing architecture as well as the additional overhead induced by the sub-optimal quantum compiler -- expressly designed through the paper to achieve three key features, namely, general-purpose, efficient and effective. Finally, we validate the analytical results and we confirm the validity of the compiler design through an extensive performance analysis.

Lake symbols for island parsing

Context: An island parser reads an input text and builds the parse (or abstract syntax) tree of only the programming constructs of interest in the text. These constructs are called islands and the rest of the text is called water, which the parser ignores and skips over. Since an island parser does not have to parse all the details of the input, it is often easy to develop but still useful enough for a number of software engineering tools. When a parser generator is used, the developer can implement an island parser by just describing a small number of grammar rules, for example, in Parsing Expression Grammar (PEG). Inquiry: In practice, however, the grammar rules are often complicated since the developer must define the water inside the island; otherwise, the island parsing will not reduce the total number of grammar rules. When describing the grammar rules for such water, the developer must consider other rules and enumerate a set of symbols, which we call alternative symbols. Due to this difficulty, island parsing seems to be not widely used today despite its usefulness in many applications. Approach: This paper proposes the lake symbols for addressing this difficulty in developing an island parser. It also presents an extension to PEG for supporting the lake symbols. The lake symbols automate the enumeration of the alternative symbols for the water inside an island. The paper proposes an algorithm for translating the extended PEG to the normal PEG, which can be given to an existing parser generator based on PEG. Knowledge: The user can use lake symbols to define water without specifying each alternative symbol. Our algorithms can calculate all alternative symbols for a lake symbol, based on where the lake symbol is used in the grammar. Grounding: We implemented a parser generator accepting our extended PEG and implemented 36 island parsers for Java and 20 island parsers for Python. Our experiments show that the lake symbols reduce 42 % of grammar rules for Java and 89 % of rules for Python on average, excluding the case where islands are expressions. Importance: This work eases the use of island parsing. Lake symbols enable the user to define the water inside the island simpler than before. Defining water inside the island is essential to apply island parsing for practical programming languages.

Description Of Context Free Grammars In Json Format For Parser Generators

Analysis of various presentations for context free grammars provided with parser generators. A new description format of context free grammars is proposed. Given a representation of context free grammar in JSON format. The concept of a new parser generator based on JSON data format of describing context free grammars is presented. Described a parser generation scheme based on that concept.

Staged selective parser combinators

Parser combinators are a middle ground between the fine control of hand-rolled parsers and the high-level almost grammar-like appearance of parsers created via parser generators. They also promote a cleaner, compositional design for parsers. Historically, however, they cannot match the performance of their counterparts. This paper describes how to compile parser combinators into parsers of hand-written quality. This is done by leveraging the static information present in the grammar by representing it as a tree. However, in order to exploit this information, it will be necessary to drop support for monadic computation since this generates dynamic structure. Selective functors can help recover lost functionality in the absence of monads, and the parser tree can be partially evaluated with staging. This is implemented in a library called Parsley.

Morbig: A Static parser for POSIX shell

The POSIX shell language defies conventional wisdom of compiler construction on several levels: The shell language was not designed for static parsing, but with an intertwining of syntactic analysis and execution by expansion in mind. Token recognition cannot be specified by regular expressions and lexical analysis depends on the parsing context and the evaluation context. Besides, the unorthodox design choices of the shell language fit badly in the usual specification languages used to describe other programming languages. This makes the standard usage of Lex and Yacc as a pipeline inadequate for the implementation of a parser for POSIX shell. The existing implementations of shell parsers are complex and use low-level character-level parsing code that is difficult to relate to the POSIX specification. We find it hard to trust such parsers, especially when using them for writing automatic verification tools for shell scripts. This paper offers an overview of the technical difficulties related to the syntactic analysis of the POSIX shell language. It also describes how we have resolved these difficulties using advanced parsing techniques (namely speculative parsing, parser state introspection, context-dependent lexical analysis and longest-prefix parsing) while keeping the implementation at a sufficiently high level of abstraction so that experts can check that the POSIX standard is respected. The resulting tool, called Morbig, is an open-source static parser for a well-defined and realistic subset of the POSIX shell language. Its implementation crucially relies on the purity and incrementality of LR(1) parsers generated by Menhir, a parser generator for OCaml.

Purely functional GLL parsing

Abstract Generalised parsing has become increasingly important in the context of software language design and several compiler generators and language workbenches have adopted generalised parsing algorithms such as GLR and GLL. The original GLL parsing algorithms are described in low-level pseudo-code as the output of a parser generator. This paper explains GLL parsing differently, defining the FUN-GLL algorithm as a collection of pure, mathematical functions and focussing on the logic of the algorithm by omitting implementation details. In particular, the data structures are modelled by abstract sets and relations rather than specialised implementations. The description is further simplified by omitting lookahead and adopting the binary subtree representation of derivations to avoid the clerical overhead of graph construction. Conventional parser combinators inherit the drawbacks from the recursive descent algorithms they implement. Based on FUN-GLL, this paper defines generalised parser combinators that overcome these problems. The algorithm is described in the same notation and style as FUN-GLL and uses the same data structures. Both algorithms are explained as a generalisation of basic recursive descent algorithms. The generalised parser combinators of this paper have several advantages over combinator libraries that generate internal grammars. For example, with the generalised parser combinators it is possible to parse larger permutation phrases and to write parsers for languages that are not context-free. The ‘BNF combinator library’ is built around the generalised parser combinators. With the library, embedded and executable syntax specifications are written. The specifications contain semantic actions for interpreting programs and constructing syntax trees. The library takes advantage of Haskell’s type-system to type-check semantic actions and Haskell’s abstraction mechanism enables ‘reuse through abstraction’. The practicality of the library is demonstrated by running parsers obtained from the syntax descriptions of several software languages.

A deterministic parsing algorithm for ambiguous regular expressions

We introduce a new parser generator, called Berry–Sethi Parser (BSP), for ambiguous regular expressions (RE). The generator constructs a deterministic finite-state transducer that recognizes an input string, as the classical Berry–Sethi algorithm does, and additionally outputs a linear representation of all the syntax trees of the string; for infinitely ambiguous strings, a policy for selecting representative sets of trees is chosen. To construct the transducer, the RE symbols, including letters, parentheses and other metasymbols, are distinctly numbered, so that the corresponding language becomes locally testable. In this way a deterministic position automaton can be constructed, which recognizes and translates the input into a compact DAG representation of the syntax trees. The correctness of the construction is proved. The transducer operates in a linear time on the input. Its descriptive complexity is analyzed as a function of established RE parameters: the alphabetic width, the number of null string symbols and the height of the RE tree. A condition for checking RE ambiguity on the transducer graph is stated. Experimental results of running the parser generator and the parser on a large RE collection are presented. The POSIX RE disambiguation criterion has also been applied to the parser.

Automatic Compilation of Diverse CNNs Onto High-Performance FPGA Accelerators

A broad range of applications are increasingly benefiting from the rapid and flourishing development of convolutional neural networks (CNNs). The FPGA-based CNN inference accelerator is gaining popularity due to its high-performance and low-power as well as FPGA’s conventional advantage of reconfigurability and flexibility. Without a general compiler to automate the implementation, however, significant efforts and expertise are still required to customize the design for each CNN model. In this paper, we present an register-transfer level (RTL)-level CNN compiler that automatically generates customized FPGA hardware for the inference tasks of various CNNs, in order to enable high-level fast prototyping of CNNs from software to FPGA and still keep the benefits of low-level hardware optimization. First, a general-purpose library of RTL modules is developed to model different operations at each layer. The integration and dataflow of physical modules are predefined in the top-level system template and reconfigured during compilation for a given CNN algorithm. The runtime control of layer-by-layer sequential computation is managed by the proposed execution schedule so that even highly irregular and complex network topology, e.g., GoogLeNet and ResNet, can be compiled. The proposed methodology is demonstrated with various CNN algorithms, e.g., NiN, VGG, GoogLeNet, and ResNet, on two standalone Intel FPGAs, Arria 10, and Stratix 10, achieving end-to-end inference throughputs of 969 GOPS and 1604 GOPS, respectively, with batch size of one.

LR 오토마타 생성 모듈을 공유하고 범용 프로그래밍언어로 명세를 작성하는 파서 생성 도구

본 논문은 LR 파서를 쉽게 개발하기 위하여 두 가지 아이디어를 제안한다. 첫째, 오토마타 생성을 모듈화하여 새로운 프로그래밍 언어를 위한 파서 생성 도구를 쉽게 개발 할 수 있다. 둘째, 파서 명세를 일반 프로그래밍언어로 작성하도록 구성하여 이 언어 개발 환경에서 제공하는 구문 오류, 자동 완성, 타입 오류 검사 기능들을 이용하여 파서 명세의 오류를 바로잡을 수 있다. 이 연구에서 제안한 아이디어로 Python, Java, C++, Haskell로 파서를 작성할 수 있는 도구를 개발하였고, 실험을 통하여 위 두 가지 장점을 보였다.

SIFO: Secure Computational Infrastructure Using FPGA Overlays

Secure Function Evaluation (SFE) has received recent attention due to the massive collection and mining of personal data, but remains impractical due to its large computational cost. Garbled Circuits (GC) is a protocol for implementing SFE which can evaluate any function that can be expressed as a Boolean circuit and obtain the result while keeping each party’s input private. Recent advances have led to a surge of garbled circuit implementations in software for a variety of different tasks. However, these implementations are inefficient, and therefore GC is not widely used, especially for large problems. This research investigates, implements, and evaluates secure computation generation using a heterogeneous computing platform featuring FPGAs. We have designed and implemented SIFO: secure computational infrastructure using FPGA overlays. Unlike traditional FPGA design, a coarse-grained overlay architecture is adopted which supports mapping SFE problems that are too large to map to a single FPGA. Host tools provided include SFE problem generator, parser, and automatic host code generation. Our design allows repurposing an FPGA to evaluate different SFE tasks without the need for reprogramming and fully explores the parallelism for any GC problem. Our system demonstrates an order of magnitude speedup compared with an existing software platform.

How to Accurately and Privately Identify Anomalies.

Identifying anomalies in data is central to the advancement of science, national security, and finance. However, privacy concerns restrict our ability to analyze data. Can we lift these restrictions and accurately identify anomalies without hurting the privacy of those who contribute their data? We address this question for the most practically relevant case, where a record is considered anomalous relative to other records. We make four contributions. First, we introduce the notion of sensitive privacy, which conceptualizes what it means to privately identify anomalies. Sensitive privacy generalizes the important concept of differential privacy and is amenable to analysis. Importantly, sensitive privacy admits algorithmic constructions that provide strong and practically meaningful privacy and utility guarantees. Second, we show that differential privacy is inherently incapable of accurately and privately identifying anomalies; in this sense, our generalization is necessary. Third, we provide a general compiler that takes as input a differentially private mechanism (which has bad utility for anomaly identification) and transforms it into a sensitively private one. This compiler, which is mostly of theoretical importance, is shown to output a mechanism whose utility greatly improves over the utility of the input mechanism. As our fourth contribution we propose mechanisms for a popular definition of anomaly ((β, r)-anomaly) that (i) are guaranteed to be sensitively private, (ii) come with provable utility guarantees, and (iii) are empirically shown to have an overwhelmingly accurate performance over a range of datasets and evaluation criteria.

The Fishes of the Amazon: Distribution and Biogeographical Patterns, with a Comprehensive List of Species

We provide a general compilation of the diversity and geographical distribution of Amazonian fishes, updated to the end of 2018. Our database includes documented distributions of 4214 species (both Amazonian and from surrounding basins), compiled from published information plus original data from ichthyological collections. Our results show that the Amazon basin comprises the most diverse regional assemblage of freshwater fishes in the world, with 2716 valid species (1696 of which are endemic) representing 529 genera, 60 families, and 18 orders. These data permit a view of the diversity and distribution of Amazonian fishes on a basinwide scale, which in turn allows the identification of congruent biogeographical patterns, here defined as the overlapping distributions of two or more lineages (species or monophyletic groups). We recognize 20 distinct distributional patterns of Amazonian fishes, which are herein individually delimited, named, and diagnosed. Not all these patterns are associated with identifiable geographical barriers, and some may result from ecological constraints. All the major Amazonian subdrainages fit into more than one biogeographical pattern. This fact reveals the complex history of hydrographical basins and shows that modern basin-defined units contribute relatively little as explanatory factors for the present distributions of Amazonian fishes. An understanding of geomorphological processes and associated paleographic landscape changes provides a far better background for interpreting observed patterns. Our results are expected to provide a framework for future studies on the diversification and historical biogeography of the Amazonian aquatic biota.

DABKE: Secure deniable attribute-based key exchange framework

We introduce the first deniable attribute-based key exchange (DABKE) framework that is resilient to impersonation attacks. We define the formal security models for DABKE framework, and propose a generic compiler that converts any attribute-based key exchanges into deniable ones. We prove that it can achieve session key security and user privacy in the standard model, and strong deniability in the simulation-based paradigm. In particular, the proposed generic compiler ensures: 1) a dishonest user cannot impersonate other user’s session participation in conversations since implicit authentication is used among authorized users; 2) an authorized user can plausibly deny his/her participation after secure conversations with others; 3) the strongest form of deniability is achieved using one-round communication between two authorized users.

Tolerant parsing using modified LR(1) and LL(1) algorithms with embedded “Any” symbol

Tolerant parsing is a form of syntax analysis aimed at capturing the structure of certain points of interest presented in a source code. While these points should be well-described in a tolerant grammar of the language, other parts of the program are allowed to be described coarse-grained, thereby parser remains tolerant to the possible variations of the irrelevant area. Island grammars are one of the basic tolerant parsing techniques. “Islands” term is used as the relevant code alias, the irrelevant code is called “water”. Efforts required to write water rules are supposed to be as small as possible. Previously, we extended island grammars theory and introduced a novel formal concept of a simplified grammar based on the idea of eliminating water description by replacing it with a special “Any” symbol. To work with this concept, a standard LL(1) parsing algorithm was modified and LanD parser generator was developed. In the paper, “Any”-based modification is described for LR(1) parsing algorithm. In comparison with LL(1) tolerant grammars, LR(1) tolerant grammars are easier to develop and explore due to solid island rules. Supplementary “Any” processing techniques are introduced to make this symbol easier to use while staying in the boundaries of the given simplified grammar definition. Specific error recovery algorithms are presented both for LL and LR tolerant parsing. They allow one to further minimize the number and complexity of water rules and make tolerant grammars extendible. In the experiments section, results of a large-scale LL and LR tolerant parsers testing on the basis of 9 open-source project repositories are presented.

Grammar engineering for multiple front‐ends for Python

SummaryIn this paper, we describe our experience in grammar engineering to construct multiple parsers and front ends for the Python language. We present a metrics‐based study of the evolution of the Python grammars through the multiple versions of the language in an effort to distinguish and measure grammar evolution and to provide a basis of comparison with related research in grammar engineering. To conduct this research, we have built a toolkit, pygrat, which builds on tools developed in other research. We use pygrat to build a system that automates much of the process needed to translate the Python grammars from EBNF to a formalism acceptable to the bison parser generator. We exploit the suite of Python test cases, used by the Python developers, to validate our parser generation. Finally, we describe our use of the menhir parser generator to facilitate the parser and front‐end construction, eliminating some of the transformations and providing practical support for grammar modularisation.

Error recovery in parsing expression grammars through labeled failures and its implementation based on a parsing machine

Parsing Expression Grammars (PEGs) are a formalism used to describe top-down parsers with backtracking. As PEGs do not provide a good error recovery mechanism, PEG-based parsers usually do not recover from syntax errors in the input, or recover from syntax errors using ad-hoc, implementation-specific features. The lack of proper error recovery makes PEG parsers unsuitable for use with Integrated Development Environments (IDEs), which need to build syntactic trees even for incomplete, syntactically invalid programs.We discuss a conservative extension, based on PEGs with labeled failures, that adds a syntax error recovery mechanism for PEGs. This extension associates recovery expressionsto labels, where a label now not only reports a syntax error but also uses this recovery expression to reach a synchronization point in the input and resume parsing. We give an operational semantics of PEGs with this recovery mechanism, as well as an operational semantics for a parsing machinethat we can translate labeled PEGs with error recovery to, and prove the correctness of this translation. We use an implementation of labeled PEGs with error recovery via a parsing machine to build robust parsers, which use different recovery strategies, for the Lua language. We evaluate the effectiveness of these parsers, alone and in comparison with a Lua parser with automatic error recovery generated by ANTLR, a popular parser generator .

Merlin: a language server for OCaml (experience report)

We report on the experience of developing Merlin, a language server for the OCaml programming language in development since 2013. Merlin is a daemon that connects to your favourite text editor and provides services that require a fine-grained understanding of the programming language syntax and static semantics: instant feedback on warnings and errors, autocompletion, "type of the code under the cursor", "go to definition", etc. Language servers need to handle incomplete and partially-incorrect programs, and try to be incremental to minimize recomputation after small editing actions. Merlin was built by carefully adapting the existing tools (the OCamllex lexer and Menhir parser generators) to better support incrementality, incompleteness and error handling. These extensions are elegant and general, as demonstrated by the interesting, unplanned uses that the OCaml community found for them. They could be adapted to other frontends -- in any language. Besides incrementality, we discuss the way Merlin communicates with editors, describe the design decisions that went into some demanding features and report on some of the non-apparent difficulties in building good editor support, emerging from expressive programming languages or frustrating tooling ecosystems. We expect this experience report to be of interest to authors of interactive language tooling for any programming language; many design choices may be reused, and some hard-won lessons can serve as warnings.

A Step-by-Step Solution Methodology for Mathematical Expressions

In this paper, we propose a methodology for the step-by-step solution of problems, which can be incorporated into a computer algebra system. Our main aim is to show all the intermediate evaluation steps of mathematical expressions from the start to the end of the solution. The first stage of the methodology covers the development of a formal grammar that describes the syntax and semantics of mathematical expressions. Using a compiler generation tool, the second stage produces a parser from the grammar description. The parser is used to convert a particular mathematical expression into an Abstract Syntax Tree (AST), which is evaluated in the third stage by traversing al its nodes. After every evaluation of some nodes, which corresponds to an intermediate solution step of the related expression, the resulting AST is transformed into the corresponding mathematical expression and then displayed. Many other algebra-related issues such as simplification, factorization, distribution and substitution can be covered by the solution methodology. We currently focuses on the solutions of various problems associated with the subject of derivative, equations, single variable polynomials, and operations on functions. However, it can easily be extended to cover the other subjects of general mathematics.