Parser Generator Research Articles

ModelCC — A Pragmatic Parser Generator

Syntax-directed translation tools require the specification of a language by means of a formal grammar. This grammar must also conform to the specific requirements of the parser generator to be used. Software engineers then annotate the resulting grammar with semantic actions for the resulting system to perform its desired functionality. Whenever the input text format is modified, the grammar has to be updated and the subsequent changes propagate throughout the entire language processing tool chain. Moreover, if several applications use the same language, multiple copies of the same language specification have to be maintained in sync, since language specification (i.e. the grammar) is tightly coupled to language processing (i.e. the semantic actions that annotate that grammar). In this paper, we introduce ModelCC, a model-based parser generator that decouples language specification from language processing, hence avoiding the aforementioned problems that are caused by grammar-driven parser generators. ModelCC receives a conceptual model as input, along with constraints that annotate it. It is then able to create a parser for a given textual representation format, so that the generated parser fully automates the instantiation of the desired conceptual model. ModelCC is also a powerful tool for the design of domain-specific languages.

Abstract syntax driven approach for language composition

Abstract Popularity of domain-specific languages brings the problem of language components reuse. It should be possible to use parts of different languages in development of new one to lower costs and also allow incremental development. This problem could be solved using the composition of languages. In this paper we discuss the view of language composition from the perspective of concepts composition, where the role of concrete syntax is lowered. We present examples of language composition based on the principles of object composition implemented using YAJCo parser generator, that allows to specify the language based on its abstract syntax.

Pocl: A Performance-Portable OpenCL Implementation

OpenCL is a standard for parallel programming of heterogeneous systems. The benefits of a common programming standard are clear; multiple vendors can provide support for application descriptions written according to the standard, thus reducing the program porting effort. While the standard brings the obvious benefits of platform portability, the performance portability aspects are largely left to the programmer. The situation is made worse due to multiple proprietary vendor implementations with different characteristics, and, thus, required optimization strategies. In this paper, we propose an OpenCL implementation that is both portable and performance portable. At its core is a kernel compiler that can be used to exploit the data parallelism of OpenCL programs on multiple platforms with different parallel hardware styles. The kernel compiler is modularized to perform target-independent parallel region formation separately from the target-specific parallel mapping of the regions to enable support for various styles of fine-grained parallel resources such as subword SIMD extensions, SIMD datapaths and static multi-issue. Unlike previous similar techniques that work on the source level, the parallel region formation retains the information of the data parallelism using the LLVM IR and its metadata infrastructure. This data can be exploited by the later generic compiler passes for efficient parallelization. The proposed open source implementation of OpenCL is also platform portable, enabling OpenCL on a wide range of architectures, both already commercialized and on those that are still under research. The paper describes how the portability of the implementation is achieved. Our results show that most of the benchmarked applications when compiled using pocl were faster or close to as fast as the best proprietary OpenCL implementation for the platform at hand.

On the incremental growth and shrinkage of LR goto-graphs

The LR(0) goto-graph is the basis for the construction of parsers for several interesting grammar classes such as LALR and GLR. Early work has shown that even when a grammar is an extension to another, the goto-graph of the first is not necessarily a subgraph of the second. Some authors presented algorithms to grow and shrink these graphs incrementally, but the formal proof of the existence of a particular relation between a given goto-graph and a grown or shrunk counterpart seems to be still missing in literature as of today. In this paper we use the recursive projection of paths of limited length to prove the existence of one such relation, when the sets of productions are in a subset relation. We also use this relation to present two algorithms (Grow and Shrink) that transform the goto-graph of a given grammar into the goto-graph of an extension or a restriction to that grammar. We implemented these algorithms in a dynamically updatable LALR parser generator called DEXTER (the Dynamically EXTEnsible Recognizer) that we are now shipping with our current implementation of the Neverlang framework for programming language development.

Ambiguity Detection Methods in Context Free Grammar

Problem arising in CFG (Context Free Grammar) due to ambiguity can be trace to 1962. Even now there is no general method or procedure introduced to detect ambiguity in CFG. In parser generation and in language design, ambiguity in context free grammar, is a frequent problem as well as in application where it is used for the representation of physical structure. For creating a language it should be necessary that it is unambiguous. Ambiguity has some advantages as well as disadvantages. The aim of this study is to analyze different methods dealing with the ambiguity detection in Context Free Grammars. In this study, we will observe usefulness of Ambiguity Detection Method (ADM) in CFG with respect to ambiguity detection, assurance of termination of the process and accuracy.

An Effective Error Recovery Method for GLR Parsers

Deterministic parsing methods such as LALR(1) and LL(1) are widely used in traditional parser generators.Nondeterministic parsing methods such as GLR are used in recent parser generators with the development of software reverse engineering and reengineering.However,there is no published literature to show how to make automatic error recovery for GLR parsers.A new substring based error recovery method is presented,which can detect and recover from parsing errors effectively.Experimental data shows that this method has good performance,and can be used in GLR parser generators.

Authenticated key exchange with synchronized state

ABSTRACTWe study the problem on how to either prevent identity impersonation (IDI) attacks or limit its consequences by on‐line detecting previously unidentified IDI attacks, where IDI attacks are normally caused by the leakage of identity related long‐term key. Such problem has, up until now, lacked a provably good solution. We deal with this problem through the scenario on authenticated key exchange with synchronized state (AKESS). This work provides a security model for AKESS, in which we particularly formalize the security of synchronized state based on indistinguishability. We propose a two party execution state synchronization framework for symmetric case, based on which we propose a generic compiler for AKESS protocols. Our goal is to transform any existing passively secure KE protocols to AKESS protocols using synchronized state, without any modification on those KE protocols. The new generic compiler is probably secure in the standard model under standard assumptions. Copyright © 2014 John Wiley & Sons, Ltd.

A Practical GLR Parser Generator for Software Reverse Engineering

Traditional parser generators use deterministic parsing methods. These methods can not meet the parsing requirements of software reverse engineering effectively. A new parser generator is presented which can generate GLR parser with automatic error recovery. The generated GLR parser has comparable parsing speed with the traditional LALR(1) parser and can be used in the parsing of software reverse engineering.

Singe

We present Singe, a Domain Specific Language (DSL) compiler for combustion chemistry that leverages warp specialization to produce high performance code for GPUs. Instead of relying on traditional GPU programming models that emphasize data-parallel computations, warp specialization allows compilers like Singe to partition computations into sub-computations which are then assigned to different warps within a thread block. Fine-grain synchronization between warps is performed efficiently in hardware using producer-consumer named barriers. Partitioning computations using warp specialization allows Singe to deal efficiently with the irregularity in both data access patterns and computation. Furthermore, warp-specialized partitioning of computations allows Singe to fit extremely large working sets into on-chip memories. Finally, we describe the architecture and general compilation techniques necessary for constructing a warp-specializing compiler. We show that the warp-specialized code emitted by Singe is up to 3.75X faster than previously optimized data-parallel GPU kernels.

Proof-producing translation of higher-order logic into pure and stateful ML

AbstractThe higher-order logic found in proof assistants such as Coq and various HOL systems provides a convenient setting for the development and verification of functional programs. However, to efficiently run these programs, they must be converted (or ‘extracted’) to functional programs in a programming language such as ML or Haskell. With current techniques, this step, which must be trusted, relates similar looking objects that have very different semantic definitions, such as the set-theoretic model of a logic and the operational semantics of a programming language. In this paper, we show how to increase the trustworthiness of this step with an automated technique. Given a functional program expressed in higher-order logic, our technique provides the corresponding program for a functional language defined with an operational semantics, and it provides a mechanically checked theorem relating the two. This theorem can then be used to transfer verified properties of the logical function to the program. We have implemented our technique in the HOL4 theorem prover, translating functions to a subset of Standard ML, and have applied the implementation to examples including functional data structures, a parser generator, cryptographic algorithms, a garbage collector and the 500-line kernel of the HOL light theorem prover. This paper extends our previous conference publication with new material that shows how functions defined in terms of a state-and-exception monad can be translated, with proofs, into stateful ML code. The HOL light example is also new.

Bootstrapping a Compiler for an Equation-Based Object-Oriented Language

What does it mean to bootstrap a compiler, and why do it? This paper reports on the first bootstrapping of a full-scale EOO (Equation-based Object-Oriented) modeling language such as Modelica. Bootstrapping means that the compiler of a language can compile itself. However, the usual application area for the Modelica is modeling and simulation of complex physical systems. Fortunately it turns out that with some minor extensions, the Modelica language is well suited for the modeling of language semantics. We use the name MetaModelica for this slightly extended Modelica. This is a prerequisite for bootstrapping which requires that the language can be used to model and/or implement itself. The OpenModelica Compiler (OMC) has been written in this MetaModelica language. It originally supported only the standard Modelica language but has been gradually extended to also cover the MetaModelica language extensions. After substantial work, OMC is able to quickly compile itself and produces an executable with good performance. The benefits include a more extensible and maintainable compiler by introducing improved language constructs and a more powerful runtime that makes it easy to add functionality such as parser generators, debuggers, and profiling tools. Future work includes extracting and restructuring parts of OMC, making the compiler smaller and more modular and extensible. This will also make it easier to interface with OMC, making it possible to create more powerful and user-friendly OpenModelica-based tools. The compiler and its bootstrapping is a major effort -- it is currently about 330 000 lines of code, and the MetaModelica extensions are used routinely by approximately ten developers on a daily basis.

Compiling irregular applications for reconfigurable systems

Algorithms that exhibit irregular memory access patterns are known to show poor performance on multiprocessor architectures, particularly when memory access latency is variable. Many common data structures, including graphs, trees, and linked-lists, exhibit these irregular memory access patterns. While FPGA-based code accelerators have been successful on applications with regular memory access patterns, they have not been further explored for irregular memory access patterns. Multithreading has been shown to be an effective technique in masking long latencies. We describe the compiler generation of concurrent hardware threads for FPGAs with the objective of masking the memory latency caused by irregular memory access patterns. The CHAT compiler extends the ROCCC toolset to generate customised state information for each dynamically generated thread. Initial results show a speed-up of 50x.

Parser description-based bitstream parser generation for MPEG RMC framework

In this paper, we present a bitstream syntax description scheme that enables the automatic generation of a bitstream parser in the reconfigurable media coding (RMC) framework. In the RMC framework, a standard video decoder can be defined with a set of functional units (FUs) pre-defined in a tool library and a decoder description containing the interconnections between the FUs and the bitstream parser description. The FUs in the tool library are codec-independent in that any FU can be reused. In contrast, the decoder description is codec-specific. In particular, the bitstream parser description is required for the generation of the bitstream parser FU in the RMC framework. There have been several works on bitstream parser description and parser FU generation; however, conventional methods have mainly focused on built-in parser generation during the design time, rather than on the automatic generation of the parser FU in the run-time from the description. As a response to this problem, we propose a bitstream parser description format and a run-time parser generation mechanism based on a generic parser FU. The proposed language aims to support run-time parser FU generation by describing three types of information: control flow, external functions, and parser interfaces. The control flow description is described based on the finite state machine concept, making the translation of the BSD in run-time easy. Throughout the paper, we show how the proposed scheme is better suited for the MPEG RMC framework than the existing description formats. We also discuss the implementation of the generic parser FU (GPFU) to show that the run-time parser generation can be done efficiently with the proposed description format. A simulation result using the generic parser FU based on the existing MPEG standards (MPEG-4 SP and SC3DMC) is provided as an existential proof.

On the semantics of parsing actions

Parsers, whether constructed by hand or automatically via a parser generator tool, typically need to compute some useful semantic information in addition to the purely syntactic analysis of their input. Semantic actions may be added to parsing code by hand, or the parser generator may have its own syntax for annotating grammar rules with semantic actions. In this paper, we take a functional programming view of such actions. We use concepts from the semantics of mostly functional programming languages and adapt them to give meaning to the actions of the parser. Specifically, the semantics is inspired by the categorical semantics of lambda calculi and the use of premonoidal categories for the semantics of effects in programming languages. This framework is then applied to our leading example, the transformation of grammars to eliminate left recursion. The syntactic transformation of left-recursion elimination leads to a corresponding semantic transformation of the actions for the grammar. We prove the semantic transformation correct and relate it to continuation passing style, a widely studied transformation in lambda calculi and functional programming. As an idealization of the input language of parser generators, we define a call-by-value calculus with first-order functions and a type-and-effect system where the effects are given by sequences of grammar symbols. The account of left-recursion elimination is then extended to this calculus.

Principled parsing for indentation-sensitive languages

Several popular languages, such as Haskell, Python, and F#, use the indentation and layout of code as part of their syntax. Because context-free grammars cannot express the rules of indentation, parsers for these languages currently use ad hoc techniques to handle layout. These techniques tend to be low-level and operational in nature and forgo the advantages of more declarative specifications like context-free grammars. For example, they are often coded by hand instead of being generated by a parser generator. This paper presents a simple extension to context-free grammars that can express these layout rules, and derives GLR and LR(k) algorithms for parsing these grammars. These grammars are easy to write and can be parsed efficiently. Examples for several languages are presented, as are benchmarks showing the practical efficiency of these algorithms.

Implementing monads for C++ template metaprograms

C++ template metaprogramming is used in various application areas, such as expression templates, static interface checking, active libraries, etc. Its recognized similarities to pure functional programming languages–like Haskell–make the adoption of advanced functional techniques possible. Such a technique is using monads, programming structures representing computations. Using them actions implementing domain logic can be chained together and decorated with custom code. C++ template metaprogramming could benefit from adopting monads in situations like advanced error propagation and parser construction. In this paper we present an approach for implementing monads in C++ template metaprograms. Based on this approach we have built a monadic framework for C++ template metaprogramming. As real world examples we present a generic error propagation solution for C++ template metaprograms and a technique for building compile-time parser generators. All solutions presented in this paper are implemented and available as an open source library.

An interactive parser generator for context-free grammars

This paper describes a parser generator that accepts arbitrary context-free grammars. It generates a parser using the Earley algorithm [1]. It enables the user to develop, edit, and test a grammar in an interactive graphical environment. This GUI visualizes both the operation of the Earley algorithm as well as the generated parse trees. The generated parsers are saved as fully-functional Java source files, ready to be incorporated into an application. These Java programs can be reloaded into the GUI for further editing of the grammar. Employing this parser generator in a sophomore-level software development course enables students to become proficient in writing a parser with two days of lecture and one assignment.

Proof-producing synthesis of ML from higher-order logic

The higher-order logic found in proof assistants such as Coq and various HOL systems provides a convenient setting for the development and verification of pure functional programs. However, to efficiently run these programs, they must be converted (or "extracted") to functional programs in a programming language such as ML or Haskell. With current techniques, this step, which must be trusted, relates similar looking objects that have very different semantic definitions, such as the set-theoretic model of a logic and the operational semantics of a programming language. In this paper, we show how to increase the trustworthiness of this step with an automated technique. Given a functional program expressed in higher-order logic, our technique provides the corresponding program for a functional language defined with an operational semantics, and it provides a mechanically checked theorem relating the two. This theorem can then be used to transfer verified properties of the logical function to the program. We have implemented our technique in the HOL4 theorem prover, translating functions to a core subset of Standard ML, and have applied it to examples including functional data structures, a parser generator, cryptographic algorithms, and a garbage collector.

A Flexible tool for Discovery and Selection of Sensor Web Registry Services with Extended SOA Framework

Sensor Web Services are recently emerged as promising technology and have potential usage in a wide range of application domain. However due to advancement of OGC SWE standard, the discovery and selection of Sensor Web Registry services throughout heterogeneous environment become a challenging task. Many approaches and frameworks have been proposed to discover the sensor web registry services and some of the approaches assume that the requests are placed in SOAP compatible formats while others focus on GUI based parametric query processing. In this paper, a flexible tool has been proposed that uses the Natural Language Query Processing which is a convenient and easy method of sensor data access in comparison to SQL or XML based Query Language like XQuery and XPath. An architecture based on extended SOA Framework that organizes the method of sensor web registry service discovery and selection in an efficient and structured manner has been suggested by adding some new layers like Request Parser & Query Generator (RPQ), Service Verifier and Certifier (SVC) and Service Rank Calculator (SRC). A typical weather sensor web service scenario is considered in this paper for finding the most relevant service according to requester‟s functional and QoS requirements.

Jaccie: A Java-based compiler–compiler for generating, visualizing and debugging compiler components

Many programmers live in happy ignorance of their compilers’ internal workings. Others may want to take a look at what is going on inside a compiler in much the same way that they use a debugger to watch their compiled programs execute. While conventional compilers are black boxes whose internals are hidden from the user, the Jaccie tool set helps to open up the box and have a look at what is going on inside.Technically speaking, Jaccie is a compiler–compiler that, from suitable formal descriptions, generates the scanner, parser, and attribute evaluator components of a compiler and presents them in a visual debugging environment. It offers a number of alternative parser generators producing both top-down (LL) and bottom-up parsers of the LR variety, including SLR(1) and LALR(1) parsers, thus allowing users to experiment with different parsing strategies and to get a “feel” for their relative pros and cons. When designing Jaccie, the main emphasis was on two ergonomic goals that we considered important for educational software. Firstly, give user control over the program and not vice versa, e.g., our parsers (and other components) can be directed to go step-by-step forwards or backwards or to leap to some point in the input indicated by the users. Secondly, overcome the sometimes severe size limitations of computer displays by offering the same information in multiple representations that complement each other and by dividing information in smaller chunks that can be traversed in a meaningful way.In this paper, after outlining the architecture of Jaccie, we discuss some of its technical and ergonomic aspects in detail, give a brief introduction into the use of Jaccie and its documentation, show an example application done with Jaccie, and finally discuss related work and future plans.