LR Parsing Research Articles

Spectrum-based rule- and item-level localization of faults in context-free grammars

We describe and evaluate spectrum-based methods aimed at finding faults in context-free grammars. In their basic form, they take as input a test suite and a parser for the grammar that is modified to collect grammar spectra (i.e., the sets of grammar elements used in attempts to parse the individual test cases), and return as output a ranked list of suspicious elements. We define grammar spectra suitable for localizing faults on the level of the grammar rules (i.e., rule spectra) and the rules’ individual symbols (i.e., item spectra), respectively. We show how both types of grammar spectra can be collected by both LL and LR parsers, and how the JavaCC, ANTLR, and CUP parser generators can be modified and used to automate the collection of the grammar spectra. We also show how grammar spectra can be synthesized directly from test cases derived from a grammar, and how such synthetic spectra can be used to localize differences between a grammar and a black-box system under test.We first evaluate our approach over a large number of medium-sized single fault grammars, which we constructed by fault seeding from a common origin grammar. At the rule level, it ranks the rules containing the seeded faults within the top five rules in about 40%–70% of the cases, depending on the applied parsing technique, test suite, and ranking metric, and pinpoints them (i.e., correctly identifies them as unique most suspicious rule) in about 10%–30% of the cases, with significantly better results for the synthetic spectra. At the item level, our approach remains remarkably effective despite the larger number of possible locations, provided it is coupled with a simple tie-breaking strategy that prefers items with the right-most designated position over other items from the same rules in a tie. It typically ranks the seeded faults within the top five positions in about 30%–60% of the cases, and pinpoints them in about 15%–40% of the cases. This specialized item-level localization also significantly outperforms a simplistic extension of the rule-level localization, where all positions within a rule are given the same score.We further evaluate our approach over grammars that contain real faults. We show that an iterative method can be used to localize and manually remove one by one multiple faults in grammars submitted by students enrolled in various compiler engineering courses; in most iterations, the top-ranked rule already contains an error, and no error is ranked outside the top five ranked rules. We finally apply our approach to a large open-source SQLite grammar and show where this version deviates from the language accepted by the actual SQLite system.

A text-based syntax completion method using LR parsing and its evaluation

A text-based syntax completion method using LR parsing and its evaluation

RNGSGLR: Generalization of the Context-Aware Scanning Architecture for All Character-Level Context-Free Languages

The limitations of traditional parsing architecture are well known. Even when paired with parsing methods that accept all context-free grammars (CFGs), the resulting combination for any given CFG accepts only a limited subset of corresponding character-level context-free languages (CFL). We present a novel scanner-based architecture that for any given CFG accepts all corresponding character-level CFLs. It can directly parse all possible specifications consisting of a grammar and regular definitions. The architecture is based on right-nulled generalized LR (RNGLR) parsing and is a generalization of the context-aware scanning architecture. Our architecture does not require any disambiguation rules to resolve lexical conflicts, it conceptually has an unbounded parser and scanner lookahead and it is streaming. The added robustness and flexibility allow for easier grammar development and modification.

Parglare: A LR/GLR parser for Python

• Deterministic LR parsing. • Generalized GLR parsing. • Integrated scanning. • Declarative disambiguation. • GLR trace visualization. Parglare is a Python parsing library that implements deterministic LR and its generalized extension GLR algorithms. Parglare strives to be easy to use by providing rich error messages, visualization, a CLI tool for grammar development, and good documentation. The same grammar format is used for both algorithms. It is easy to choose either LR parsing if performance is more important or GLR in case a grammar cannot fit into the constraints of deterministic LR parsing and a more powerful parsing is needed. Parglare has been used in data extraction from various textual formats, analysis of legacy source code, and developing and teaching DSL development. The GitHub repository of the project contains multiple examples and a comprehensive functionality and performance test suite.

A Drop-in Replacement for LR(1) Table-Driven Parsing

This paper presents a construction method for a deterministic one-symbol look-ahead LR parser which allows non-terminals in the parser look-ahead. This effectively relaxes the requirement of parsing the reverse of the right-most derivation of a string/sentence. This is achieved by replacing the deterministic push down automata of LR parsing by a two-stack automata. The class of grammars accepted by the two-stack parser properly contains the LR( k ) grammars. Since the modification to the table-driven LR parsing process is relatively minor and mostly impacts the creation of the goto and action tables, a parser modified to adopt the two-stack process should be comparable in size and performance to LR parsers.

Morbig: A Static parser for POSIX shell

The POSIX shell language defies conventional wisdom of compiler construction on several levels: The shell language was not designed for static parsing, but with an intertwining of syntactic analysis and execution by expansion in mind. Token recognition cannot be specified by regular expressions and lexical analysis depends on the parsing context and the evaluation context. Besides, the unorthodox design choices of the shell language fit badly in the usual specification languages used to describe other programming languages. This makes the standard usage of Lex and Yacc as a pipeline inadequate for the implementation of a parser for POSIX shell. The existing implementations of shell parsers are complex and use low-level character-level parsing code that is difficult to relate to the POSIX specification. We find it hard to trust such parsers, especially when using them for writing automatic verification tools for shell scripts. This paper offers an overview of the technical difficulties related to the syntactic analysis of the POSIX shell language. It also describes how we have resolved these difficulties using advanced parsing techniques (namely speculative parsing, parser state introspection, context-dependent lexical analysis and longest-prefix parsing) while keeping the implementation at a sufficiently high level of abstraction so that experts can check that the POSIX standard is respected. The resulting tool, called Morbig, is an open-source static parser for a well-defined and realistic subset of the POSIX shell language. Its implementation crucially relies on the purity and incrementality of LR(1) parsers generated by Menhir, a parser generator for OCaml.

Run-time conflict detection in visual language parsing

Abstract The widespread use of diagrammatic languages has motivated the need for grammar-based tools to support designers in the definition and implementation of graphical environments. The effective use of such systems requires efficient parsing techniques. In previous years, many approaches have been devised and one of them is based on the extension of the LR parsing methodology from traditional string languages to visual languages generated by the so-called Positional Grammars. The technique, named pLR parsing, is a deterministic technique able to automatically generate an efficient LR-like parser for many interesting and practical visual languages such as flowcharts, statecharts, automata, UML notations, and others. However, differently from the traditional string case, there exist particular positional grammars whose generated pLR parsing table does not present conflicts but are, however, not pLR parsable because, due to their two-dimensional nature, they can produce run-time conflicts. In this paper, we introduce an algorithm that statically verifies the pLR parsability of a positional grammar by detecting whether the pLR parser built on it may produce run-time conflicts. Such an algorithm is of importance during the implementation of a diagrammatic language since it is able to inform the language designer about the pLR parsability of the designed grammar and to detect the (possible) places where the grammar should be modified.

Speeding up Generalized PSR Parsers by Memoization Techniques

Predictive shift-reduce (PSR) parsing for hyperedge replacement (HR) grammars is very efficient, but restricted to a subclass of unambiguous HR grammars. To overcome this restriction, we have recently extended PSR parsing to generalized PSR (GPSR) parsing along the lines of Tomita-style generalized LR parsing. Unfortunately, GPSR parsers turned out to be too inefficient without manual tuning. This paper proposes to use memoization techniques to speed up GPSR parsers without any need of manual tuning, and which has been realized within the graph parser distiller Grappa. We present running time measurements for some example languages; they show a significant speed up by some orders of magnitude when parsing valid graphs. But memoization techniques do not help when parsing invalid graphs or if all parses of an ambiguous input graph shall be determined.

Generating a fluent API with syntax checking from an LR grammar

This paper proposes a fluent API generator for Scala, Haskell, and C++. It receives a grammar definition and generates a code skeleton of the library in the host programming language. The generated library is accessed through a chain of method calls; this style of API is called a fluent API. The library uses the host-language type checker to detect an invalid chain of method calls. Each method call is regarded as a lexical token in the embedded domain specific language implemented by that library. A sequence of the lexical tokens is checked and, if the sequence is not acceptable by the grammar, a type error is reported during compilation time. A contribution of this paper is to present an algorithm for generating the code-skeleton for a fluent API that reports a type error when a chain of method calls to the library does not match the given LR grammar. Our algorithm works in Scala, Haskell, and C++. To encode LR parsing, it uses the method/function overloading available in those languages. It does not need an advanced type system, or exponential compilation time or memory consumption. This paper also presents our implementation of the proposed generator.

Cascaded Finite-State Chunk Parsing for Hindi Language

The Cascaded Finite-state Chunk Parsing for Hindi Language (CFC) is a deterministic bottom-up version based on an LR parser that employs a best-first search. It uses finite-state cascade as a sequence of strata. Each stratum is defined by a category and a set of regular-expression patterns for recognizing phrases. Phrases at one level are built on phrases at the previous level. The final chunk structure is a multi-level sequence to span the entire text as single parse tree. Our research is presented with a design, implementation steps and result analysis. Though, we present a case study for an Indian language i.e. Hindi, it is language independent approach and extendible for further scope.

PAVT: a tool to visualize and teach parsing algorithms

A parsing algorithm visualizer is a tool that visualizes the construction of a parser for a given context-free grammar and then illustrates the use of that parser to parse a given string. Parsing algorithm visualizers are used to teach the course on compiler construction which in invariably included in all undergraduate computer science curricula. This paper presents a new parsing algorithm visualizer that can visualize six parsing algorithms, viz. predictive parsing, simple LR parsing, canonical LR parsing, look-ahead LR parsing, Earley parsing and CYK parsing. The tool logically explains the process of parsing showing the calculations involved in each step. The output of the tool has been structured to maximize the learning outcomes and contains important constructs like FIRST and FOLLOW sets, item sets, parsing table, parse tree and leftmost or rightmost derivation depending on the algorithm being visualized. The tool has been used to teach the course on compiler construction at both undergraduate and graduate levels. An overall positive feedback was received from the students with 89% of them saying that the tool helped them in understanding the parsing algorithms. The tool is capable of visualizing multiple parsing algorithms and 88% students used it to compare the algorithms.

Fast deterministic parsers for transition networks

Extended BNF grammars (EBNF) allow regular expressions in the right parts of their rules. They are widely used to define languages, and can be represented by recursive Transition Networks (TN) consisting of a set of finite-state machines. We present a novel direct construction of efficient shift-reduce ELR(1) parsers for TNs. We show that such a parser works deterministically if the TN is free from the classical shift-reduce and reduce–reduce conflicts of the LR(1) parsers, and from a new conflict type called convergence conflict. Such a novel condition for determinism is proved correct and is more general than those proposed in the past for EBNF grammars or TNs. Such ELR(1) parsers perform fewer shift moves than the equivalent LR(1) parsers. A simple optimization of the reduction moves is described.

A Simple, Possibly Correct LR Parser for C11

The syntax of the C programming language is described in the C11 standard by an ambiguous context-free grammar, accompanied with English prose that describes the concept of “scope” and indicates how certain ambiguous code fragments should be interpreted. Based on these elements, the problem of implementing a compliant C11 parser is not entirely trivial. We review the main sources of difficulty and describe a relatively simple solution to the problem. Our solution employs the well-known technique of combining an LALR(1) parser with a “lexical feedback” mechanism. It draws on folklore knowledge and adds several original aspects, including a twist on lexical feedback that allows a smooth interaction with lookahead; a simplified and powerful treatment of scopes; and a few amendments in the grammar. Although not formally verified, our parser avoids several pitfalls that other implementations have fallen prey to. We believe that its simplicity, its mostly declarative nature, and its high similarity with the C11 grammar are strong informal arguments in favor of its correctness. Our parser is accompanied with a small suite of “tricky” C11 programs. We hope that it may serve as a reference or a starting point in the implementation of compilers and analysis tools.

On different LL and LR parsers used in LLLR parsing

As described in Slivnik (2016), LLLR parsing is a method that parses as much of its input string as possible using the backbone SLL(k) parser and uses small embedded canonical left LR(k) parsers to resolve LL conflicts. Once the LL conflict is resolved, the embedded parser produces the left parse of the substring it has just parsed and passes the control back to the backbone parser together with the information about how the backbone parser should realign its stack as a part of the input has been read by the embedded parser. The LLLR parser produces the left parse of the input string without any backtracking and, if used for a syntax-directed translation, it evaluates semantic actions using the same top-down strategy as the canonical LL(k) parser. In this paper, a more general approach towards LLLR parsing is presented as it is described how any kind of canonical LL(k) or LA(k)LL(k′) parser can be used as the backbone parser and how different kinds of embedded canonical left LR(k) or left LA(k)LR(k′) parsers can be used for LL conflict resolution.

LR 문법에 대한 단일상태파싱오토마톤의 적용

LR 문법에 대한 단일상태파싱오토마톤의 적용

LR Parsing for LCFRS

LR parsing is a popular parsing strategy for variants of Context-Free Grammar (CFG). It has also been used for mildly context-sensitive formalisms, such as Tree-Adjoining Grammar. In this paper, we present the first LR-style parsing algorithm for Linear Context-Free Rewriting Systems (LCFRS), a mildly context-sensitive extension of CFG which has received considerable attention in the last years in the context of natural language processing.

Generalized LR Parsing Algorithm for Grammars with One-Sided Contexts

The Generalized LR parsing algorithm for context-free grammars is notable for having a decent worst-case running time (cubic in the length of the input string, if implemented efficiently), as well as much better performance on “good” grammars. This paper extends the Generalized LR algorithm to the case of “grammars with left contexts” (M. Barash, A. Okhotin, “An extension of context-free grammars with one-sided context specifications”, Inform. Comput., 2014), which augment the context-free grammars with special operators for referring to the left context of the current substring, along with a conjunction operator (as in conjunctive grammars) for combining syntactical conditions. All usual components of the LR algorithm, such as the parsing table, shift and reduce actions, etc., are extended to handle the context operators. The resulting algorithm is applicable to any grammar with left contexts and has the same worst-case cubic-time performance as in the case of context-free grammars.

Parallel parsing made practical

The property of local parsability allows to parse inputs through inspecting only a bounded-length string around the current token. This in turn enables the construction of a scalable, data-parallel parsing algorithm, which is presented in this work. Such an algorithm is easily amenable to be automatically generated via a parser generator tool, which was realized, and is also presented in the following. Furthermore, to complete the framework of a parallel input analysis, a parallel scanner can also combined with the parser. To prove the practicality of a parallel lexing and parsing approach, we report the results of the adaptation of JSON and Lua to a form fit for parallel parsing (i.e. an operator-precedence grammar) through simple grammar changes and scanning transformations. The approach is validated with performance figures from both high performance and embedded multicore platforms, obtained analyzing real-world inputs as a test-bench. The results show that our approach matches or dominates the performances of production-grade LR parsers in sequential execution, and achieves significant speedups and good scaling on multi-core machines. The work is concluded by a broad and critical survey of the past work on parallel parsing and future directions on the integration with semantic analysis and incremental parsing.

Syntax Analysis: The Left-Most-Derivation-and-Reduction Trees and its Compare with the LR Parsing Methods

A new syntax analysis method is presented, called as “left-most-derivation-and-reduction trees.” With the new way, the syntax analysis is carried out under the direction of the syntax trees, while in the traditional syntax analysis, the syntax tree is obtained when the analysis has finished. Furthermore, through the point of view of the “left-most-derivation-and-reduction trees”, the properties of the LR parsing method is newly surveyed and explained in an easier, simpler, and more directed way. Finally, the new way is expected to get further development in future.

Indentation-sensitive parsing for Parsec

Several popular languages including Haskell and Python use the indentation and layout of code as an essential part of their syntax. In the past, implementations of these languages used ad hoc techniques to implement layout. Recent work has shown that a simple extension to context-free grammars can replace these ad hoc techniques and provide both formal foundations and efficient parsing algorithms for indentation sensitivity. However, that previous work is limited to bottom-up, LR($k$) parsing, and many combinator-based parsing frameworks including Parsec use top-down algorithms that are outside its scope. This paper remedies this by showing how to add indentation sensitivity to parsing frameworks like Parsec. It explores both the formal semantics of and efficient algorithms for indentation sensitivity. It derives a Parsec-based library for indentation-sensitive parsing and presents benchmarks on a real-world language that show its efficiency and practicality.