LR Parsing Research Articles

State Merging in LR Parser under Count based Reduction

An LR parser shows error only during scanning input symbol. Error is never shown during the reduction of a handle (substring of stack) into nonterminal. It is because a symbol is put into the stack only when it is guaranteed to be the correct one. If the method of reduction of a handle is known then errors can also be shown during reduction. Hence a wrong symbol can be shifted on the stack and error can be detected during reduce operation. It may permit the merging of few states. The simplest type of reduction scheme is to remove few symbols (the number of symbols equal to the length of the handle) from the top of the stack and push the corresponding nonterminal on the stack. In this paper, a state merging scheme is proposed under this method of reduction. General Terms Your general terms must be any term which can be used for general classification of the submitted material such as Pattern Recognition, Security, Algorithms et. al.

Principles of LR Parsing

Principles of LR Parsing

An LR Parser with Less States

This paper presents a new construction method of LR parser which is based on the theory of deterministic finite automata. The deterministic infinite automata are simplified for obtaining the LR parsing table with fewer states by merging the equivalent states. The new construction method has the same reorganization power as LALR parser but avoids the “reduce/reduce” conflict in LALR method. The experiment results show that the number of states in parsing table can be reduced greatly if the proposed method is used.

SuperC

C tools, such as source browsers, bug finders, and automated refactorings, need to process two languages: C itself and the preprocessor. The latter improves expressivity through file includes, macros, and static conditionals. But it operates only on tokens, making it hard to even parse both languages. This paper presents a complete, performant solution to this problem. First, a configuration-preserving preprocessor resolves includes and macros yet leaves static conditionals intact, thus preserving a program's variability. To ensure completeness, we analyze all interactions between preprocessor features and identify techniques for correctly handling them. Second, a configuration-preserving parser generates a well-formed AST with static choice nodes for conditionals. It forks new subparsers when encountering static conditionals and merges them again after the conditionals. To ensure performance, we present a simple algorithm for table-driven Fork-Merge LR parsing and four novel optimizations. We demonstrate the effectiveness of our approach on the x86 Linux kernel.

LL conflict resolution using the embedded left LR parser

A method for resolving LL(k) conflicts using small LR(k) parsers (called embedded left LR(k) parsers) is described. An embedded left LR(k) parser is capable of (a) producing the prefix of the left parse of the input string and (b) stopping not on the end-of-file marker but on any string from the set of lookahead strings fixed at the parser generation time. The conditions regarding the termination of the embedded left LR(k) parser if used within LL(k) (and similar) parsers are defined and examined in-depth. It is proved that an LL(k) parser augmented with a set of embedded left LR(k) parsers can parse any deterministic context-free grammar in the same asymptotic time as LR(k) parser. As the embedded left LR(k) parser produces the prefix of the left parse, the LL(k) parser augmented with embedded left LR(k) parsers still produces the left parse and the compiler writer does not need to bother with different parsing strategies during the compiler implementation.

LR(K) Parser Construction Using Bottom-up Formal Analysis

Design and construction of an error-free compiler is a difficult and challenging process. The main functionality of a compiler is to translate a source code to an executable machine code correctly and efficiently. In formal verification of software, semantics of a language has more meanings than the syntax. It means source program verification does not give guarantee the generated code is correct. This is because the compiler may lead to an incorrect target program due to bugs in itself. It means verification of a compiler is much more important than verification of a source program. In this paper, we present a new approach by linking context-free grammar and Z notation to construct LR(K) parser. This has several advantages because correctness of the compiler depends on describing rules that must be written in formal languages. First, we have defined grammar then language derivation procedure is given using right-most derivations. Verification of a given language is done by recursive procedures based on the words. Ambiguity of a language is checked and verified. The specification is analyzed and validated using Z/Eves tool. Formal proofs are presented using powerful techniques of reduction and rewriting available in Z/Eves.

LR(1) parser generation system

The LR(1) Parser Generation System generates full LR(1) parsers that are comparable in speed and size to those generated by LALR(1) parser generators, such as yacc [5]. In addition to the inherent advantages of full LR(1) parsing, it contains a number of novel features. This paper discusses three of them in detail: an LR(1) grammar specified automatic error recovery algorithm, oracles, and generic tokens. The error recovery algorithm depends on the fact that full LR(1) parse tables preserve context. Oracles are pieces of code that are defined in a grammar and that are executed between the scanner and parser. They are used to resolve token ambiguities, including semantic ones. Generic tokens are used to replace syntactically identical tokens with a single token, which is, in effect, a variable representing a set of tokens.

Solving difficult LR parsing conflicts by postponing them

Though yacc-like LR parser generators provide ways to solve shift-reduce conflicts using token precedences, no mechanisms are provided for the resolution of difficult shift-reduce or reduce-reduce conflicts. To solve this kind of conflicts the language designer has to modify the grammar. All the solutions for dealing with these difficult conflicts branch at each alternative, leading to the exploration of the whole search tree. These strategies differ in the way the tree is explored: GLR, Backtracking LR, Backtracking LR with priorities, etc. This paper explores an entirely different path: to extend the yacc conflict resolution sublanguage with new constructs allowing the programmers to explicit the way the conflict must be solved. These extensions supply ways to resolve any kind of conflicts, including those that can not be solved using static precedences. The method makes also feasible the parsing of grammars whose ambiguity must be solved in terms of the semantic context. Besides, it brings to LR-parsing a common LL-parsing feature: the advantage of providing full control over the specific trees the user wants to build.

Multithreaded parsing for predicting RNA secondary structures

Many computational approaches have been developed for modelling and analysing the RNA secondary structure. These approaches are based on diverse methods such as grammars, dynamic programming, matching and evolutionary algorithms. This paper proposes a new parsing algorithm for the prediction of RNA secondary structures. The proposed algorithm is based on the shift-reduce LR parsing algorithm for programming languages. It has two main contributions: it extends the LR parsing algorithm by using a Stochastic Context-Free Grammar (SCFG) instead of Context-Free Grammar (CFG) for parsing RNA secondary structures; it extends the LR parsing algorithm by using a multithreaded approach to handle the LR parsing conflicts resulting from the use of ambiguous grammars.

The IELR(1) algorithm for generating minimal LR(1) parser tables for non-LR(1) grammars with conflict resolution

There has been a recent effort in the literature to reconsider grammar-dependent software development from an engineering point of view. As part of that effort, we examine a deficiency in the state of the art of practical LR parser table generation. Specifically, LALR sometimes generates parser tables that do not accept the full language that the grammar developer expects, but canonical LR is too inefficient to be practical particularly during grammar development. In response, many researchers have attempted to develop minimal LR parser table generation algorithms. In this paper, we demonstrate that a well known algorithm described by David Pager and implemented in Menhir, the most robust minimal LR(1) implementation we have discovered, does not always achieve the full power of canonical LR(1) when the given grammar is non-LR(1) coupled with a specification for resolving conflicts. We also detail an original minimal LR(1) algorithm, IELR(1) (Inadequacy Elimination LR(1)), which we have implemented as an extension of GNU Bison and which does not exhibit this deficiency. Using our implementation, we demonstrate the relevance of this deficiency for several real-world parser specifications, and we demonstrate the feasibility of IELR(1). Finally, we demonstrate that, if canonical LR(1) were employed instead, grammar development would be severely impeded regardless of the power of the computer hardware.

Verifiable composition of deterministic grammars

There is an increasing interest in extensible languages, (domain-specific) language extensions, and mechanisms for their specification and implementation. One challenge is to develop tools that allow non-expert programmers to add an eclectic set of language extensions to a host language. We describe mechanisms for composing and analyzing concrete syntax specifications of a host language and extensions to it. These specifications consist of context-free grammars with each terminal symbol mapped to a regular expression, from which a slightly-modified LR parser and context-aware scanner are generated. Traditionally, conflicts are detected when a parser is generated from the composed grammar, but this comes too late since it is the non-expert programmer directing the composition of independently developed extensions with the host language. The primary contribution of this paper is a modular analysis that is performed independently by each extension designer on her extension (composed alone with the host language). If each extension passes this modular analysis, then the language composed later by the programmer will compile with no conflicts or lexical ambiguities. Thus, extension writers can verify that their extension will safely compose with others and, if not, fix the specification so that it will. This is possible due to the context-aware scanner's lexical disambiguation and a set of reasonable restrictions limiting the constructs that can be introduced by an extension. The restrictions ensure that the parse table states can be partitioned so that each state can be attributed to the host language or a single extension.

Component-based LR parsing

A language implementation with proper compositionality enables a compiler developer to divide-and-conquer the complexity of building a large language by constructing a set of smaller languages. Ideally, these small language implementations should be independent of each other such that they can be designed, implemented and debugged individually, and later be reused in different applications (e.g., building domain-specific languages). However, the language composition offered by several existing parser generators resides at the grammar level, which means all the grammar modules need to be composed together and all corresponding ambiguities have to be resolved before generating a single parser for the language. This produces tight coupling between grammar modules, which harms information hiding and affects independent development of language features. To address this problem, we have developed a novel parsing algorithm that we call Component-based LR (CLR) parsing, which provides code-level compositionality for language development by producing a separate parser for each grammar component. In addition to shift and reduce actions, the algorithm extends general LR parsing by introducing switch and return actions to empower the parsing action to jump from one parser to another. Our experimental evaluation demonstrates that CLR increases the comprehensibility, reusability, changeability and independent development ability of the language implementation. Moreover, the loose coupling among parser components enables CLR to describe grammars that contain LR parsing conflicts or require ambiguous token definitions, such as island grammars and embedded languages.

IDS alerts correlation using grammar-based approach

Intrusion Detection System (IDS) is a security technology that attempts to identify intrusions. Defending against multi-step intrusions which prepare for each other is a challenging task. In this paper, we propose a novel approach to alert post-processing and correlation, the Alerts Parser. Different from most other alert correlation methods, our approach treats the alerts as tokens and uses modified version of the LR parser to generate parse trees representing the scenarii in the alerts. An Attribute Context-Free Grammar (ACF-grammar) is used for representing the multi-step attacks. Attack scenarii information and prerequisites/consequences knowledge are included together in the ACF-grammar enhancing the correlation results. The modified LR parser depends on these ACF-grammars to generate parse trees. The experiments were performed on two different sets of network traffic traces, using different open-source and commercial IDS sensors. The discovered scenarii are represented by Correlation Graphs (CGs). The experimental results show that Alerts Parser can work in parallel, effectively correlate related alerts with low false correlation rate, uncover the attack strategies, and generate concise CGs.

A tool for teaching LL and LR parsing algorithms

In this paper we present a tool to assist in teaching top-down and bottom-up analysis algorithms. The tool provides simulation for the following analysis algorithms: LL, SLR, LALR and LR. During the simulation the student can simultaneously see the pending input, the analysis stack and tables, the generated output and the parse tree.

Automatic recursion engineering of reduction incorporated parsers

Reduction Incorporated (RI) parsers deliver high performance by suppressing the stack activity except for those rules that generate embedded recursion. Automaton constructions for RI parsing have been presented by Aycock and Horspool [J. Aycock, N. Horspool, Faster generalised LR parsing, in: Compiler Construction, 8th Intnl. Conf, CC’99, in: Lecture Notes in Computer Science, vol. 1575, Springer-Verlag, 1999, pp. 32–46] and by Scott and Johnstone [A. Johnstone, E. Scott, Generalised regular parsers, in: G. Hedin (Ed.), Compiler Construction, 12th Intnl. Conf, CC’03, in: Lecture Notes in Computer Science, vol. 2622, Springer-Verlag, Berlin, 2003, pp. 232–246] but both can yield very large tables. An unusual aspect of the RI automaton is that the degree of stack activity suppression can be varied in a fine-grained way by choosing different grammar terminalisation sets, and this provides a large family of potential RI automata for real programming languages, some of which have manageable table size but still show high performance. In this paper we describe automatic construction of minimal grammar terminalisation sets, giving examples drawn from ANSI-C, Cobol and Pascal; we describe the use of profiling to inform the choice of RI automaton; we investigate the use of RI parsers for scannerless parsing; and we explain some of the phenomena that influence the time/space trade-off for RI parsers.

LR Formalisms as Abstract Interpretations of Grammar Semantics

The concept of LR(k) validity is represented as an abstract interpretation of a refinement of the derivation semantics of a given grammar. Also the algorithm of LR(k) parsing is represented as an abstract interpretation of the refined semantics. Such representations of LR formalisms provide us with more intuitive and easier means by which to understand LR parsing.

Right nulled GLR parsers

The right nulled generalized LR parsing algorithm is a new generalization of LR parsing which provides an elegant correction to, and extension of, Tomita's GLR methods whereby we extend the notion of a reduction in a shift-reduce parser to include right nulled items. The result is a parsing technique which runs in linear time on LR(1) grammars and whose performance degrades gracefully to a polynomial bound in the presence of nonLR(1) rules. Compared to other GLR-based techniques, our algorithm is simpler and faster.

GENERALIZED LR PARSING ALGORITHM FOR BOOLEAN GRAMMARS

The generalized LR parsing algorithm for context-free grammars is extended for the case of Boolean grammars, which are a generalization of the context-free grammars with logical connectives added to the formalism of rules. In addition to the standard LR operations, Shift and Reduce, the new algorithm uses a third operation called Invalidate, which reverses a previously made reduction. This operation makes the mathematical justification of the algorithm significantly different from its prototype. On the other hand, the changes in the implementation are not very substantial, and the algorithm still works in time O(n4).

Probabilistic parsing strategies

We present new results on the relation between purely symbolic context-free parsing strategies and their probabilistic counterparts. Such parsing strategies are seen as constructions of push-down devices from grammars. We show that preservation of probability distribution is possible under two conditions, viz. the correct-prefix property and the property of strong predictiveness. These results generalize existing results in the literature that were obtained by considering parsing strategies in isolation. From our general results, we also derive negative results on so-called generalized LR parsing.

States merging in LR parser

Here a scheme is presented, which reduces the size of an LR parser. It is accomplished by merging two states, which do not perform differently on the same input symbol or non-terminal. This state merging has following advantages:1. The reduction in number of states is greater than the reduction in an LALR parser.2. The power of parser is not reduced as it would be in an LALR parser.The above merging scheme has following disadvantages:1. Even erroneous inputs are accepted. However, it is stopped by changing the notion of acceptance.2. Errors are report late and in a misleading manner.3. Parsing takes more time because string matching is done during reduction.