Deterministic Regular Expressions Research Articles

Deterministic regular expressions with back-references

Most modern libraries for regular expression matching allow back-references (i.e., repetition operators) that substantially increase expressive power, but also lead to intractability. In order to find a better balance between expressiveness and tractability, we combine these with the notion of determinism for regular expressions used in XML DTDs and XML Schema. This includes the definition of a suitable automaton model, and a generalization of the Glushkov construction. We demonstrate that, compared to their non-deterministic superclass, these deterministic regular expressions with back-references have desirable algorithmic properties (i.e., efficiently solvable membership problem and some decidable problems in static analysis), while, at the same time, their expressive power exceeds that of deterministic regular expressions without back-references.

Towards an Effective Syntax and a Generator for Deterministic Standard Regular Expressions

Abstract Deterministic regular expressions are a core part of XML Schema and used in other applications. But unlike regular expressions, deterministic regular expressions do not have a simple syntax, instead they are defined in a semantic manner. Moreover, not every regular expression can be rewritten to an equivalent deterministic regular expression. These properties of deterministic regular expressions put a burden on the user to develop XML Schema Definitions and to use deterministic regular expressions. In this paper, we propose a syntax for deterministic standard regular expressions (DREGs), and prove that the syntax of DREGs is context-free. Based on the context-free grammars for DREGs, we further design a generator for DREGs, which can generate DREGs randomly, and be used in applications associated with DREGs, e.g. benchmarking a validator for DTD or XML Schema, and inclusion checking of DTD and XML Schema. Experimental results demonstrate the efficiency and usefulness of the generator.

Complexity of universality and related problems for partially ordered NFAs

Partially ordered NFAs (poNFAs) are NFAs where cycles occur only in the form of self-loops. A poNFA is universal if it accepts all words over its alphabet. Deciding universality is PSpace -complete for poNFAs. We show that this remains true when restricting to fixed alphabets. This is nontrivial since standard encodings of symbols in, e.g., binary can turn self-loops into longer cycles. A lower coNP -complete complexity bound is obtained if all self-loops in the poNFA are deterministic. We find that such restricted poNFAs (rpoNFAs) characterize R-trivial languages, and establish the complexity of deciding if the language of an NFA is R-trivial. The limitation to fixed alphabets is essential even in the restricted case: deciding universality of rpoNFAs with unbounded alphabets is PSpace -complete. Consequently, we obtain the complexity results for inclusion and equivalence problems. Finally, we show that the languages of rpoNFAs are definable by deterministic (one-unambiguous) regular expressions.

Efficient testing and matching of deterministic regular expressions

A linear time algorithm is presented for testing determinism of a regular expression. It is shown that an input word of length n can be matched against a deterministic regular expression of length m in time O(m+nlog⁡log⁡m). If the deterministic regular expression has bounded depth of alternating union and concatenation operators, then matching can be performed in time O(m+n). These results extend to regular expressions containing numerical occurrence indicators.

Deciding definability by deterministic regular expressions

We investigate the complexity of deciding whether a given regular language can be expressed by a deterministic regular expression. Our main technical result shows that deciding if the language of a given regular expression (or, non-deterministic finite automaton) can be defined by a deterministic regular expression is PSPACE-complete. The problem becomes EXPSPACE-complete if the input language is represented as a regular expression with counters and NL-hard if the input language is given by a minimal deterministic finite automaton.

On the hierarchy of generalizations of one-unambiguous regular languages

A regular language is lookahead (resp. block) deterministic if it is specified by a lookahead (resp. block) deterministic regular expression. These two subclasses of regular languages have been respectively introduced by Han and Wood for lookahead determinism and by Giammarresi et al. for block determinism, as a possible extension of one-unambiguous languages defined and characterized by Brüggemann-Klein and Wood.In this paper, we study the hierarchies and inclusion links of these families. We first show that each block deterministic language is the alphabetical image of some one-unambiguous language. Moreover, we show that deciding the block determinism of a regular language from its minimal DFA does not only require state elimination. Han and Wood state that there is a proper hierarchy in block deterministic languages, but prove it using this erroneous requirement. However, we prove their statement by giving our own parametrized family. We also prove that there is a proper hierarchy in lookahead deterministic languages by studying particular properties of unary regular expressions. Finally, using our valid results, we confirm that the family of block deterministic languages is strictly included in the one of lookahead deterministic languages by showing that any block deterministic unary language is one-unambiguous.

Closure properties and descriptional complexity of deterministic regular expressions

We study the descriptional complexity of regular languages that are definable by deterministic regular expressions, i.e., we examine worst-case blow-ups in size when translating between different representations for such languages. As representations of languages, we consider regular expressions, deterministic regular expressions, and deterministic finite automata. Our results show that exponential blow-ups between these representations cannot be avoided. Furthermore, we study the descriptional complexity of these representations when applying boolean operations. Here, we start by investigating the closure properties of such languages under various language-theoretic operations such as union, intersection, concatenation, Kleene star, and reversal. Our results show that languages that are definable by deterministic regular expressions are not closed under any of these operations. Finally, we show that for all these operations except the Kleene star an exponential blow-up in the size of deterministic regular expressions cannot be avoided.

Regular Expressions with Counting: Weak versus Strong Determinism

We study deterministic regular expressions extended with the counting operator. There exist two notions of determinism, strong and weak determinism, which are equally expressive for standard regular expressions. This, however, changes dramatically in the presence of counting. In particular, we show that weakly deterministic expressions with counting are exponentially more succinct and strictly more expressive than strongly deterministic ones, even though they still do not capture all regular languages. In addition, we present a finite automaton model with counters, study its properties, and investigate the natural extension of the Glushkov construction translating expressions with counting into such counting automata. This translation yields a deterministic automaton if and only if the expression is strongly deterministic. These results then also allow us to derive upper bounds for decision problems for strongly deterministic expressions with counting.

Learning Deterministic Regular Expressions for the Inference of Schemas from XML Data

Inferring an appropriate DTD or XML Schema Definition (XSD) for a given collection of XML documents essentially reduces to learning deterministic regular expressions from sets of positive example words. Unfortunately, there is no algorithm capable of learning the complete class of deterministic regular expressions from positive examples only, as we will show. The regular expressions occurring in practical DTDs and XSDs, however, are such that every alphabet symbol occurs only a small number of times. As such, in practice it suffices to learn the subclass of deterministic regular expressions in which each alphabet symbol occurs at most k times, for some small k . We refer to such expressions as k -occurrence regular expressions ( k -OREs for short). Motivated by this observation, we provide a probabilistic algorithm that learns k -OREs for increasing values of k, and selects the deterministic one that best describes the sample based on a Minimum Description Length argument. The effectiveness of the method is empirically validated both on real world and synthetic data. Furthermore, the method is shown to be conservative over the simpler classes of expressions considered in previous work.

The validation of SGML content models

The Standard Generalized Markup Language (SGML) is an ISO standard that provides a syntactic meta-language for the definition of textual markup systems, which are used to indicate the structure of documents so that they can be electronically typeset, searched, and communicated. We address only one problem raised by the standard, namely: in SGML, the right-hand sides of context-free productions are regular expressions, called content models, that are restricted to be what the standard calls “unambiguous”, but what is more appropriately called deterministic. We solve the problem of how to define determinism precisely, how to recognize deterministic regular expressions efficiently, and how to recognize deterministic regular languages. Any SGML parser must check that a given document grammar conforms to the standard; that is, it must validate it. Hence, our results are an important step in the clarification of the standard and in the efficient implementation of an SGML parser for SGML document grammars.

Regular expressions into finite automata

It is a well-established fact that each regular expression can be transformed into a nondeterministic finite automaton (NFA) with or without ε-transitions, and all authors seem to provide their own variant of the construction. Of these, Berry and Sethi (1986) have shown that the construction of an ε-free NFA due to Glushkov (1961) is a natural representation of the regular expression because it can be described in terms of Brzozowski derivatives (Brzozowski 1964) of the expression. Moreover, the Glushkov construction also plays a significant role in the document processing area: The SGML standard (ISO 8879 1986), now widely adopted by publishing houses and government agencies for the syntactic specification of textual markup systems, uses deterministic regular expressions, i.e. expressions whose Glushkov automaton is deterministic, as a description language for document types. In this paper, we first show that the Glushkov automaton can be constructed in a time quadratic in the size of the expression, and that this is worst-case optimal. For deterministic expressions, our algorithm has even linear run time. This improves on the cubic time methods suggested in the literature (Book 1971; Aho 1986; Berry and Sethi 1986). A major step of the algorithm consists in bringing the expression into what we call star normal form. This concept is also useful for characterizing the relationship between two types of unambiguity that have been studied in the literature. Namely, we show that, modulo a technical condition, an expression is strongly unambiguous (Sippu and Soisalon-Soininen 1988) if and only if it is weakly unambiguous (Book 1971) and in star-normal form. This leads to our third result, a quadratic-time decision algorithm for weak unambiguity, that improves on the biquadratic method introduced by Book (1971).