Set Of Regular Expressions Research Articles

Developing programming language with compilers using JFlex in NetBean: Expanding and testing simple operators by implementing a calculator

Abstract: JFlex is a lexical analyzer generator and takes input requirements with a set of regular expressions and corresponding actions. It creates a program (a lexer) that reads input, matches the input against the regular expressions, and runs the matching action. This paper shows how Programming Language can be developed. This work was done to develop a simple programming language with compilers using JFlex in NetBean so that it can support assignment statements, if then else, while do and type checking and its execution. The data type included are int, real, char, and Boolean/String. The key concept used in this work was the execution of the grammar or rules in the cup file. The parse tree records a sequence of rules the parser applies to recognize the input. The tool used for the development of lexical analyzers was JFlex. JFlex Lexers was based on deterministic finite automata (DFAs). To show the implementation and working of operators, a simple calculator was designed that supports addition and multiplication operations. Further, the key compiler concepts like lexical analyzers, semantic analysis, and parse trees are discussed. This paper will help understand the syntax and way to develop simple language. For the programming language developed, the evaluations of the expressions and statements are recursively done. Type checking and Error checking are also done where two operands are checked for their compatibility with the operator and are shown if incompatible expressions are found.

Effective Filter for Common Injection Attacks in Online Web Applications

Injection attacks against web applications are still frequent, and organizations like OWASP places them within the Top Ten of security risks to web applications. The main goal of this work is to contribute to the community with the design of an effective protection of web applications against common injection attacks. Our proposal is a validation filter of input fields that is based on OWASP Stinger, a set of regular expressions, and a sanitization process. It validates both fundamental characters (letters, numbers, dot, dash, question marks, and exclamation point) and complex statements (JSON and XML files) for each field. The procedure of deploying the proposed filter is detailed, specifying the sections and contents of the configuration file. In addition, the infrastructure for running the tests is described, including the setting of an attack tool, and the implementation of a controller. The attack tool is used as a security scanner for common injection attacks, and the controller is developed for routing the requests in two steps; first a request is addressed to the filter, and if it is valid, it will redirect to the web application itself. The proposal filter has been tested on three public as well as on a real private web application. An accuracy of 98,4% and an average processing time of 50 ms are achieved, based on wich it is possible to conclude the proposed filter is highly reliable and does not require additional computational resources.

Regular Expression Learning from Positive Examples Based on Integer Programming

This paper presents a novel method to infer regular expressions from positive examples. The method consists of a candidate’s construction phase and an optimization phase. We first propose multiscaling sample augmentation to capture the cycle patterns from single examples during the candidate’s construction phase. We then use common substrings to build regular expressions that capture patterns across multiple examples, and we show this algorithm is more general than those based on common prefixes or suffixes. Furthermore, we propose a pruning mechanism to improve the efficiency of useful common substring mining, which is an important part of common substring-based expression building algorithm. Finally, in the optimization phase, we model the problem of choosing a set of regular expressions with the lowest cost as an integer linear program, which can be solved to obtain the optimal solution. The experimental results on synthetic and real-life samples demonstrate the effectiveness of our approach in inferring concise and semantically meaningful regular expressions for string datasets.

A Technical Survey on Intelligent Optimization Grouping Algorithms for Finite State Automata in Deep Packet Inspection

Construction and deployment of finite state automata from the regular expressions might results in huge overhead and results in the state explosion problem which is in need of large memory space, high bandwidth and additional computational time. To overcome this problem, a new framework is proposed, and several intelligent optimization algorithms are reviewed and compared in this framework. The proposed approach is called intelligent optimization grouping algorithms (IOGA), which intends to group regular expression intelligently. IOGAs are used to allocate the regular expression sets into various groups and to build independent deterministic finite automata (DFA) for each group. Grouping the regular expression efficiently solves the state explosion problem by achieving large-scale best tradeoff among memory utilization and computational time. This study reviews and compares the various alternatives of IOGA including genetic algorithm, ant colony optimization, particle swarm optimization, bacterial foraging optimization, artificial bee colony algorithm, biogeography based optimization, cuckoo search, firefly algorithm, bat algorithm and flower pollination algorithm for solving the problem of DFA state explosion and also for improving the overall efficiency of deep packet inspection (DPI). The discussions state that by effectively using these grouping algorithms along with DFA based DPI, the number of states can be reduced, providing a balance between the memory consumption, time complexity, throughput, inspection speed, convergence speed and grouping time.

A Pattern-Based Method for Medical Entity Recognition From Chinese Diagnostic Imaging Text.

Background: The identification of medical entities and relations from electronic medical records is a fundamental research issue for medical informatics. However, the task of extracting valuable knowledge from these records is challenging due to its high complexity. The accurate identification of entity and relation is still an open research problem in medical information extraction.Methods: A pattern-based method for extracting certain tumor-related entities and attributes from Chinese unstructured diagnostic imaging text is proposed. This method is a composition of three steps. Firstly, an algorithm based on keyword matching is designed to obtain the primary sites of tumors. Then a set of regular expressions is applied to identify primary tumor size information. Finally, a set of rules is defined to acquire metastatic sites of tumors.Results: Our method achieves a recall of 0.697, a precision of 0.825 and an F1 score of 0.755 using an overall weighted metric. For each of the extraction tasks, the F1 scores are 0.784, 0.822 and 0.740.Conclusions: The method proves to be stable and robust with different amounts of testing data. It achieves a comparatively high performance in the CHIP 2018 open challenge, demonstrating its effectiveness in extracting tumor-related entities from Chinese diagnostic imaging text.

Regular Expression Based Medical Text Classification Using Constructive Heuristic Approach

Medical text classification assigns medical related text into different categories such as topics or disease types. Machine learning based techniques have been widely used to perform such tasks despite the obvious drawback in such “black box” approach, leaving no easy way to fine-tune the resultant model for better performance. We propose a novel constructive heuristic approach to generate a set of regular expressions that can be used as effective text classifiers. The main innovation of our approach is that we develop a novel regular expression based text classifier with both satisfactory classification performance and excellent interpretability. We evaluate our framework on real-world medical data provided by our collaborator, one of the largest online healthcare providers in the market, and observe the high performance and consistency of this approach. Experimental results show that the machine-generated regular expressions can be effectively used in conjunction with machine learning techniques to perform medical text classification tasks. The proposed methodology improves the performance of baseline methods (Naive Bayes and Support Vector Machines) by 9% in precision and 4.5% in recall. We also evaluate the performance of modified regular expressions by human experts and demonstrate the potential of practical applications using the proposed method.

Improved deep packet inspection in data stream detection

Finite state automata are widely used in firewalls, data detection and content audit systems to match complex sets of regular expressions in network packets. However, with the continuous increase in the types of network contents and network traffics in recent years, the deep packet inspection systems based on finite state automata also require regular engines for less memory consumption and higher operating speed. This paper analyzes the feature and problem of finite state automata and improves non-deterministic finite automata by reducing the conversion edge to reduce the memory usage. The experiment results which are made by real-world dataset show that the memory usage is reduced more than half.

Intelligent and efficient grouping algorithms for large-scale regular expressions

Regular expressions are widely used in various applications. Due to its low time complexity and stable performance, Deterministic Finite Automaton (DFA) has become the first choice to perform fast regular expression matching. Unfortunately, compiling a large set of regular expressions into one DFA often leads to the well-known state explosion problem, and consequently requires huge memory consumption. Regular expression grouping is a practical approach to mitigate this problem. However, existing grouping algorithms are either brute-force or locally optimal and thus not efficient for large-scale regular expressions. In this paper, we propose two grouping algorithms, namely Reevo and Reant, to solve this problem intelligently and efficiently. In addition, two optimization methods are presented to accelerate the convergence speed and reduce the running time of proposed algorithms. Experimental results on large-scale rulesets from real world show that our algorithms achieve around 25% to 45% improvement compared to previous regular expression grouping algorithms.

Scalable Algorithms for NFA Multi-Striding and NFA-Based Deep Packet Inspection on GPUs

Finite state automata (FSA) are used by many network processing applications to match complex sets of regular expressions in network packets. In order to make FSA-based matching possible even at the ever-increasing speed of modern networks, multi-striding has been introduced. This technique increases input parallelism by transforming the classical FSA that consumes input byte by byte into an equivalent one that consumes input in larger units. However, the algorithms used today for this transformation are so complex that they often result unfeasible for large and complex rule sets. This paper presents a set of new algorithms that extend the applicability of multi-striding to complex rule sets. These algorithms can transform nondeterministic finite automata (NFA) into their multi-stride form with reduced memory and time requirements. Moreover, they exploit the massive parallelism of graphical processing units for NFA-based matching. The final result is a boost of the overall processing speed on typical regex-based packet processing applications, with a speedup of almost one order of magnitude compared to the current state-of-the-art algorithms.

Recognition of chemical entities: combining dictionary-based and grammar-based approaches.

BackgroundThe past decade has seen an upsurge in the number of publications in chemistry. The ever-swelling volume of available documents makes it increasingly hard to extract relevant new information from such unstructured texts. The BioCreative CHEMDNER challenge invites the development of systems for the automatic recognition of chemicals in text (CEM task) and for ranking the recognized compounds at the document level (CDI task). We investigated an ensemble approach where dictionary-based named entity recognition is used along with grammar-based recognizers to extract compounds from text. We assessed the performance of ten different commercial and publicly available lexical resources using an open source indexing system (Peregrine), in combination with three different chemical compound recognizers and a set of regular expressions to recognize chemical database identifiers. The effect of different stop-word lists, case-sensitivity matching, and use of chunking information was also investigated. We focused on lexical resources that provide chemical structure information. To rank the different compounds found in a text, we used a term confidence score based on the normalized ratio of the term frequencies in chemical and non-chemical journals.ResultsThe use of stop-word lists greatly improved the performance of the dictionary-based recognition, but there was no additional benefit from using chunking information. A combination of ChEBI and HMDB as lexical resources, the LeadMine tool for grammar-based recognition, and the regular expressions, outperformed any of the individual systems. On the test set, the F-scores were 77.8% (recall 71.2%, precision 85.8%) for the CEM task and 77.6% (recall 71.7%, precision 84.6%) for the CDI task. Missed terms were mainly due to tokenization issues, poor recognition of formulas, and term conjunctions.ConclusionsWe developed an ensemble system that combines dictionary-based and grammar-based approaches for chemical named entity recognition, outperforming any of the individual systems that we considered. The system is able to provide structure information for most of the compounds that are found. Improved tokenization and better recognition of specific entity types is likely to further improve system performance.

Natural Language Processing Techniques for Extracting and Categorizing Finding Measurements in Narrative Radiology Reports.

Accumulating quantitative outcome parameters may contribute to constructing a healthcare organization in which outcomes of clinical procedures are reproducible and predictable. In imaging studies, measurements are the principal category of quantitative para meters. The purpose of this work is to develop and evaluate two natural language processing engines that extract finding and organ measurements from narrative radiology reports and to categorize extracted measurements by their "temporality". The measurement extraction engine is developed as a set of regular expressions. The engine was evaluated against a manually created ground truth. Automated categorization of measurement temporality is defined as a machine learning problem. A ground truth was manually developed based on a corpus of radiology reports. A maximum entropy model was created using features that characterize the measurement itself and its narrative context. The model was evaluated in a ten-fold cross validation protocol. The measurement extraction engine has precision 0.994 and recall 0.991. Accuracy of the measurement classification engine is 0.960. The work contributes to machine understanding of radiology reports and may find application in software applications that process medical data.

Fast Regular Expression Matching Using Small TCAM

Regular expression (RE) matching is a core component of deep packet inspection in modern networking and security devices. In this paper, we propose the first hardware-based RE matching approach that uses ternary content addressable memory (TCAM), which is available as off-the-shelf chips and has been widely deployed in modern networking devices for tasks such as packet classification. We propose three novel techniques to reduce TCAM space and improve RE matching speed: transition sharing, table consolidation, and variable striding. We tested our techniques on eight real-world RE sets, and our results show that small TCAMs can be used to store large deterministic finite automata (DFAs) and achieve potentially high RE matching throughput. For space, we can store each of the corresponding eight DFAs with 25 000 states in a 0.59-Mb TCAM chip. Using a different TCAM encoding scheme that facilitates processing multiple characters per transition, we can achieve potential RE matching throughput of 10-19 Gb/s for each of the eight DFAs using only a single 2.36-Mb TCAM chip.

An Efficient Regular Expression Matching Method Based on Guess and Verification

Nowadays, Deterministic Finite Automaton (DFA) has been widely used to compare packet contents at a constant speed against a set of regular expressions in network security inspections. However, combining multiple regular expressions into a single DFA may cause a serious state explosion, which makes them impractical on large-scale rule set. In order to address this issue, this paper proposed a matching method based on “guess and verification”. It first searches the sub-expressions of each rule with DFA, and then verifies the result with NFA once the previous guess is successful. This method takes advantage of the high processing efficiency of DFA and the compact representation of NFA. The result shows that this proposal can provide a high throughout with a moderate memory requirement.

Automated extraction of ejection fraction for quality measurement using regular expressions in Unstructured Information Management Architecture (UIMA) for heart failure

Left ventricular ejection fraction (EF) is a key component of heart failure quality measures used within the Department of Veteran Affairs (VA). Our goals were to build a natural language processing system to extract the EF from free-text echocardiogram reports to automate measurement reporting and to validate the accuracy of the system using a comparison reference standard developed through human review. This project was a Translational Use Case Project within the VA Consortium for Healthcare Informatics. We created a set of regular expressions and rules to capture the EF using a random sample of 765 echocardiograms from seven VA medical centers. The documents were randomly assigned to two sets: a set of 275 used for training and a second set of 490 used for testing and validation. To establish the reference standard, two independent reviewers annotated all documents in both sets; a third reviewer adjudicated disagreements. System test results for document-level classification of EF of <40% had a sensitivity (recall) of 98.41%, a specificity of 100%, a positive predictive value (precision) of 100%, and an F measure of 99.2%. System test results at the concept level had a sensitivity of 88.9% (95% CI 87.7% to 90.0%), a positive predictive value of 95% (95% CI 94.2% to 95.9%), and an F measure of 91.9% (95% CI 91.2% to 92.7%). An EF value of <40% can be accurately identified in VA echocardiogram reports. An automated information extraction system can be used to accurately extract EF for quality measurement.

Implication of regular expressions

In this work we study the following implication problem for regular expressions: “Given a set of regular expressions R and a regular expression S , is it true that every string which matches the regular expressions in R also matches S ?” The problem comes in two flavors: “non-disjoint” and “disjoint”. We show that both of them are PSPACE-complete. While the complexity for the first variant is not surprising–the problem is coNP-complete even for very simple patterns (given by wildcards)–the complexity result for the second variant represents a big jump since the problem is in PTIME for the case of patterns with wildcards. Towards the goal of charting the boundary of tractability, we then present an analysis of when the problem remains in PTIME.

Using Grid services to parallelize IBM's Generic Log Adapter

Since their definition in the Open Grid Services Architecture, Grid services has been used in many Grid-enabled applications to leverage the computational power offered by Grid Systems. An important research issue addressed in this regard is how to increase the efficiency of the Grid services for a massive processing and scientific computing computations arising in data intensive computations, for example the processing of large log data files arising in “problem determination” in today's IT computing environments. In this paper we present an approach that uses Grid services to efficiently parallelize the IBM's Generic Log Adapter (GLA). GLA is a generic parsing engine shipped with the IBM's Autonomic Computing Toolkit that has been conceived to convert proprietary log data into a standard log data event-based format in real time. However, in order to provide generic support for parsing the majority of today's unstructured log data formats the GLA makes heavy use of regular expressions that incur in performance limitations. Until now all the approaches that have been proposed to increase GLA's performance have revolved around fine-tuning the set of regular expressions used to configure the GLA for a particular log data format or writing specific parsing code. In this work we propose a new approach consisting in transparently parallelizing the GLA by taking advantage of its internal architecture and the fact that structuring log data is a task that lends itself very well to parallelization. We present a Master–Worker strategy that uses Grid services to parallelize GLA efficiently and in a completely transparent way for the user.

Efficient Regular Expression Compression Algorithm for Deep Packet Inspection

提出一种基于确定的有穷状态自动机(deterministic finite automaton,简称DFA)的正则表达式压缩算法.首先,定义了膨胀率DR(distending rate)来描述正则表达式的膨胀特性.然后基于DR提出一种分片的算法RECCADR (regular expressions cut and combine algorithm based on DR),有效地选择出导致DFA状态膨胀的片段并隔离,降低了单个正则表达式存储需求.同时,基于正则表达式的组合关系提出一种选择性分群算法REGADR(regular expressions group algorithm based on DR),在可以接受的存储需求总量下,通过选择性分群大幅度减少了状态机的个数,有效地降低了匹配算法的复杂性.;A memory efficient regular expression compression algorithm is devised for deep packet inspection. First, a parameter DR (distending rate) is defined to quantify the explosive quality of regular expressions. Then based on DR, a regular expression cutting algorithm is proposed to downsize the storage needs of individual regular expression, by detecting and confining the parts which cause DFA (deterministic finite automaton) states’ exponential growth. Then according to the relation of different regular expressions, a selective grouping algorithm is introduced for regular expression sets, which could cut down the number of finite automata, and reduce the runtime memory consumption.

Synthesis of regular expressions for FPGAs

Regular expressions are being used in many applications to specify multiple and complex text patterns in a compact way. In some of these applications large sets of regular expressions need to be evaluated to detect matched content. Specialised hardware engines are employed when software-based regular expression engines are not able to meet the performance requirements imposed by such applications. Since the sets of regular expressions are periodically modified and/or extended, FPGAs are an attractive hardware solution to achieve both programmability and high-performance demands. However, efficient automatic synthesis tools are of paramount importance to achieve fast prototyping of regular expression engines on these devices. This paper presents an overview of the synthesis of regular expressions with the aim of achieving high-performance engines for FPGAs. We focus on describing current solutions, proposing new solutions for constraint repetitions and overlapped matching, and discussing a number of challenges and open issues. As a case study, we present FPGA implementations of the regular expressions included in two rule-sets of network intrusion detection system (NIDS), Bleeding Edge and Snort, obtained using a state-of-the-art synthesis approach.

A characterization of regular expressions under bisimulation

We solve an open question of Milner [1984]. We define a set of so-called well-behaved finite automata that, modulo bisimulation equivalence, corresponds exactly to the set of regular expressions, and we show how to determine whether a given finite automaton is in this set. As an application, we consider the star height problem.

NORMALIZED EXPRESSIONS AND FINITE AUTOMATA

There exist two well-known quotients of the position automaton of a regular expression. The first one, called the equation automaton, was first introduced by Mirkin from the notion of prebase and has been redefined by Antimirov from the notion of partial derivative. The second one, due to Ilie and Yu and called the follow automaton, can be obtained by eliminating ε-transitions in an ε-NFA that is always smaller than the classical ε-NFAs (Thompson, Sippu and Soisalon–Soininen). Ilie and Yu discussed the difficulty of succeeding in a theoretical comparison between the size of the follow automaton and the size of the equation automaton and concluded that it is very likely necessary to realize experimental studies. In this paper we solve the theoretical question, by first defining a set of regular expressions, called normalized expressions, such that every regular expression can be normalized in linear time, and proving then that the equation automaton of a normalized expression is always smaller than its follow automaton.