Multiple Regular Expressions Research Articles

A Memory Efficient DFA Using Compression and Pattern Segmentation

Abstract The role of network security systems such as network intrusion detection system (NIDS) has become more important. The regular expression (a.k.a. regex) matching algorithm used for inspecting the payloads of packets is one of the most intensive tasks in NIDS. When multiple regular expressions are processed together, the corresponding Deterministic Finite Automata (DFA) becomes so complicated and needs a large amount of memory. In this paper, we propose a memory efficient parallel compatible DFA algorithm that uses the techniques of compression and pattern segmentation to reduce the memory usage. Extended from PFAC (Parallel Failureless-AC) algorithm4, the proposed compressed and segmented DFA (CSDFA) needs less numbers of states and transitions than δFA. Without considering the leading symbols “.*” in the regular expressions, the transition table can be compressed very efficiently by the run-length encoding. The number of transitions in CSDFA is about a half of the transitions needed in δFA, and uses only 74% of the memory consumed by δFA. Based on our experiments, the throughput of the proposed CSDFA is also much better than δFA.

ENREM: An efficient NFA-based regular expression matching engine on reconfigurable hardware for NIDS

Abstract Regular expression is a critical mechanism in modern network security and widely used in network intrusion detection system to describe malicious patterns. In order to speed up the pattern matching process, a number of studies have been investigated to implement regular expression matching on reconfigurable hardware. Several optimizations have been proposed, however the problem of sharing sub-patterns between multiple regular expressions is not solved completely. In this paper we present ENREM, an Efficient NFA-based Regular Expression Matching Engine on reconfigurable hardware. We introduce a new infix and suffix sharing architecture and employ it along with several techniques to optimize the required area of pattern matching circuits. In addition we developed tools for automatically generating the Verilog HDL source code of ENREM circuit from any given set of Perl compatible regular expression patterns. In order to evaluate proposed architecture, we exploit Snort rules and implement ENREM on Xilinx Virtex-II Pro XC2VP-50 FPGA. The system is tested on NetFPGA platform with DARPA intrusion detection as input data to verify the accuracy of circuit. The experimental results show that ENREM can reduce 42% LUTs and 32% FlipFlops compared with previous approaches while maintains high-speed matching throughput from 1.45 to 2.35 Gbps.

An Efficient Regular Expression Matching Method Based on Guess and Verification

Nowadays, Deterministic Finite Automaton (DFA) has been widely used to compare packet contents at a constant speed against a set of regular expressions in network security inspections. However, combining multiple regular expressions into a single DFA may cause a serious state explosion, which makes them impractical on large-scale rule set. In order to address this issue, this paper proposed a matching method based on “guess and verification”. It first searches the sub-expressions of each rule with DFA, and then verifies the result with NFA once the previous guess is successful. This method takes advantage of the high processing efficiency of DFA and the compact representation of NFA. The result shows that this proposal can provide a high throughout with a moderate memory requirement.

Traffic-Aware Frequent Elements Matching Algorithms for Deep Packet Inspection

Deep packet inspection sometimes is called application level semantic detection, which capable of examining the content of data packets in order to provide application-specific services and improve network security. Application traffic classification based on regular expressions is an essential step for deep packet inspection. However regular expression, especially multiple regular expressions matching is known to require intensive system resources and is often a performance bottleneck. Currently, the DFAs of regular expression are constructed in the preprocessing stage and the context of network streams is excluded which leads to low throughputs. In this paper, we analyzed the application level protocols and found that the protocols are uniformly distributed and it is changing dynamically. From the protocol distribution characteristic, we proposed an adaptive multiple regular expressions matching method for application traffic classification with deep packet inspection. The adaptive method, schedule the multiple DFAs through splay tree by matching probability other than linear scheduling in linked list, can adjust scheduling sequence according with the changing dynamic traffics. We evaluate the proposed method with the L7 rules; experiments proved that our method can improve the throughputs more than three times.

Reconfigurable content-based router using hardware-accelerated language parser

This article presents a dense logic design for matching multiple regular expressions with a field programmable gate array (FPGA) at 10+ Gbps. It leverages on the design techniques that enforce the shortest critical path on most FPGA architectures while optimizing the circuit size. The architecture is capable of supporting a maximum throughput of 12.90 Gbps on a Xilinx Virtex 4 LX200 and its performance is linearly scalable with size. Additionally, this article presents techniques for parsing data streams to provide semantic information for patterns found within a data stream. We illustrate how a content-based router can be implemented with our parsing techniques using an XML parser as an example. The content-based router presented was designed, implemented, and tested in a Xilinx Virtex XCV2000E FPGA on the FPX platform. It is capable of processing 32-bits of data per clock cycle and runs at 100 MHz. This allows the system to process and route XML messages at 3.2 Gbps.

Optimization of Pattern Matching Circuits for Regular Expression on FPGA

Regular expressions are widely used in the network intrusion detection system (NIDS) to represent attack patterns. Previously, many hardware architectures have been proposed to accelerate regular expression matching using field-programmable gate array (FPGA) because FPGAs allow updating of new attack patterns. Because of the increasing number of attacks, we need to accommodate a large number of regular expressions on FPGAs. Although the minimization of logic equations has been studied intensively in the area of computer-aided design (CAD), the minimization of multiple regular expressions has been largely neglected. This paper presents a novel sharing architecture allowing our algorithm to extract and share common subregular expressions. Experimental results show that our sharing scheme significantly reduces the area of pattern matching circuits for regular expression.

Ab initio identification of human microRNAs based on structure motifs.

BackgroundMicroRNAs (miRNAs) are short, non-coding RNA molecules that are directly involved in post-transcriptional regulation of gene expression. The mature miRNA sequence binds to more or less specific target sites on the mRNA. Both their small size and sequence specificity make the detection of completely new miRNAs a challenging task. This cannot be based on sequence information alone, but requires structure information about the miRNA precursor. Unlike comparative genomics approaches, ab initio approaches are able to discover species-specific miRNAs without known sequence homology.ResultsMiRPred is a novel method for ab initio prediction of miRNAs by genome scanning that only relies on (predicted) secondary structure to distinguish miRNA precursors from other similar-sized segments of the human genome. We apply a machine learning technique, called linear genetic programming, to develop special classifier programs which include multiple regular expressions (motifs) matched against the secondary structure sequence. Special attention is paid to scanning issues. The classifiers are trained on fixed-length sequences as these occur when shifting a window in regular steps over a genome region. Various statistical and empirical evidence is collected to validate the correctness of and increase confidence in the predicted structures. Among other things, we propose a new criterion to select miRNA candidates with a higher stability of folding that is based on the number of matching windows around their genome location. An ensemble of 16 motif-based classifiers achieves 99.9 percent specificity with sensitivity remaining on an acceptable high level when requiring all classifiers to agree on a positive decision. A low false positive rate is considered more important than a low false negative rate, when searching larger genome regions for unknown miRNAs. 117 new miRNAs have been predicted close to known miRNAs on human chromosome 19. All candidate structures match the free energy distribution of miRNA precursors which is significantly shifted towards lower free energies. We employed a human EST library and found that around 75 percent of the candidate sequences are likely to be transcribed, with around 35 percent located in introns.ConclusionOur motif finding method is at least competitive to state-of-the-art feature-based methods for ab initio miRNA discovery. In doing so, it requires less previous knowledge about miRNA precursor structures while programs and motifs allow a more straightforward interpretation and extraction of the acquired knowledge.

Algorithms to accelerate multiple regular expressions matching for deep packet inspection

There is a growing demand for network devices capable of examining the content of data packets in order to improve network security and provide application-specific services. Most high performance systems that perform deep packet inspection implement simple string matching algorithms to match packets against a large, but finite set of strings. owever, there is growing interest in the use of regular expression-based pattern matching, since regular expressions offer superior expressive power and flexibility. Deterministic finite automata (DFA) representations are typically used to implement regular expressions. However, DFA representations of regular expression sets arising in network applications require large amounts of memory, limiting their practical application.In this paper, we introduce a new representation for regular expressions, called the Delayed Input DFA (D 2 FA), which substantially reduces space equirements as compared to a DFA. A D 2 FA is constructed by transforming a DFA via incrementally replacing several transitions of the automaton with a single default transition. Our approach dramatically reduces the number of distinct transitions between states. For a collection of regular expressions drawn from current commercial and academic systems, a D2FA representation reduces transitions by more than 95%. Given the substantially reduced space equirements, we describe an efficient architecture that can perform deep packet inspection at multi-gigabit rates. Our architecture uses multiple on-chip memories in such a way that each remains uniformly occupied and accessed over a short duration, thus effectively distributing the load and enabling high throughput. Our architecture can provide ostffective packet content scanning at OC-192 rates with memory requirements that are consistent with current ASIC technology.

An effective query pruning technique for multiple regular path expressions

Regular path expressions are essential for formulating queries over the semistructured data without specifying the exact structure. The query pruning is an important optimization technique to avoid useless traversals in evaluating regular path expressions. While the previous query pruning optimizes a single regular path expression well, it often fails to fully optimize multiple regular path expressions. Nevertheless, multiple regular path expressions are very frequently used in nontrivial queries, and so an effective optimization technique for them is required. In this paper, we present a new technique called the two-phase query pruning that consists of the preprocessing phase and the pruning phase. Our two-phase query pruning is effective in optimizing multiple regular path expressions, and is more scalable and efficient than the combination of the previous query pruning and post-processing in that it never deals with exponentially many combinations of sub-results produced from all the regular path expressions.

A two phase optimization technique for XML queries with multiple regular path expressions

As XML (eXtensible Markup Language) has emerged as a standard for information exchange on the World Wide Web, it has gained attention in database communities to extract information from XML seen as a database model. XML queries are based on regular path queries, which find objects reachable by given regular expressions. To answer many kinds of user queries, it is necessary to evaluate queries that have multiple regular path expressions. However, previous work on subjects such as query rewriting and query optimization in the frame work of semistructured data has usually dealt with a single regular path expression. For queries that have multiple regular path expressions we suggest a two phase optimizing technique: query rewriting using views by finding the mappings from the view’s body to the query’s body and for rewritten queries, evaluating each query conjunct and combining them. We show that our rewriting algorithm is sound and our query evaluation technique is more efficient than that of previous work on optimizing semistructured queries.