Efficient Regular Expression Research Articles

P4Rex: Accelerating regular expression matching with programmable switches

Regular expression matching is pivotal in numerous network applications. With the ever-increasing scale of data center traffic, deploying regular expression matching modules on traditional servers struggles to meet throughput demands. Emerging programmable switches have brought new prospects for high-speed pattern matching. However, deploying regular expression matching onto programmable switches presents the challenge of space explosion incurred by compiling regular expressions into a Deterministic Finite Automata (DFA). In this paper, we introduce P4Rex, a regular expression matching system designed for programmable switches. P4Rex synergistically leverages two following techniques: an efficient regular expression grouping algorithm that partitions regular expressions into different groups to reduce memory consumption, and DFA compression technology to achieve transition sharing. Experimental results demonstrate that P4Rex exhibits an average improvement of 17% and maximum improvement of 30% on memory consumption compared to prior regular expression grouping schemes and saves more than 10x memory consumption compared to deploying DFA directly on programmable switch.

Efficient regular expression matching over hybrid dictionary-based compressed data

As a core technique of deep packet inspection, regular expression matching effectively extracts valuable contents from network traffic. Its performance is crucial to network security and big data applications. However, mobile internet brings massive network traffic, and most are compressed for a better user experience, which challenges the performance of matching compressed data. The present works focus on the LZ77 compressed data rather than the new excellent hybrid dictionary-based compression data. Vcdiff, as the representative scheme, combines a static shared dictionary and a self-adaptive dynamic dictionary to ensure better compression density. It also exhibits excellent performance on compression and decompression speed. However, accelerating matching Vcdiff compressed data is only suitable for the string matching scenarios and fails to show an impressive performance. In this paper, we propose ERASER to accelerate regular expression matching over hybrid dictionary-based compressed data, which is also the first approach to accelerate regular expression matching over Vcdiff compressed data. The experiments show that ERASER can double the speed of the existing methods on matching Vcdiff compressed data. It also remarkably improves preprocessing and matching performance, which is better than the state-of-the-art approach over LZ77 compressed data. We also propose a model to explore the correlation between the matching speed and the compression density. The experiments demonstrate that the matching speed of a method will increase with the better compression density of compressed data.

An efficient regular expression inference approach for relevant image extraction

Traditional approaches for extracting relevant images automatically from web pages are error-prone and time-consuming. To improve this task, operations such as preparing a larger dataset and finding new features are used in the web data extraction approaches. However, these operations are difficult and laborious. In this study, we propose a fully-automated approach based on alignment of regular expressions to automatically extract the relevant images from web pages. The automatically constructed regular expressions has been applied to a classification task for the first time. In this respect, a multi-stage inference approach is developed for generating regular expressions from the attribute values of relevant and irrelevant image elements in web pages. The proposed approach reduces the complexity of the alignment of two regular expressions by applying a constraint on a version of the Levenshtein distance algorithm. The classification accuracy of regular expression approaches is compared with the naive Bayes, logistic regression, J48, and multilayer perceptron classifiers on a balanced relevant image retrieval dataset consisting of 360 image element samples for 10 shopping websites. According to the cross-validation results, the regular expression inference-based classification achieved a 0.98 f-measure with only 5 frequent n-grams, and it outperformed other classifiers on the same set of features. The classification efficiency of the proposed approach is measured at 0.108 ms, which is very competitive with other classifiers.

Proof-directed program transformation: A functional account of efficient regular expression matching

Abstract We show how to systematically derive an efficient regular expression (regex) matcher using a variety of program transformation techniques, but very little specialized formal language and automata theory. Starting from the standard specification of the set-theoretic semantics of regular expressions, we proceed via a continuation-based backtracking matcher, to a classical, table-driven state machine. All steps of the development are supported by self-contained (and machine-verified) equational correctness proofs.

Regex matching with counting-set automata

We propose a solution to the problem of efficient matching regular expressions (regexes) with bounded repetition, such as (ab){1,100}, using deterministic automata. For this, we introduce novel counting-set automata (CsAs) , automata with registers that can hold sets of bounded integers and can be manipulated by a limited portfolio of constant-time operations. We present an algorithm that compiles a large sub-class of regexes to deterministic CsAs. This includes (1) a novel Antimirov-style translation of regexes with counting to counting automata (CAs) , nondeterministic automata with bounded counters, and (2) our main technical contribution, a determinization of CAs that outputs CsAs. The main advantage of this workflow is that the size of the produced CsAs does not depend on the repetition bounds used in the regex (while the size of the DFA is exponential to them). Our experimental results confirm that deterministic CsAs produced from practical regexes with repetition are indeed vastly smaller than the corresponding DFAs. More importantly, our prototype matcher based on CsA simulation handles practical regexes with repetition regardless of sizes of counter bounds. It easily copes with regexes with repetition where state-of-the-art matchers struggle.

Efficient Regular Expression Matching Based on Positional Inverted Index

We study efficient regular expression (regex) matching problem. Existing algorithms are scanning-based algorithms which typically use an equivalent automaton compiled from the regex query to verify a document. Although some works propose various strategies to quickly jump to candidate locations in a document where a query result may appear, they still need to utilize the scanning-based method to verify candidate locations. These methods become inefficient when there are still many candidate locations needed to be verified. In this paper, we propose a novel approach to efficiently compute all matching positions for a regex query purely based on a positional q-gram inverted index. We propose a gram-driven NFA to represent the language of a regex and show all regex matching locations can be obtained by finding positions on q-grams of GNFA that satisfy certain positional constraints. Then we propose several GNFA-based query plans to answer the query using the positional inverted index. In order to improve the query efficiency, we design the algorithm to build a tree-based query plan by carefully choosing a checking order for positional constraints. Experimental results on real-world datasets show that our method outperforms state-of-the-art methods by up to an order of magnitude in query efficiency.

Efficient regular expression matching over compressed traffic

Nowadays, regular expression matching becomes a critical component of the network traffic detection applications, which describes the fine-grained signature of traffic. Web services tend to compress their traffic for less data transmission, which is a great challenge for regular expression matching to achieve wire-speed processing. In this paper, we propose Twins, an efficient regular expression matching method over compressed traffic, which leverages the returned states encoding in the compression to skip repeated scanning. We also present an evaluation model to elaborate the factors that influence the performance of compressed traffic matching methods. Our evaluations demonstrate that Twins could skip ~ 90% compression data and can achieve 1.2 Gbps throughput with a single CPU core. It gains 2.2–3.0 times performance boost than the state-of-the-art works. With a parallel implementation using multiple CPU cores, the throughput could be up to 10 Gbps.

Efficient regular expression matching on LZ77 compressed strings using negative factors

The state-of-the-art approaches for regular expression matching on LZ78 compressed strings do not perform efficiently. Moreover, LZ78 compression has some shortcomings, such as higher compression ratio and slower decompression speed than LZ77 (a variant of LZ78). In this paper, we study regular expression matching on LZ77 compressed strings. To address this problem, we propose an efficient algorithm, namely, RELZ, utilizing the positive factors, i.e., the prefix and the suffix, and negative factors (Negative factors are substrings that cannot appear in an answer.) of the regular expression to prune the candidates. For the sake of quickly locating these two kinds of factors on the compressed string without decompression, we design a variant of suffix trie index, called SSLZ. We construct bitmaps for factors of regular expression to detect candidates. Moreover, due to the high space cost of SSLZ, we propose a variant index that partially maintain suffixes of the phrases with high frequency and develop an efficient regular expression algorithm based on the novel index, namely, RELZ+. In addition, two optimization strategies employing block filtering and LZ filtering are proposed to prune false negative candidates. At last, we conduct a comprehensive performance evaluation depending on four real data sets to validate our ideas and the proposed algorithms. The experimental results show that our RELZ and RELZ+ algorithms significantly outperform the existing algorithms.

A Memory Efficient Pattern Matching Scheme for Regular Expressions

Many malicious packets are common and spread over the Internet in recent years when the scale of Internet traffic grows at a rapid speed. Thus, the regular expression matching in Network Intrusion Detection System (NIDS) that supervises network activities needs to be very fast and consume small memory. In this paper, we propose a memory efficient regular expression matching algorithm called Failureless Segmented Finite Automata (FSFA) with an acceptable searching speed. In FSFA, We eliminate Kleene closures by using additional data structures to reduce a large amount of states. We further reduce the transitions by using default state compression technique. Our performance results implemented on a PC software environment show that our scheme only needs 1% of states needed in DFA and 2 to 22% of states needed in JFA.

Compact representations of automata for regular expression matching

Glushkov's nondeterministic finite automaton leads to efficient regular expression matching. But it is memory greedy for long regular expressions. We develop space efficient representations for the deterministic finite automata obtained from Glushkov automata. The approach reduces the space of the DFA from O(m2m) bits to O(m2k) bits where k is the number of strings in the regular expression, m is the number of characters excluding operators. The average space usage is O(m(1+1/σ)k) bits where σ is the size of the alphabet. The state transition function runs in constant time when the length of a word is not less than m. Experiments show that our method is as efficient as the previous method and uses less space.

A Memory Efficient Regular Expression Matching by Compressing Deterministic Finite Automata

Regular expressions are very meaningful and now-a-days broadly used to represent signatures of various attacks. The focal component of today’s security systems like intrusion detection and prevention system is a signature based regular expression matching. Deterministic finite automaton is often used to represent regular expressions. In regular expression matching, storage space of Deterministic finite automata is very important concern. A massive amount of memory is essential to store transition function of Deterministic finite automata. The method described in this paper reduces size of Deterministic finite automata which is in regular expression format. The performance of the regular expression matching by compressing Deterministic finite automata is evaluated by using regular expression set. General Terms Pattern matching algorithms, Network Security, Theory of Computations.

Design and optimizations for efficient regular expression matching in DPI systems

Efficient techniques for pattern matching are essential in a number of networked systems and services, such as intrusion detection systems, application identification and classification services, and traffic management. Most pattern matching applications describe patterns using regular expression and the support engine is Deterministic Finite Automaton (FA). Previous research studies address either performance or space requirements issues. From the original DFA formalism we design and evaluate optimizations to its representation and operation to meet Deep Packet Inspection (DPI) systems’ requirements for commodity platforms, such as (i) decreasing the original DFA memory consumption (high compression ratio) and (ii) performing pattern matching as fast as the original DFA. Our approach spans from designing the DFA to developing memory layouts to get the most of the underlying architecture. The contributions of this work are threefold: (i) a new and improved finite automaton model, called Ranged Compressed DFA (RCDFA), (ii) three RCDFA optimizations for achieving more compression and matching speed, and (iii) three advanced layouts for implementing the compressed automaton without memory and performance penalties. The experimental evaluation shows that RCDFA compresses DFA up to 99% without imposing additional memory lookups. The proposed advanced layouts reach memory compression of around 97%. RCDFA together with the advanced layouts outperforms the standard DFA by up to 32 times in terms of processing speed.

ENREM: An efficient NFA-based regular expression matching engine on reconfigurable hardware for NIDS

Abstract Regular expression is a critical mechanism in modern network security and widely used in network intrusion detection system to describe malicious patterns. In order to speed up the pattern matching process, a number of studies have been investigated to implement regular expression matching on reconfigurable hardware. Several optimizations have been proposed, however the problem of sharing sub-patterns between multiple regular expressions is not solved completely. In this paper we present ENREM, an Efficient NFA-based Regular Expression Matching Engine on reconfigurable hardware. We introduce a new infix and suffix sharing architecture and employ it along with several techniques to optimize the required area of pattern matching circuits. In addition we developed tools for automatically generating the Verilog HDL source code of ENREM circuit from any given set of Perl compatible regular expression patterns. In order to evaluate proposed architecture, we exploit Snort rules and implement ENREM on Xilinx Virtex-II Pro XC2VP-50 FPGA. The system is tested on NetFPGA platform with DARPA intrusion detection as input data to verify the accuracy of circuit. The experimental results show that ENREM can reduce 42% LUTs and 32% FlipFlops compared with previous approaches while maintains high-speed matching throughput from 1.45 to 2.35 Gbps.

Exploring different automata representations for efficient regular expression matching on GPUs

Regular expression matching is a central task in several networking (and search) applications and has been accelerated on a variety of parallel architectures. All solutions are based on finite automata (either in deterministic or non-deterministic form), and mostly focus on effective memory representations for such automata. Recently, a handful of work has proposed efficient regular expression matching designs for GPUs; however, most of them aim at achieving good performance on small datasets. Nowadays, practical solutions must support the increased size and complexity of real world datasets. In this work, we explore the deployment and optimization of different GPU designs of regular expression matching engines, focusing on large datasets containing a large number of complex patterns.

An Efficient Regular Expression Matching Method Based on Guess and Verification

Nowadays, Deterministic Finite Automaton (DFA) has been widely used to compare packet contents at a constant speed against a set of regular expressions in network security inspections. However, combining multiple regular expressions into a single DFA may cause a serious state explosion, which makes them impractical on large-scale rule set. In order to address this issue, this paper proposed a matching method based on “guess and verification”. It first searches the sub-expressions of each rule with DFA, and then verifies the result with NFA once the previous guess is successful. This method takes advantage of the high processing efficiency of DFA and the compact representation of NFA. The result shows that this proposal can provide a high throughout with a moderate memory requirement.

GPU-based NFA implementation for memory efficient high speed regular expression matching

Regular expression pattern matching is the foundation and core engine of many network functions, such as network intrusion detection, worm detection, traffic analysis, web applications and so on. DFA-based solutions suffer exponentially exploding state space and cannot be remedied without sacrificing matching speed. Given this scalability problem of DFA-based methods, there has been increasing interest in NFA-based methods for memory efficient regular expression matching. To achieve high matching speed using NFA, it requires potentially massive parallel processing, and hence represents an ideal programming task on Graphic Processor Unit (GPU). Based on in-depth understanding of NFA properties as well as GPU architecture, we propose effective methods for fitting NFAs into GPU architecture through proper data structure and parallel programming design, so that GPU's parallel processing power can be better utilized to achieve high speed regular expression matching. Experiment results demonstrate that, compared with the existing GPU-based NFA implementation method [9], our proposed methods can boost matching speed by 29~46 times, consistently yielding above 10Gbps matching speed on NVIDIA GTX-460 GPU. Meanwhile, our design only needs a small amount of memory space, growing exponentially more slowly than DFA size. These results make our design an effective solution for memory efficient high speed regular expression matching, and clearly demonstrate the power and potential of GPU as a platform for memory efficient high speed regular expression matching.

Jump-CFA: Fast and Memory Efficient Regular Expression Matching for Counting Constraints

Jump-CFA: Fast and Memory Efficient Regular Expression Matching for Counting Constraints

Efficient Regular Expression Compression Algorithm for Deep Packet Inspection

提出一种基于确定的有穷状态自动机(deterministic finite automaton,简称DFA)的正则表达式压缩算法.首先,定义了膨胀率DR(distending rate)来描述正则表达式的膨胀特性.然后基于DR提出一种分片的算法RECCADR (regular expressions cut and combine algorithm based on DR),有效地选择出导致DFA状态膨胀的片段并隔离,降低了单个正则表达式存储需求.同时,基于正则表达式的组合关系提出一种选择性分群算法REGADR(regular expressions group algorithm based on DR),在可以接受的存储需求总量下,通过选择性分群大幅度减少了状态机的个数,有效地降低了匹配算法的复杂性.;A memory efficient regular expression compression algorithm is devised for deep packet inspection. First, a parameter DR (distending rate) is defined to quantify the explosive quality of regular expressions. Then based on DR, a regular expression cutting algorithm is proposed to downsize the storage needs of individual regular expression, by detecting and confining the parts which cause DFA (deterministic finite automaton) states’ exponential growth. Then according to the relation of different regular expressions, a selective grouping algorithm is introduced for regular expression sets, which could cut down the number of finite automata, and reduce the runtime memory consumption.