Regex Patterns Research Articles

Sensitive Data Flagging within Data Quality reports: R and Regex Integration for Effective Text Flagging in Large Datasets

ObjectiveWe’re introducing a comprehensive approach to enhance sensitive data flagging within our Data Quality (DQ) tool. Within de-identified health records, sensitive data such as user ids, post codes, phone numbers, and addresses pose significant privacy risks if exposed in their raw form. To address this challenge, we propose using all necessary regex patterns and free text field checks for sensitive data flagging with the objective of efficient detection within large datasets and ultimately reporting such data. ApproachA regex map has been developed along free text field thresholds, distribution counts and percentage checks to allow a generic approach. This ensures that any new regexes can be added to the flagging process without significant code changes. Once identified, sensitive data instances, including fields that contain free text are flagged and displayed in an html report that is human readable by DQ reviewers. ResultsThrough a combination of SQL and R regex based text processing, our approach allows a seamless identification and flagging of sensitive data within datasets. Sensitive data for large tables of +25 million records with over 50 columns are getting flagged in less than 150 seconds (approx. 2 minutes). ConclusionOur development offers a practical solution for sensitive data flagging in complex datasets where team members can implement robust sensitive data flagging by adding more regex formulas in the codebase. ImplicationsDQ reviewers can inspect the sensitive flagged fields in a human readable way, providing an extra measure of confidence when determining the quality of de-identified health records.

Mobile Security Operation Centre (mSOC)

Attacks on the internet are becoming increasingly threatening. For naïve home users, who are poorly protected, there is always an imminent danger of getting cyber attacked. 
 This paper is aimed to design and build an IoT-based Network Security device that would run as an access point for users to connect to the Internet in a home setting. The paper discusses a standalone perimeter security solution with Incident Response (IR) life cycle management and controls through an IoT device – Raspberry PI. Enterprise-level features such as Next Generation Firewall (NGFW), Network Intrusion Detection System (NIDS), Domain Control for Ad/Spam blocking, Security Information and Event Management (SIEM) for Log Co-ran System on Chip (SoC), which can be installed anywhere and carried for mobile operations. Hence, the name, Mobile Security Operation Centre (mSOC).
 This solution intends to protect the user when browsing the internet and blocking or providing visibility to the malicious connections made to or from users. The mSOC can filter domains based on whitelist/blacklist and Regex Pattern. It can also identify the domains that are blocked or allowed. It also provides visibility to traffic, application statistics, and IP reputation. IP reputation and Malicious Domains then can act as input to the iptables for L3/L4 blocking. A Software User Interface is developed to integrate and manage multiple Open-Sourced applications like dnsmasq/ elk/ graylog/ SQlite3/ Iptables/ adminlte as a single product that could serve as a complete security solution for a home or Small Medium Business (SMB). Thus, the proposed solution secures naïve users from security exploitations.

REDoS Detection in “Domino” Regular Expressions by Ambiguity Analysis

The Regular Expression Denial of Service (REDoS) problem refers to a time explosion caused by the high computational complexity of matching a string against a regex pattern. This issue is prevalent in popular regex engines, such as PYTHON, JAVASCRIPT, and C++. In this paper, we examine several existing open-source tools for detecting REDoS and identify a class of regexes that can create REDoS situations in popular regex engines but are not detected by these tools. To address this gap, we propose a new approach based on ambiguity analysis, which combines a strong star-normal form test with an analysis of the transformation monoids of Glushkov automata orbits. Our experiments demonstrate that our implementation outperforms the existing tools on regexes with polynomial matching complexity and complex subexpression overlap structures.

Reconfigurable regular expression matching architecture for real-time pattern update and payload inspection

Regular expression (regex) matching is an integral part of deep packet inspection (DPI), but its efficiency becomes a question due to low performance. For regex matching (REM) acceleration, FPGA-based solutions have emerged to maximize parallelism by processing multiple regex patterns concurrently. However, even though they significantly accelerate the performance, they have a critical problem that they do not support dynamic regex pattern updates in run time, which is the key functionality along with frequently altered signatures to cover newly identified vulnerabilities. Hence, we present Reinhardt, a new reconfigurable hardware architecture for REM. Reinhardt introduces new FPGA blocks, called reconfigurable cells, that form regex patterns in hardware, enabling real-time regex pattern update and match in run time while providing high performance. With the prototype of Reinhardt on NetFPGA-SUME, our evaluation shows that Reinhardt updates hundreds of regex patterns within a second and performs REM at up to 10 Gbps throughput (max. hardware bandwidth) with the constant latency. Our case studies also show that Reinhardt can operate in multiple modes (e.g., as a standalone NIDS/NIPS or as the REM accelerator for them).

Predicting input validation vulnerabilities based on minimal SSA features and machine learning

Structured Query Language injection (SQLi) and Cross-Site Scripting (XSS) are the most renowned kinds of input validation vulnerabilities. Of late, vulnerability prediction models based on machine learning have been gaining acceptance in the domain of Web security. Such models offer an easy and effective way of dealing with web application security concerns. However, most of them, in particular, rely on complex graphs generated from source code or regex patterns based on expert knowledge. This paper proposed a method for extracting features from source code and predicting input validation vulnerabilities using machine learning algorithms. The proposed method can extract all features related to the flow of vulnerabilities among the programs and remove the features that are irrelevant to the vulnerability flow. In addition, each vulnerability’s context has been assigned, providing additional data for our model to use in learning about the vulnerability context. Compared to other related methods, the feature extraction method proposed in this paper has been found to have high reusability and better performance. The best model related to the LSTM classifier had a 98.1% recall rate, a 97.9% precision, an accuracy of 98.67%, and a 99.03% area under the curve (AUC) in the test dataset.

Developing Smart Web-Search using Regex

Due to the increasing storage data on the Web Applications it becomes very difficult to use only keywordbased searches to provide comprehensive search results, thus increasing the difficulty for web users to search information on the web. In this paper, we proposed using a combined method of keyword-based and Regular expressions (RegEx) searches to perform search using strings of targeted items for optimal results even as the volume of data around the world on the Internet continues to explode. The idea is to embed RegEx patterns as part of search engine’s algorithm in a web application project to provide strings related to the targeted items for more comprehensive coverage of search results. The user's search query is a string of characters guided by search boundaries selected from the entry point. The results returned from the search operation are different results within a category determined by the search boundaries. This is designed to be beneficial to a user who has an obscure idea about the information he/she wanted to search but knows the boundries within which to get the information. This technique can be applied to data processing tasks such as information extraction and search refinements.

High throughput token driven FSM based regex pattern matching for network intrusion detection system

A single core SRAM memory with a unified memory controller is proposed for regex patterns which contains complex signatures. Many existing NIDS systems are based on token stream with dedicated hardware units to accommodate byte oriented matching with moderate network throughput rate. The overall NIDS performance metrics is degraded when the number of bytes in the database increased. It will lead the demands of coordinated payload monitoring scheme in any memory based digital NIDS system. In this work, a bit based pattern matching algorithm with FSM state transition is proposed which gives both parallel processing and network assertion payload validity check during the conversion of segments into tokens. The Finite State Machine state controller is used in early misdetection to optimize overall matching time in case of segments with multiple non-trivial tokens where the regex patterns are divided into sub-patterns and the concurrent matches are halted in a parallel manner. Finally synchronization problem during hierarchical state transition and vector matching over multiple pages unique SRAM controllers are used which is driven with fully down converted sub-groups. The efficiency of proposed NIDS system over moderate intrusion database process is proved in terms of classification rate. Hence, proposed NIDS system consumes lesser memory resources and it is verified through comparison with state-of-the-art methods.

Automating test oracles from restricted natural language agile requirements

AbstractManual testing of software requirements written in natural language for agile or any other methodology requires more time and human resources. This leaves the testing process error prone and time consuming. For satisfied end users with bug‐free software delivered on time, there is a need to automate the test oracle process for natural language or informal requirements. The automation of the test oracle is relatively easier with formal requirements, but this task is difficult to achieve with natural language requirements. This study proposes an approach called Restricted Natural Language Agile Requirements Testing (ReNaLART) to automate the test oracle from restricted natural language agile requirements. For this purpose, it uses an existing user story template with some modifications for writing user stories. This helps in identifying test input and expected output for a user story. For comparison of expected and observed outputs it makes use of a regex pattern and string distance functions. It is capable of assigning different types of verdicts automatically depending upon the similarity/dissimilarity between observed and expected outputs of user stories. ReNaLART is validated using several case studies of different domains, namely, OLX Pakistan, Mental Health Tests, McDelivery Pakistan, BlueStacks, Power Searching with Google, TensorFlow Playground, w3Schools 2018 offline and Touch'D. It revealed several faults in five of the above listed eight applications. Plus, the proposed test oracle on an average took 0.02 s for test data generation, expected output generation and verdict assignment. Both these facts show the fault revealing effectiveness and efficiency of ReNaLART.

TAP: A static analysis model for PHP vulnerabilities based on token and deep learning technology.

With the widespread usage of Web applications, the security issues of source code are increasing. The exposed vulnerabilities seriously endanger the interests of service providers and customers. There are some models for solving this problem. However, most of them rely on complex graphs generated from source code or regex patterns based on expert experience. In this paper, TAP, which is based on token mechanism and deep learning technology, was proposed as an analysis model to discover the vulnerabilities of PHP: Hypertext Preprocessor (PHP) Web programs conveniently and easily. Based on the token mechanism of PHP language, a custom tokenizer was designed, and it unifies tokens, supports some features of PHP and optimizes the parsing. Besides, the tokenizer also implements parameter iteration to achieve data flow analysis. On the Software Assurance Reference Dataset(SARD) and SQLI-LABS dataset, we trained the deep learning model of TAP by combining the word2vec model with Long Short-Term Memory (LSTM) network algorithm. According to the experiment on the dataset of CWE-89, TAP not only achieves the 0.9941 Area Under the Curve(AUC), which is better than other models, but also achieves the highest accuracy: 0.9787. Further, compared with RIPS, TAP shows much better in multiclass classification with 0.8319 Kappa and 0.0840 hamming distance.

Harnessing Twitter for Answering Opinion List Queries

Opinion list (OL) queries like “valentines day gift ideas” and “best anniversary messages for your parents” are quite popular on web search engines. Users expect instant answers comprising of a list of relevant items (OL) for such a query. Surprisingly, current search engines do not provide any crisp instant answers for queries in this critical query segment. To the best of our knowledge, we present the first system that tackles such queries. Although such social factors are heavily discussed on online social networks like Twitter, extracting such lists from tweets is quite challenging. The challenges lie in discovering such lists from tweets, rank the discovered list items as well as handle lists with very low cardinality (tail OLs). We present an end-to-end system that: 1) identifies these “OLs” from a large number of Twitter hashtags using a classifier trained using novel task-specific features; 2) extracts suitable list answers from relevant tweets using carefully designed regex patterns; 3) uses the learning to rank framework to present a ranked list of these items; and 4) handles tail lists using a novel algorithm to borrow list items from similar lists. Crowd-sourced evaluation shows that the proposed system can extract OLs with a good accuracy.

MEMORY-Based Hardware Architectures to Detect ClamAV Virus Signatures with Restricted Regular Expression Features

We aim to implement a single-chip hardware detection engine for virus scanning. Our study is based on the ClamAV virus database, which contains 88.9K strings and 9.6K extended hex-signatures with restricted regular expression (regex) features. We have previously presented cost-effective hardware architectures to detect the 88.9K strings and 3.2K regex patterns that are composed of multiple string segments. In this paper, we shall present hardware architectures to detect the remaining 6.4K regex patterns. Our method is based on the information reduction approach. We transform the byte-oriented matching problem to a token-based matching problem. A regex pattern contains one or more segments, and a segment may be subdivided into multiple non-trivial tokens. In general, a token is associated with one or a few regexes only. The input byte-stream is converted into a token-stream using dedicated hardware units, where the number of tokens is much less than the number of bytes. The token-stream is processed by a NFA-based aggregation unit to determine if any segment can be found. Detected segments are further processed by a scoreboard to determine if any multi-segment pattern can be found. For proof-of-concept, our method is implemented on a Virtex-6 FPGA which consumes 1.84 MB on-chip memory.

동일한 입력 문자를 가지는 상태의 병합을 통한 메모리 효율적인 결정적 유한 오토마타 구현

패턴 정합 알고리듬은 침입 탐지 및 방지 시스템의 성능을 좌우하는 중요한 기능 요소로서 일반적으로 정규 표현식(Regualr Expressions)을 사용해 패턴을 표현한다. 공격 패턴이 복잡해지고 다양해짐에 따라, 정규 표현식 또한 복잡해지고 그 수가 증가하고 있으며 이로 인해, 패턴 매칭 알고리듬에서 정규 표현식을 인식하기 위해 사용된 결정적 유한 오토마타(Deterministic Finite Automata)를 구성하는 상태가 폭발적으로 증가(states blowup)하고 있다. 이러한 상태의 폭발적 증가 문제를 해결하고 메모리 효율적인 자료 구조를 구현하기 위해 많은 연구가 이루어졌다. 대부분의 연구 결과들에서는 하나의 정규 표현식을 변환한 결정적 유한 오토마톤(Automaton) 내 상태의 수를 감소시키기 위한 효과적인 방안들을 제안하였다. 하지만, 이들 연구 결과는 단일 패턴 내 상태의 수만을 감소시킬 뿐 패턴의 수에 따라 증가하는 상태의 수를 감소시키지 못하는 한계점을 가지고 있다. 본 논문에서는 이를 해결하기 위해 정규 표현식으로 구성된 유한 오토마타(Automata) 상의 상태 병합을 통한 상태 감소 방안을 제안한다. 이는 동일한 입력 문자를 가지는 상태를 병합함으로써 유한 오토마타 상의 상태의 수를 감소시켜, 기존 결정적 유한 오토마타에 비해 평균 40.0%의 메모리 감소 효과를 나타낸다. A pattern matching algorithm plays an important role in traffic identification and classification based on predefined patterns for intrusion detection and prevention. As attacks become prevalent and complex, current patterns are written using regular expressions, called regexes, which are expressed into the deterministic finite automata(DFA) due to the guaranteed worst-case performance in pattern matching process. Currently, because of the increased complexity of regex patterns and their large number, memory-efficient DFA from states reduction have become the mainstay of pattern matching process. However, most of the previous works have focused on reducing only the number of states on a single automaton, and thus there still exists a state blowup problem under the large number of patterns. To solve the above problem, we propose a new state compression algorithm that merges states on multiple automata. We show that by merging states with the same input character on multiple automata, the proposed algorithm can lead to a significant reduction of the number of states in the original DFA by as much as 40.0% on average.