Use Of Regular Expressions Research Articles

Extracting Factual Information about the Pandemic from Open Internet Sources

A large number of different source data is needed for multi-agent models of the spread of infectious diseases. Most of them are not directly accessible. Therefore, one of the key problems to design such models is the development of tools for obtaining data from various sources. This article presents approaches that allow to extract the values of the parameters of the functioning of the simulated society and statistical data on the development of the pandemic from text messages published in the Internet. The proposed method and software implementation provide intelligent search of open source information in the Internet and process of unstructured data. The data collected this way used to set up parameters of mathematical model, which provides ability to study various scenarios and predict progress of the epidemic in concrete regions. The emphasis of the proposed approach is placed on two main technologies. The first is the use of regular expressions. The second is analysis using machine learning methods. The use of the regular expression method allows for high-speed text processing, but its applicability is limited by a strong dependence on the context. Machine learning allows to adapt the information context of the message, but at the same time there is a relatively large amount of time spent on analysis. To improve the accuracy of the analysis and to level the shortcomings of each of these approaches, ways of combining these technologies are proposed. The article presents the obtained results of optimization of algorithms for obtaining the necessary data.

Usage of social media Twitter in the Intervention of Diabetes Mellitus

Social networking sites have been a common forum for exchanging health-related insights and information. This study aims to look at Twitter use in the intervention of diabetes. Specifically, utilizing a prior analysis as a reference, we use a revised approach to analyse trends in the existing use of hash-tags, trending hash-tags, and the incidence of diabetes-related tweets. Our findings indicate that the diabetes population on Twitter has grown significantly over time, as well as proof that this community is becoming more capable of spreading diabetes-related health information. An enhanced system for storing, cleaning, and reviewing Twitter data relevant to diabetes, as well as the use of regular expressions to categorize subsets of tweets, are among our computational contributions. To recognize tweets from diabetic patients, we built a model focused on word embedding and long short- term memory. Based on our findings, our model is working well, since the term embedded with the LSTM classifiers.

Pattern Matching in YARA: Improved Aho-Corasick Algorithm

YARA is a tool for pattern matching used by malware analysts all over the world. YARA can scan files, as well as process memory. It allows us to define sequences of symbols as text strings, hexadecimal strings and regular expressions. However, the use of regular expressions is limited because of the concern that it can slow down the scanning process. In this paper, we analyze the true nature of regular expressions in YARA and their implementation. We have, in fact, discovered several reasons why regular expressions can slow down scanning based on the nature of the used algorithm, Aho-Corasick. We have proposed a new version of this algorithm and have implemented it in the original version of this tool. The experiments are presented, proving that the speed of pattern matching with regular expressions can indeed be improved. In selected cases, the proposed version was about 27% faster than the original version. And in instances where strings were optimized for the original version, their speed was found to be comparable.

Регулярные выражения для обнаружения Web-рекламы на основе автоматического скользящего алгоритма

This paper presents the automation of a Web advertising recognition algorithm, using regular expressions. Currently, the use of regular expressions, optical character recognition, Databases, and automation tests have been critical for multiple Software implementations. The tests were carried out in three Web browsers. As a result, the detection of advertisements in Spanish, that distract attention and that above all extract information from users was achieved. The main feature of the algorithm is that automatic and versatile execution does not require access to the code of the page in question and that in the future it can be an application with background operation. In addition, being supported by optical character recognition gives us acceptable efficiency in detecting advertising.

Regular Expressions for Web Advertising Detection Based on an Automatic Sliding Algorithm

This paper presents the automation of a Web advertising recognition algorithm, using regular expressions. Currently, the use of regular expressions, optical character recognition, Databases, and automation tests have been critical for multiple Software implementations. The tests were carried out in three Web browsers. As a result, the detection of advertisements in Spanish, that distract attention and that above all extract information from users was achieved. The main feature of the algorithm is that automatic and versatile execution does not require access to the code of the page in question and that in the future it can be an application with background operation. Being supported by optical character recognition gives us acceptable efficiency in detecting advertising. Thanks to this identification, it may be possible to generate different applications, both in favor of the user and the brands, always with the aim of improving current online marketing models.

Construction of a semi-automatic ICD-10 coding system

BackgroundThe International Classification of Diseases, 10th Revision (ICD-10) has been widely used to describe the diagnosis information of patients. Automatic ICD-10 coding is important because manually assigning codes is expensive, time consuming and error prone. Although numerous approaches have been developed to explore automatic coding, few of them have been applied in practice. Our aim is to construct a practical, automatic ICD-10 coding machine to improve coding efficiency and quality in daily work.MethodsIn this study, we propose the use of regular expressions (regexps) to establish a correspondence between diagnosis codes and diagnosis descriptions in outpatient settings and at admission and discharge. The description models of the regexps were embedded in our upgraded coding system, which queries a diagnosis description and assigns a unique diagnosis code. Like most studies, the precision (P), recall (R), F-measure (F) and overall accuracy (A) were used to evaluate the system performance. Our study had two stages. The datasets were obtained from the diagnosis information on the homepage of the discharge medical record. The testing sets were from October 1, 2017 to April 30, 2018 and from July 1, 2018 to January 31, 2019.ResultsThe values of P were 89.27 and 88.38% in the first testing phase and the second testing phase, respectively, which demonstrate high precision. The automatic ICD-10 coding system completed more than 160,000 codes in 16 months, which reduced the workload of the coders. In addition, a comparison between the amount of time needed for manual coding and automatic coding indicated the effectiveness of the system-the time needed for automatic coding takes nearly 100 times less than manual coding.ConclusionsOur automatic coding system is well suited for the coding task. Further studies are warranted to perfect the description models of the regexps and to develop synthetic approaches to improve system performance.

A Comparative Analysis of Simulated Annealing and Variable Neighborhood Search in the ATCo Work-Shift Scheduling Problem

This paper deals with the air traffic controller (ATCo) work shift scheduling problem. This is a multi-objective optimization problem, as it involves identifying the best possible distribution of ATCo work and rest periods and positions, ATCo workload and control center changes in order to cover an airspace sector configuration, while, at the same time, complying with ATCo working conditions. We propose a three-phase problem-solving methodology based on the variable neighborhood search (VNS) to tackle this problem. The solution structure should resemble the previous template-based solution. Initial infeasible solutions are built using a template-based heuristic in Phase 1. Then, VNS is conducted in Phase 2 in order to arrive at a feasible solution. This constitutes the starting point of a new search process carried out in Phase 3 to derive an optimal solution based on a weighted sum fitness function. We analyzed the performance in the proposed methodology of VNS against simulated annealing, as well as the use of regular expressions compared with the implementation in the code to verify the feasibility of the analyzed solutions, taking into account four representative and complex instances of the problem corresponding to different airspace sectorings.

DESIGN AND IMPLEMENTATION OF A LANGUAGE SCANNER GENERATOR (KnitAutoGen)

The need for fast, efficient and simple scanner generator that has the primary responsibility to perform efficiently gave rise to this paper. This is due to the fact that, on daily basis new technologies arise which brings a great improvement on the design of computer architecture. However, attention was given to speed, run time and resource availability of the design machine to be used since lexical analysis has an impact on how the compiler works. This paper seeks to develop a lexical analyzer (scanner generator) automatically by specifying the lexemes patterns to a lexical analyzer generator and compiling those patterns into a code that functions as a lexical analyzer. The scanner accepts characters as input and breaks them down to produce tokens by grouping the characters and not deviating from specifications. The project employs one of the different methods of lexical analyzer generator to perform pattern-matching on text using regular expression over a global character set. The paper shows how input is matched and specifies what to do when a pattern is matched. This was achieved with the use of regular expressions (RE) which were converted to non-deterministic finite automata (NFA) or deterministic finite automata (DFA). The regular expression and the regular grammar were thus joined together mathematically. Various results are presented and further work on micro compilers was proposed. Keyword: Scanner generator, Regular expression, KnitAutoGen, DFA, NFA. DOI : 10.7176/CTI/8-02

Automatyzacja wizualizacji grafowej synonimów dla potrzeb dydaktyki języków obcych (na przykładzie przymiotników rosyjskich z prefiksem „без-”/„бес-”)

The article presents methods for application of lexicographic data for the purposes of foreign language teaching and linguistic studies. The automated graph visualization technique is introduced. Both the retrieval and the visualization of linguistic data in form of directed graphs have been conducted with the use of available computer software. The material on which the above-mentioned operations have been performed comprises the group of Russian adjectives with без- / бес- prefixes and the synonymic chains initiated by them. The automated extraction with the use of regular expressions, the steps leading to the edition of the generated database and the import of the final data base into a visualization software are described in the article. The grammatical and semantic characteristics of the sample group of lexemes are not the focus of this article, although further analysis of the selected vocabulary units is planned to be conducted in future studies.

Opioid Overdose Ambulance Runs: How Wisconsin Uses Free Text Data

Objective: 1. Develop an understanding of the benefits and challenges of analyzing free text fields on a population level.2. Observe how a complex surveillance definition can be created from free text fields.3. Observe how an ambulance data system can be used to describe the opioid epidemic.Introduction: In 2016, twelve states received Center for Disease Control and Prevention (CDC) Enhanced State Opioid Overdose Surveillance grants. The purpose of the grant is to explore enhanced data sources to track nonfatal opioid overdoses. One data source is ambulance runs. Wisconsin collects ambulance run information within the Wisconsin Ambulance Runs Data System (WARDS). Around 84% of all Wisconsin administrative services report into this electronic system. This is a timely, robust data system that has not been used previously to examine drug overdoses and presents an analytical challenge as it contains many free text fields.Methods: Wisconsin’s ambulance data system is robust, well-populated, and includes the majority of Emergency Medical Services (EMS) within the state. The analytic challenge with this data is that most of the reported fields are free text, which can be difficult to analyze on a population level. Wisconsin created a case definition using SAS regular expressions to take advantage of the free text fields. A combination of fields (chief complaint, secondary complaint, medications given, and the EMS narrative) were used to determine if the ambulance run was due to an opioid overdose. It was necessary to create a definition that used a diverse combination of phrases as free text fields are prone to spelling errors and there are many phrases used to identify opioid overdoses. It was also necessary to create a definition for unwanted phrases that signal a false positive, for example, “withdrawal”.Results: Wisconsin’s opioid definition uses regular expressions to search for the words “heroin”, “opioid”, “narcan”, or “methadone” (including various spellings). The overdose definition searches for words and phrases like “drug abuse”, “drug use”, “poisoning”, “drug ingestion”, and “overdose”. The medication administered fields are examined for “narcan”. In Wisconsin, the medication is listed every time it is used, so it is possible to determine the number of times Narcan was administered to a single person as well as how many ambulance runs used at least one dose of Narcan. False positives are identified with words and phrases like “withdrawal”, “detox”, and if Narcan was given but there is no indication that the ambulance run was due to drugs. From January 2016 – June 2017, Wisconsin had over 917,000 ambulance runs for people aged 11 years and older. We excluded non-emergency ambulance runs, like medical transports, and so our final denominator was 627,536 runs (32% of all runs were classified as non-emergencies). Suspected opioid overdoses were determined to be 1% of emergency ambulance runs. Narcan was administered in a total of 5,900 runs and the false positive flag picked up 10,399 runs that may not have been due to suspected opioid overdoses. Applying all of these components together, it was determined that in Wisconsin from January 2016 – June 2017, there were 4,041 emergency ambulance runs due to suspected, unintentional opioid overdoses for people 11 years and older (rate of 6 per 1,000 people).Conclusions: The use of regular expressions enables Wisconsin to extend analyses to data systems that contain robust information within free text fields. Within Wisconsin, this has been utilized to enhance opioid overdose surveillance with the use of a rapid data system previously not examined. Ambulance run information is a valuable resource to Wisconsin with the opioid epidemic. By creating case definitions with free text fields, we can quantify ambulance runs on a population level and create linkable analytic data sets to provide a more complete picture of the health of Wisconsin.

Linear Time Membership in a Class of Regular Expressions with Counting, Interleaving, and Unordered Concatenation

Regular Expressions (REs) are ubiquitous in database and programming languages. While many applications make use of REs extended with interleaving ( shuffle ) and unordered concatenation operators, this extension badly affects the complexity of basic operations, and, especially, makes membership NP-hard, which is unacceptable in most practical scenarios. In this article, we study the problem of membership checking for a restricted class of these extended REs, called conflict-free REs , which are expressive enough to cover the vast majority of real-world applications. We present several polynomial algorithms for membership checking over conflict-free REs. The algorithms are all polynomial and differ in terms of adopted optimization techniques and in the kind of supported operators. As a particular application, we generalize the approach to check membership of Extensible Markup Language trees into a class of EDTDs (Extended Document Type Definitions) that models the crucial aspects of DTDs (Document Type Definitions) and XSD (XML Schema Definitions) schemas. Results about an extensive experimental analysis validate the efficiency of the presented membership checking techniques.

A Lightweight Approach for Detection of Code Smells

The accurate removal of code smells from source code supports activities such as refactoring, maintenance, examining code quality etc. A large number of techniques and tools are presented for the specification and detection of code smells from source code in the last decade, but they still lack accuracy and flexibility due to different interpretations of code smell definitions. Most techniques target just detection of few code smells and render different results on the same examined systems due to different informal definitions and threshold values of metrics used for detecting code smells. We present a flexible and lightweight approach based on multiple searching techniques for the detection and visualization of all 22 code smells from source code of multiple languages. Our approach is lightweight and flexible due to application of SQL queries on intermediate repository and use of regular expressions on selected source code constructs. The concept of approach is validated by performing experiments on eight publicly available open source software projects developed using Java and C# programming languages, and results are compared with existing approaches. The accuracy of presented approach varies from 86–97 % on the eight selected software projects.

Helping tools for the regular expression author for test questions in LMS Moodle

Composing regular expressions for test questions is often a difficult thing for the teachers; so many teachers avoid using regular expression questions. Similar problems hinder students learning regular expressions as a part of computer science. There are many programs developed to help composing and learning of the regular expressions, but they are using different forms of regular expression visualization. The goal of this research was to compare efficiency of different forms of regular expression representation for their learning and composing, methods for linking them together and with regular expression text. A set of helping tools for regular expressions authors (as a plugin for Moodle CMS) was developed, using three form of regular expression representation: syntax tree (visualizes expression structure), explanation graph (visualizes paths of expression execution) and text description – and testing tool, showing regular expression match with test strings. Developed instruments was used by students learning regular expressions, the students fill a survey after that. Students were divided into four groups by their year of study and country. Survey shows that different group of students prefer different instruments. Most generally popular ones were explanation graph and regular testing, but even text description – a general outsider – was leading in the group of students from Africa learning in English language. The survey also shows that ability to select part of regular expression representation and see that part selected in other representations and regular expression text was very useful in linking representations together and understanding complex expressions. About a quarter of students used other regular expression construction tools before taking part in this experiment, most of them said that developed tools were better than those they used before. Several teachers, which had used regular expressions in their questions, have written reviews stating that developed instruments make learning regular expressions easier and help debug regular expressions in their questions. So, the survey of students and teachers reviews shows that system of several regular expression representations linked together by subexpression selection is more effective that any particular representation in itself; different classes of users prefer different forms of representation. Including helping tools for regular expression authors in the quiz creation software allows increasing the use of regular expressions for quiz questions and helps test and debug them.

Airborne Behaviour Monitoring Using Regular-Expressions Based Pattern Matching

This paper addresses the problem of target manoeuvre detection and behaviour recognition using string-matching techniques which could bring a great impact to the multiple UAV operations. The overall proposed solution consists of a trajectory analysis tool, which classifies the target position in time into a sequence of driving modes, and a string-matching method. For the latter, two techniques are described and compared: a classical approach based on simple strings and a newly proposed regular-expressions-based method. The advantages obtained from the use of regular expressions are extensively analysed, and results from numerical simulations supporting our conclusions are provided.

A NOVEL PATTERN MATCHING METHOD FOR MEMORY REDUCTION APPLICATIONS USING AC ALGORITHM

Advances in computer networks and storage subsystems continue to push the rate at which data streams must be processed between and within computer systems. Meanwhile, the content of such data streams is subjected to ever increasing scrutiny, as components at all levels mine the streams for patterns that can trigger time-sensitive action. The problem of discovering credit card numbers, currency values, or telephone numbers requires a more general specification mechanism. While there is a well developed theory for regular expressions and their implementation via Finite-State Machines (FSMs), the use of regular expressions for high-performance pattern matching is more difficult and is an area of ongoing research. In this Paper, a memory-efficient pattern matching algorithm which can significantly reduce the number of states and transitions by merging pseudo-equivalent states while maintaining correctness of string matching. In addition, the new algorithm is complementary to other memory reduction approaches and provides further reductions in memory needs.

Automated Assessment of Free Text Questions for MOOC Using Regular Expressions

Presently, most platforms used on the selection of Massive Open Online Courses (MOOC) available online have various automated methods of assessment. These type of tools are based on applications that analyze the answers using a pre-correction algorithm. Most of these programs run several types of automatic assessment, but the possible use of the technology for each of them differs with respect to the kind of automation applied. The role of technology in the objective test for online education has become extremely common, so it can be found in various MOOC platforms with this type of questions or quizzes, because the assessment system can be fully computerized (from the test design to its correction and reporting). However, not all of the assessment instruments can be easily implemented in automatic mode with the use of technology. This paper seeks to research and clarify a type of assessment tool in which the use of technologies is quite low, namely, the essay question type and within them, the short answer question type or free text question type, using regular expressions. The large number of students who would be in MOOC prevents a teacher from assessing responses of thousands of students in a finite time without the aid of technology. This research analyzes the results of an MOOC from hundreds of students to verify that the use of regular expressions in an MOOC platform is not only recommended but also necessary.

Hardware acceleration of regular expression repetitions in deep packet inspection

Network Intrusion Detection Systems (NIDS) make extensive use of regular expressions (regexes) as attack signatures. Such expressions can be handled in hardware using a bit-parallel (BP) architecture based on the Glushkov non-deterministic finite automata (NFA). However, many expressions contain constrained {min, max} repetitions which first need to be unrolled so that they can be handled by the standard BP system. Such unrolling often leads to an excessive memory requirement which makes handling of such regexes unfeasible. This study presents a solution, based on the standard BP architecture, which incorporates a counting mechanism that renders unrolling unnecessary. As a result, many regexes, which were previously unsuitable for the standard BP system, can now be efficiently handled. Unlike many other approaches, this architecture is dynamically reconfigurable thanks to its memory, rather than logic, based engine. This is important as NIDS rule sets are regularly updated. It can also handle repetition of both single and multi-symbol sub-expressions.

A Quick Query Methods for ONS System based on EPC

EPCGlobal proposed for the domain name (DNS) service based on the object name service (ONS) design standards, prone to slow query response ONS system, data logging, storage space, cannot meet a single product inquiry and other issues. This paper presents a separate DNS service, the use of regular expressions and match data records stored ONS system design and implementation, test results show that independent of the DNS service ONS system has a data record storage space and query response speed, meet single product inquiries and so on.

BIXSAN

Proliferation of social networking sites, and web applications which deliver dynamic content to the clients have increased the user created HTML content in the World Wide Web. This user-created HTML content can be a notorious vector for Cross-Site Scripting,(XSS) attacks. XSS attacks have the ability to target websites, steal confidential information of the users, and hijack their accounts, etc. XSS attacks are launched to exploit the vulnerabilities of the poorly developed application code and data processing systems. In particular, improper validation of user created content and un-sanitized custom error messages introduce vulnerability for XSS attacks. It is a challenging task for any security mechanism to filter out only the harmful HTML content and retain safe content with high fidelity and robustness. This has motivated us to develop a mechanism that filters out the harmful HTML content, and allows safe HTML. The existing solutions to XSS attack include use of regular expressions to detect the presence of dynamic content and client side filtering mechanisms such as Noscript and Noxes tool. The drawbacks of these solutions are low fidelity and disallowing of benign HTML. In order to overcome these drawbacks BIXSAN, a Browser Independent XSS SANitizer for prevention of XSS attacks is proposed in this paper. BIXSAN includes the proposition of three pronged strategy. These strategies are as follows: Firstly the use of complete HTML parser is proposed rather than approximating the behavior of parser. The advantage of using complete HTML parser is that it offers high fidelity. Secondly the use of modified browser, viz., JavaScript Tester is proposed to detect the presence of JavaScript for filtering it out. Thirdly, identification of static tags is proposed for allowing the benign HTML. Further, BIXSAN includes the proposition of a parse tree generator at client side browser to reduce the anomalous behavior of browsers. BIXSAN was experimented in various browsers such as Opera, Netscape, Internet Explorer (IE), and Firefox and found to work for all the browsers. From the experiments conducted it has been found that the proposed BIXSAN prevents the injection of XSS attack code successfully. Further, it has been verified that BIXSAN reduces the anomalous behavior of browse.

Regular expression containment

We present a new sound and complete axiomatization of regular expression containment. It consists of the conventional axiomatization of concatenation, alternation, empty set and (the singleton set containing) the empty string as an idempotent semiring, the fixed- point rule E * = 1 + E × E * for Kleene-star, and a general coinduction rule as the only additional rule. Our axiomatization gives rise to a natural computational interpretation of regular expressions as simple types that represent parse trees, and of containment proofs as coercions . This gives the axiom- atization a Curry-Howard-style constructive interpretation: Containment proofs do not only certify a language-theoretic contain- ment, but, under our computational interpretation, constructively transform a membership proof of a string in one regular expression into a membership proof of the same string in another regular expression. We show how to encode regular expression equivalence proofs in Salomaa's, Kozen's and Grabmayer's axiomatizations into our containment system, which equips their axiomatizations with a computational interpretation and implies completeness of our axiomatization. To ensure its soundness, we require that the computational interpretation of the coinduction rule be a hereditarily total function. Hereditary totality can be considered the mother of syn- tactic side conditions: it "explains" their soundness, yet cannot be used as a conventional side condition in its own right since it turns out to be undecidable. We discuss application of regular expressions as types to bit coding of strings and hint at other applications to the wide-spread use of regular expressions for substring matching, where classical automata-theoretic techniques are a priori inapplicable. Neither regular expressions as types nor subtyping interpreted coercively are novel per se . Somewhat surprisingly, this seems to be the first investigation of a general proof-theoretic framework for the latter in the context of the former, however.