Large Set Of Patterns Research Articles

BlinkFill

The recent Programming By Example (PBE) techniques such as FlashFill have shown great promise for enabling end-users to perform data transformation tasks using input-output examples. Since examples are inherently an under-specification, there are typically a large number of hypotheses conforming to the examples, and the PBE techniques suffer from scalability issues for finding the intended program amongst the large space. We present a semi-supervised learning technique to significantly reduce this ambiguity by using the logical information present in the input data to guide the synthesis algorithm. We develop a data structure InputDataGraph to succinctly represent a large set of logical patterns that are shared across the input data, and use this graph to efficiently learn substring expressions in a new PBE system B link F ill . We evaluate B link F ill on 207 real-world benchmarks and show that B link F ill is significantly faster (on average 41x) and requires fewer input-output examples (1.27 vs 1.53) to learn the desired transformations in comparison to F lash F ill .

Picking Pesky Parameters: Optimizing Regular Expression Matching in Practice

Network security systems inspect packet payloads for signatures of attacks. These systems use regular expression matching at their core. Many techniques for implementing regular expression matching at line rate have been proposed. Solutions differ in the type of automaton used (i.e., deterministic versus non-deterministic) and in the configuration of implementation-specific parameters. While each solution has been shown to perform well on specific rule sets and traffic patterns, there has been no systematic comparison across a large set of solutions, rule sets and traffic patterns. Thus, it is extremely challenging for a practitioner to make an informed decision within the plethora of existing algorithmic and architectural proposals. Moreover, as multi-core processors are becoming popular, many parameters need to be tuned to maximize the multi-core potential. To address this problem, we present a comprehensive evaluation of a broad set of regular expression matching techniques. We consider both algorithmic and architectural aspects. Specifically, we explore the performance, area requirements, and power consumption of implementations targeting multi-core processors and FPGAs using rule sets of practical size and complexity. We present detailed performance results and specific guidelines for determining optimal configurations based on a simple evaluation of the rule set. These guidelines can help significantlywhen implementing regular expression matching systems in practice.

Interior models of earthquake damaged buildings for search and rescue

Survivors trapped in void spaces formed when buildings collapse in an earthquake may be saved if search and rescue (SAR) operations are quick. A novel computational approach aims to provide building information that can guide SAR teams, thus minimizing their risk and accelerating operations. The inputs are an ‘as-built’ BIM model of the building before an earthquake and a partial ‘as-damaged’ BIM model of the exterior components after the earthquake derived from a terrestrial laser scan. A large set of possible collapse patterns is generated before the earthquake. After the event, the pattern with geometry most similar to that of the ‘as-damaged’ exterior BIM can be selected rapidly. This paper details the selection methods, which use least sum of point distances and Modal Assurance Criteria (MAC) algorithms, and illustrates their operation on a series of simulated computer models of collapsed structures, thus demonstrating the potential feasibility of the proposed approach.

Visualizing Sequential Educational Datamining Patterns

The use of educational datamining techniques to elicit patterns in student behavior and learning outcomes can be a useful for the analysis of the effectiveness of teaching strategies and the coordination of study programs. However, results from those techniques are, often, large sets of symbolic patterns, numbering in the thousands, usually presented in text format. This makes them hard to understand which, coupled with the lack of an overall view, hinders a more comprehensive data analysis. The authors propose that information visualization techniques can be used to display relevant information in those patterns in effective ways, allowing decision makers to better insights about the reality at hand. They present a solution built upon two linked views, one based on node-link representations and another a multi-matrix representation. The complementarity of both visualization techniques allows the most important patterns to be immediately apparent, while at the same time permitting their interactive exploration in meaningful ways. The authors performed user tests proving their effectiveness.

Connectivity mapping uncovers small molecules that modulate neurodegeneration in Huntington's disease models.

Huntington’s disease (HD) is a genetic disease caused by a CAG trinucleotide repeat expansion encoding a polyglutamine tract in the huntingtin (HTT) protein, ultimately leading to neuronal loss and consequent cognitive decline and death. As no treatments for HD currently exist, several chemical screens have been performed using cell-based models of mutant HTT toxicity. These screens measured single disease-related endpoints, such as cell death, but had low ‘hit rates’ and limited dimensionality for therapeutic detection. Here, we have employed gene expression microarray analysis of HD samples—a snapshot of the expression of 25,000 genes—to define a gene expression signature for HD from publically available data. We used this information to mine a database for chemicals positively and negatively correlated to the HD gene expression signature using the Connectivity Map, a tool for comparing large sets of gene expression patterns. Chemicals with negatively correlated expression profiles were highly enriched for protective characteristics against mutant HTT fragment toxicity in in vitro and in vivo models. This study demonstrates the potential of using gene expression to mine chemical activity, guide chemical screening, and detect potential novel therapeutic compounds.Key messagesSingle-endpoint chemical screens have low therapeutic discovery hit-rates.In the context of HD, we guided a chemical screen using gene expression data.The resulting chemicals were highly enriched for suppressors of mutant HTT fragment toxicity.This study provides a proof of concept for wider usage in all chemical screening.Electronic supplementary materialThe online version of this article (doi:10.1007/s00109-015-1344-5) contains supplementary material, which is available to authorized users.

AN EFFICIENT AND SYSTEMATIC VIRUS DETECTION PROCESSOR FOR EMBEDDED NETWORK SECURITY

Network security has always been an important issue and its application is ready to perform powerful pattern matching to protect against virus attacks, spam and Trojan horses. However, attacks such as spam, spyware, worms, viruses, and phishing target the application layer rather than the network layer. Therefore, traditional firewalls no longer provide enough protection. However, the solutions in the literature for firewalls are not scalable, and they do not address the difficulty of an antivirus. The goal is to provide a systematic virus detection hardware solution for network security for embedded systems. Instead of placing entire matching patterns on a chip, our solution is based on an antivirus processor that works as much of the filtering information as possible onto a chip. The infrequently accessing off-chip data to make the matching mechanism scalable to large pattern sets. In the first stage, the filtering engine can filter out more than 93.1% of data as safe, using a merged shift table. Only 6.9% or less of potentially unsafe data must be precisely checked in the second stage by the exact-matching engine from off-chip memory. To reduce the memory gap and to improve the performance, we also propose three algorithms are used: 1) a skipping algorithm; 2) a cache method; and 3) a prefetching mechanism.

Evaluation of control strategies for fixed-wing drones following slow-moving ground agents

There are many situations where fixed-wing drones may be required to track ground moving agents, such as humans or cars, which are typically slower than drones. Some control strategies have been proposed and validated in simulations using the average distance between the target and the drone as a performance metric. However, besides the distance metric, energy expenditure of the flight also plays an important role in assessing the overall performance of the flight. In this paper, we propose a new methodology that introduces a new metric (energy expenditure), we compare existing methods on a large set of target motion patterns and present a comparison between the simulation and field experiments on proposed target motion patterns. Using this new methodology we examine the performance of three control strategies: the Lyapunov Guidance Vector Field strategy, the Bearing-only strategy and the Oscillatory strategy. Among the three strategies considered, we demonstrate that the Lyapunov Guidance Vector Field strategy has the best performance for all target motion patterns. Field experiments with fixed-wing drones provide additional insights into the benefits and shortcomings of each strategy in practice.

Speculative parallel pattern matching using stride-k DFA for deep packet inspection

Modern deep packet inspection (DPI) systems match network traffic against a large set of patterns which are defined using regular expressions. Deterministic finite automata (DFA) is generally preferred to parse these regular expressions. However, packets are mostly scanned one byte at a time which becomes a bottleneck for the DPI systems as they are unable to cope up with higher line rates. In this paper, we present an approach which allows a packet to be split up into two chunks. Furthermore, we scan the bytes of each chunk in parallel using speculation and multi-stride (stride-k) DFA. Stride-k DFAs results in fast processing of bytes but leads to high memory usage. Therefore, we propose a transition compression algorithm using alphabet compression table to limit the memory usage of multi-stride DFA. Experimental results show that the speculative parallel pattern matching using stride-k DFA leads to improvement in terms of speedup and latency over traditional DFA matching.

Memory Efficient Algorithm and Architecture for Multi-Pattern Matching

多模式匹配是基于内容检测的网络安全系统的重要功能,同时,它在很多领域具有广泛的应用.实际应用中,高速且性能稳定的大规模模式匹配方法需求迫切,尤其是能够在线实时处理网络包的匹配体系结构.介绍了一种存储有效的高速大规模模式匹配算法及相关体系结构.研究从算法所基于的理论入手,提出了缓存状态机模型,并结合状态机中转换规则分类,提出了交叉转换规则动态生成的匹配算法ACC(Aho-Corasick-CDFA).该算法通过动态生成转换规则降低了生成状态机的规模,适用于大规模模式集.进一步提出了基于该算法的体系结构设计.采用网络安全系统中真实模式集进行的实验结果表明,该算法相比其他状态机类模式匹配算法,可以进一步减少80%~95%的状态机规模,存储空间降低40.7%,存储效率提高近2 倍,算法单硬件结构实现可以达到11Gbps 的匹配速度.;Pattern matching is the main part of content inspection based network security systems, and it is widely used in many other applications. In practice, pattern matching methods for large scale sets with stable performance are in great demand, especially the architecture for online real-time processing. This paper presents a memory efficient pattern matching algorithm and architecture for a large scale set. This paper first proposes cached deterministic finite automata, namely CDFA, in the view of basic theory. By classifying transitions in pattern DFA, a new algorithm, ACC, based on CDFA is addressed. This algorithm can dynamically generate cross transitions and save most of memory resources, so that it can support large scale pattern set. Further, an architecture based on this method is proposed. Experiments on real pattern sets show that the number of transition rules can be reduced 80%~95% than the current most optimized algorithms. At the same time, it can save 40.7% memory space, nearly 2 times memory efficiency. The corresponding architecture in single chip can achieve about 11Gbps matching performance.

SSE Instruction Implemented Multiple String Matching Using Reverse Trie

String matching is an important technique for an intrusion detection system. Recently some algorithms, such as MPSSEFq and SSEHASH based on SIMD instruction, have been proposed to improve the performance of string matching. In this paper a SSEHASH based algorithm RTRIE employing a trie of reverse patterns substring with the help of wsfp instruction in MPSSEFq is discussed. By moving the text pointer from a chunk of 16 continuous characters, RTRIE has a best-case complexity of O(n/16) and optimum performance when searching for large sets of long patterns.

MULTI-GIGABIT PATTERN FOR DATA IN NETWORK SECURITY

In the current scenario network security is emerging the world. Matching large sets of patterns against an incoming stream of data is a fundamental task in several fields such as network security or computational biology. High-speed network intrusion detection systems (IDS) rely on efficient pattern matching techniques to analyze the packet payload and make decisions on the significance of the packet body. However, matching the streaming payload bytes against thousands of patterns at multi-gigabit rates is computationally intensive. Various techniques have been proposed in past but the performance of the system is reducing because of multi-gigabit rates.Pattern matching is a significant issue in intrusion detection systems, but by no means the only one. Handling multi-content rules, reordering, and reassembling incoming packets are also significant for system performance. We present two pattern matching techniques to compare incoming packets against intrusion detection search patterns. The first approach, decoded partial CAM (DpCAM), pre-decodes incoming characters, aligns the decoded data, and performs logical AND on them to produce the match signal for each pattern. The second approach, perfect hashing memory (PHmem), uses perfect hashing to determine a unique memory location that contains the search pattern and a comparison between incoming data and memory output to determine the match. The suggested methods have implemented in vhdl coding and we use Xilinx for synthesis.

A memory-based NFA regular expression match engine for signature-based intrusion detection

Signature-based intrusion detection is required to inspect network traffic at wire-speed. Matching packet payloads against patterns specified with regular expression is a computation intensive task. Hence, the design of hardware accelerator to speed up regular expression matching has been an active research area. A systematic approach to detect regular expression is based on finite automaton. The space-time trade-off between deterministic finite automaton (DFA) and non-deterministic finite automaton (NFA) is well-known. DFA can offer constant throughput but it may suffer from the state explosion problem. Hence, implementation of DFA for large pattern sets on embedded device with limited on-chip memory may not be viable. NFA requires linear space but the throughput can be very low. Implementations of NFA with hardwired circuits can overcome the speed deficiency by exploiting the massive parallelism offered by dedicated hardware circuitries, but this approach does not support efficient dynamic updates. In this paper, we shall present a memory-based architecture for the implementation of NFA to speed up regular expression matching for signature-based intrusion detection. The proposed method supports dynamic updates and offers constant throughput so that it can be used to supplement the existing DFA-based methods in handling large pattern sets.

Fast Deep Packet Inspection with a Dual Finite Automata

Deep packet inspection, in which packet payloads are matched against a large set of patterns, is an important algorithm in many networking applications. Nondeterministic Finite Automaton (NFA) and Deterministic Finite Automaton (DFA) are the basis of existing algorithms. However, both NFA and DFA are not ideal for real-world rule sets: NFA has the minimum storage, but the maximum memory bandwidth; while DFA has the minimum memory bandwidth, but the maximum storage. Specifically, NFA and DFA cannot handle the presence of character sets, wildcards, and repetitions of character sets or wildcards in real-world rule sets. In this paper, we propose and evaluate a dual Finite Automaton (dual FA) to address these shortcomings. The dual FA consists of a linear finite automaton (LFA) and an extended deterministic finite automaton (EDFA). The LFA is simple to implement, and it provides an alternative approach to handle the repetition of character sets and wildcards (which could otherwise cause the state explosion problem in a DFA) without increasing memory bandwidth. We evaluate the automaton in real-world rule sets using different synthetic payload streams. The results show that dual FA can reduce the number of states up to five orders of magnitude while their memory bandwidth is close to minimum.

Trunk muscle activation patterns in subjects with low back pain

nema

Discovery of Delta Closed Patterns and Noninduced Patterns from Sequences

Discovering patterns from sequence data has significant impact in many aspects of science and society, especially in genomics and proteomics. Here we consider multiple strings as input sequence data and substrings as patterns. In the real world, usually a large set of patterns could be discovered yet many of them are redundant, thus degrading the output quality. This paper improves the output quality by removing two types of redundant patterns. First, the notion of delta tolerance closed itemset is employed to remove redundant patterns that are not delta closed. Second, the concept of statistically induced patterns is proposed to capture redundant patterns which seem to be statistically significant yet their significance is induced by their strong significant subpatterns. It is computationally intense to mine these nonredundant patterns (delta closed patterns and noninduced patterns). To efficiently discover these patterns in very large sequence data, two efficient algorithms have been developed through innovative use of suffix tree. Three sets of experiments were conducted to evaluate their performance. They render excellent results when applying to genomics. The experiments confirm that the proposed algorithms are efficient and that they produce a relatively small set of patterns which reveal interesting information in the sequences.

A Scalable High-Performance Virus Detection Processor Against a Large Pattern Set for Embedded Network Security

Contemporary network security applications generally require the ability to perform powerful pattern matching to protect against attacks such as viruses and spam. Traditional hardware solutions are intended for firewall routers. However, the solutions in the literature for firewalls are not scalable, and they do not address the difficulty of an antivirus with an ever-larger pattern set. The goal of this work is to provide a systematic virus detection hardware solution for network security for embedded systems. Instead of placing entire matching patterns on a chip, our solution is a two-phase dictionary-based antivirus processor that works by condensing as much of the important filtering information as possible onto a chip and infrequently accessing off-chip data to make the matching mechanism scalable to large pattern sets. In the first stage, the filtering engine can filter out more than 93.1% of data as safe, using a merged shift table. Only 6.9% or less of potentially unsafe data must be precisely checked in the second stage by the exact-matching engine from off-chip memory. To reduce the impact of the memory gap, we also propose three enhancement algorithms to improve performance: 1) a skipping algorithm; 2) a cache method; and 3) a prefetching mechanism.

Symptom matching for event streams

Enterprise systems produce a vast amount of logging data. This critical and valuable information must be processed automatically for timely system analysis and recovery. As a result of industry demands, a standard database containing known issues has been introduced - a symptom database. Each symptom consists of a rule pattern and corresponding solutions. Patterns used for symptom identification are encoded as a XPath expression and matched against a stream of events in a standardised WSGI format common base event. The ability of an efficient matching for symptom patterns has been raised as an important requirement by industries. The authors present a real-time symptom identification in a stream of events. The implementation will allow multiple autonomic computing components such as self-monitoring sensors to effectively match known patterns in large datasets in run time. Unlike current state of the art approaches, the proposed solution allows users to define patterns using all the complex XPath functions in addition to standard numeric and Boolean operators. In particular, it was aimed at efficient simultaneous matching of a large set of XPath-based symptom patterns against a high-volume event stream, which is crucial for symptom identification but was not addressed efficiently by currently available XPath-matching engines.

Tandem Mass Spectrometry: A New Platform for Fluxomi

C substrates could be generated using gas chromatography coupled tandem MS (GC-MS/MS) and the GC-MS/MS data could be used to derive metabolic fluxes with similar accuracy and precision to those obtained using NMR and significantly better than those using full-scan GC-MS. Later, Kiefer et al. [3] reported the determination of carbon labeling distribution of intracellular metabolites for fluxomics using liquid chromatography tandem MS (LC-MS/MS) and was able to determine the natural abundances of the carbon fractions m and m+1 from six phosphorylated metabolites from glycolysis and pentose phosphate pathway in E. coli with high accuracy. More recently, Ruhl et al. [4] exploited the measurement of the 13 C patterns of totally 27 intact and 19 fragmented metabolites from the central metabolic pathways in B. subtilis using LC-MS/MS, and the metabolic fluxes calculated using the LC-MS/MS data showed higher precision than those performed using solely full-scan LC-MS data [4]. Most importantly, these research demonstrated that tandem MS, especially LC-MS/MS that does not require chemical derivation like GC-MS/MS, can provide direct measurements of large sets of isotopic labeling patterns from intracellular intermediates and thus allow resolving fluxes for dynamic systems (non-steady state) and large-scale complex metabolic pathways, which are infeasible with the traditional NMR and MS based fluxomics methods as those rely on indirect measurements of the abundant proteinogenic amino acids and thus require long labeling experimental time for steady state to allow the incorporation of isotopes into proteins and can only derive fluxes for the central metabolic pathways that have the key nodes linked with protein biosynthesis. Given the promising applications of tandem MS for fluxomics, however, the technical aspects for isotopomer measurements have to be stressed. First, the methods for metabolite extraction and separation need to be developed or optimized to increase the coverage of detection of various catalogues of metabolites considering the diversity of the physico-chemical properties of the metabolome in an organism. Second, the tandem MS instrumental parameters such as the collision energy for fragmentation, the resolution of mass filter and the number of multiple reaction monitoring (MRM) etc. need to be optimized to capture as many as possible informative daughter mass spectra that are useful for solving the mathematical model and minimize the acquisition of redundant daughter mass spectra that complicate the mathematical

Monte Carlo simulation of the n_TOF Total Absorption Calorimeter

The n_TOF Total Absorption Calorimeter (TAC) is a 4π BaF2 segmented detector used at CERN for measuring neutron capture cross-sections of importance for the design of advanced nuclear reactors. This work presents the simulation code that has been developed in GEANT4 for the accurate determination of the detection efficiency of the TAC for neutron capture events. The code allows to calculate the efficiency of the TAC for every neutron capture state, as a function of energy, crystal multiplicity, and counting rate. The code includes all instrumental effects such as the single crystal detection threshold and energy resolution, finite size of the coincidence time window, and signal pile-up.The results from the simulation have been validated with experimental data for a large set of electromagnetic de-excitation patterns: β-decay of well known calibration sources, neutron capture reactions in light nuclei with well known level schemes like natTi, reference samples used in (n,γ) measurements like 197Au and experimental data from an actinide sample like 240Pu. The systematic uncertainty in the determination of the detection efficiency has been estimated for all the cases. As a representative example, the accuracy reached for the case of 197Au(n,γ) ranges between 0.5% and 2%, depending on the experimental and analysis conditions. Such a value matches the high accuracy required for the nuclear cross-section data needed in advanced reactor design.

Sequence-Specific Ultrasonic Cleavage of DNA

We investigated the phenomenon of ultrasonic cleavage of DNA by analyzing a large set of cleavage patterns of DNA restriction fragments using polyacrylamide gel electrophoresis. The cleavage intensity of individual phosphodiester bonds was found to depend on the nucleotide sequence and the position of the bond with respect to the ends of the fragment. The relative intensities of cleavage of the central phosphodiester bond in 16 dinucleotides and 256 tetranucleotides were determined by multivariate statistical analysis. We observed a remarkable enhancement of the mean values of the relative intensities of cleavage (cleavage rates) in phosphodiester bonds following deoxycytidine, which diminished in the row of dinucleotides: d(CpG) > d(CpA) > d(CpT) >> d(CpC). The cleavage rates for all pairs of complementary dinucleotides were significantly different from each other. The effect of flanking nucleotides in tetranucleotides on cleavage rates of all 16 types of central dinucleotides was also statistically significant. The sequence-dependent ultrasonic cleavage rates of dinucleotides are consistent with reported data on the intensity of the conformational motion of their 5′-deoxyribose. As a measure of local conformational dynamics, cleavage rates may be useful for characterizing functional regions of the genome.