N-gram Matching Research Articles

Android malware dataset construction methodology to minimize bias–variance​ tradeoff

Recently, research on Android malware categorization and detection is increasingly directed toward proposing different learned models based on various features of Android apps and machine learning algorithms. For the implementation of such modeling, properly constructing a dataset is no less important than selecting a suitable algorithm. The present study examines dataset construction using Dexofuzzy and proposes methods to determine the degree of bias and variance in the process and minimize the noise in sample set labeling where there is a possibility that even the same samples can be differently labeled. The method proposed in the present study goes beyond existing dataset construction methods relying on label data provided by antivirus vendors to include an effective approach to construct new types of datasets built on unified labels combined with opcode morphology. Based on newly constructed datasets, a flexible dataset, which allows overfitting and underfitting to be considered, was obtained via N-Gram and M-Partial Matching. This flexible dataset was then subjected to clustering, and the resultant clustering performance was evaluated.

Gazetteer based unsupervised learning approach for location extraction from complaint tweets

In the era of smart city, social media platform like twitter has become the most powerful medium for sharing urban issues. Extracting the locations from the complaint tweets is very much helpful to local government for monitoring and performing spatial analysis of civic issues at city level. We propose a natural language processing and regular expression based system to identify probable locations of complaints in terms of landmarks (hospital, mall, railway station), street level (e.g. road marg), region (e.g. nagar, colony) and suburb. Our system has used n-gram matching, string similarity and spatial proximity measures to confirm and disambiguate the extracted locations using GeoNames and OSM gazetteer data. Proposed system has achieved the F-score of 93% which is very promising. Locations are also displayed on map for better situational awareness across the city.

A novel alignment-free DNA sequence similarity analysis approach based on top-k n-gram match-up

DNA sequence similarity analysis is an essential task in computational biology and bioinformatics. In nearly all research that explores evolutionary relationships, gene function analysis, protein structure prediction and sequence retrieving, it is necessary to perform similarity calculations. As an alternative to alignment-based sequence comparison methods, which result in high computational cost, alignment-free methods have emerged that calculate similarity by digitizing the sequence in a different space. In this paper, we proposed an alignment-free DNA sequence similarity analysis method based on top-k n-gram matches, with the prediction that common repeating DNA subsections indicate high similarity between DNA sequences. In our method, we determined DNA sequence similarities by measuring similarity among feature vectors created according to top-k n-gram match-up scores without the use of similarity functions. We applied the similarity calculation for three different DNA data sets of different lengths. The phylogenetic relationships revealed by our method show that our trees coincide almost completely with the results of the MEGA software, which is based on sequence alignment. Our findings show that a certain number of frequently recurring common sequence patterns have the power to characterize DNA sequences.

The Date of Alphonsus, Emperor of Germany: The Evidence of Unique N-Gram Matches

The Date of Alphonsus, Emperor of Germany: The Evidence of Unique N-Gram Matches

Wide-scope biomedical named entity recognition and normalization with CRFs, fuzzy matching and character level modeling.

We present a system for automatically identifying a multitude of biomedical entities from the literature. This work is based on our previous efforts in the BioCreative VI: Interactive Bio-ID Assignment shared task in which our system demonstrated state-of-the-art performance with the highest achieved results in named entity recognition. In this paper we describe the original conditional random field-based system used in the shared task as well as experiments conducted since, including better hyperparameter tuning and character level modeling, which led to further performance improvements. For normalizing the mentions into unique identifiers we use fuzzy character n-gram matching. The normalization approach has also been improved with a better abbreviation resolution method and stricter guideline compliance resulting in vastly improved results for various entity types. All tools and models used for both named entity recognition and normalization are publicly available under open license.

GPU Based N-Gram String Matching Algorithm with Score Table Approach for String Searching in Many Documents

String searching in documents has become a tedious task with the evolution of Big Data. Generation of large data sets demand for a high performance search algorithm in areas such as text mining, information retrieval and many others. The popularity of GPU’s for general purpose computing has been increasing for various applications. Therefore it is of great interest to exploit the thread feature of a GPU to provide a high performance search algorithm. This paper proposes an optimized new approach to N-gram model for string search in a number of lengthy documents and its GPU implementation. The algorithm exploits GPGPUs for searching strings in many documents employing character level N-gram matching with parallel Score Table approach and search using CUDA API. The new approach of Score table used for frequency storage of N-grams in a document, makes the search independent of the document’s length and allows faster access to the frequency values, thus decreasing the search complexity. The extensive thread feature in a GPU has been exploited to enable parallel pre-processing of trigrams in a document for Score Table creation and parallel search in huge number of documents, thus speeding up the whole search process even for a large pattern size. Experiments were carried out for many documents of varied length and search strings from the standard Lorem Ipsum text on NVIDIA’s GeForce GT 540M GPU with 96 cores. Results prove that the parallel approach for Score Table creation and searching gives a good speed up than the same approach executed serially.

Automated Scoring of Students’ English-to-Chinese Translations of Three Text Types

This study attempts to construct automated scoring models for Chinese EFL (English as a Foreign Language) learners’ English-to-Chinese (E-C) translations in large-scale exams. Our data consisted of 900 human-scored translated texts of three source texts – an expository text, a narrative text and a mixed narrative-argumentative text – with 300 for each source text. Text features were extracted using technologies such as n-gram matching, word alignment and Latent Semantic Analysis. Computer scoring models were constructed using multiple linear regression analysis with text features as independent variables and human-assigned scores as the dependent variable. To determine the number of training texts required to yield the most optimal results, five scoring models were developed with a training set of 50, 100, 130, 150 and 180 texts of each text type, respectively. Results indicated that the correlation coefficients between the model-computed and human-assigned scores were above 0.8 for all five models. The model trained with 130 translated texts performed the best on expository and narrative texts, while that trained with 100 translated texts performed the best on mixed narrative-argumentative texts. Therefore, it is concluded that the text features extracted in this study are effective and that the finalized models can produce reliable scores for Chinese EFL learners’ E-C translations.

Text-mining based localisation of player-specific events from a game-log of cricket

Analysis and visualisation of sports data pertaining to a particular player provides an avenue to study the game of a player (or group of players) in detail, to identify strengths and weaknesses to predict the outcome of a match or select players for a fantasy league. In this paper, we propose to detect and localise events (such as shots) specific to a player or a team and present visualisations that are easy to interpret. For this, we use a textual log of the match (or the detailed ball-by-ball commentary) of a game of cricket as input. We perform hierarchical n-gram matching to detect events of interest for the player(s) specified and match them with the location (on the field). We propose the disambiguation of action and intention verbs and resolve location through auxiliary information to improve the accuracy of detection and localisation of events. The results obtained on testing the proposed system on multiple game logs demonstrate that the accuracy of detection and localisation is comparable with that of manual detection and localisation.

Text-mining based localisation of player-specific events from a game-log of cricket

Analysis and visualisation of sports data pertaining to a particular player provides an avenue to study the game of a player (or group of players) in detail, to identify strengths and weaknesses to predict the outcome of a match or select players for a fantasy league. In this paper, we propose to detect and localise events (such as shots) specific to a player or a team and present visualisations that are easy to interpret. For this, we use a textual log of the match (or the detailed ball-by-ball commentary) of a game of cricket as input. We perform hierarchical n-gram matching to detect events of interest for the player(s) specified and match them with the location (on the field). We propose the disambiguation of action and intention verbs and resolve location through auxiliary information to improve the accuracy of detection and localisation of events. The results obtained on testing the proposed system on multiple game logs demonstrate that the accuracy of detection and localisation is comparable with that of manual detection and localisation.

Keyphrase based Evaluation of Automatic Text Summarization

The development of methods to deal with the informative contents of the text units in the matching process is a major challenge in automatic summary evaluation systems that use fixed n-gram matching. The limitation causes inaccurate matching between units in a peer and reference summaries. The present study introduces a new Keyphrase based Summary Evaluator KpEval for evaluating automatic summaries. The KpEval relies on the keyphrases since they convey the most important concepts of a text. In the evaluation process, the keyphrases are used in their lemma form as the matching text unit. The system was applied to evaluate different summaries of Arabic multi-document data set presented at TAC2011. The results showed that the new evaluation technique correlates well with the known evaluation systems: Rouge1, Rouge2, RougeSU4, and AutoSummENG MeMoG. KpEval has the strongest correlation with AutoSummENG MeMoG, Pearson and spearman correlation coefficient measures are 0.8840, 0.9667 respectively.

Concept Integration using Edit Distance and N-Gram Match

Information is growing more rapidly on the World Wide Web (WWW) has made it necessary to make all this information not only available to people but also to the machines. Ontology and token are widely being used to add the semantics in data processing or information processing. A concept formally refers to the meaning of the specification which is encoded in a logic-based language, explicit means concepts, properties that specification is machine readable and also a conceptualization model how people think about things of a particular subject area. In modern scenario more ontologies has been developed on various different topics, results in an increased heterogeneity of entities among the ontologies. The concept integration becomes vital over last decade and a tool to minimize heterogeneity and empower the data processing. There are various techniques to integrate the concepts from different input sources, based on the semantic or syntactic match values. In this paper, an approach is proposed to integrate concept (Ontologies or Tokens) using edit distance or n-gram match values between pair of concept and concept frequency is used to dominate the integration process. The proposed techniques performance is compared with semantic similarity based integration techniques on quality parameters like Recall, Precision, FMeasure & integration efficiency over the different size of concepts. The analysis indicates that edit distance value based interaction outperformed n-gram integration and semantic similarity techniques.

Modeling the scholars: Detecting intertextuality through enhanced word-level n-gram matching

The study of intertextuality, or how authors make artistic use of other texts in their works, has a long tradition, and has in recent years benefited from a variety of applications of digital methods. This article describes an approach for detecting the sorts of intertexts that literary scholars have found most meaningful, as embodied in the free Tesserae website . Tests of Tesserae Versions 1 and 2 showed that word-level n-gram matching could recall a majority of parallels identified by scholarly commentators in a benchmark set. But these versions lacked precision, so that the meaningful parallels could be found only among long lists of those that were not meaningful. The Version 3 search described here adds a second stage scoring system that sorts the found parallels by a formula accounting for word frequency and phrase density. Testing against a benchmark set of intertexts in Latin epic poetry shows that the scoring system overall succeeds in ranking parallels of greater significance more highly, allowing site users to find meaningful parallels more quickly. Users can also choose to adjust both recall and precision by focusing only on results above given score levels. As a theoretical matter, these tests establish that lemma identity, word frequency, and phrase density are important constituents of what make a phrase parallel a meaningful intertext.

Expected dependency pair match: predicting translation quality with expected syntactic structure

Recent efforts to develop new machine translation evaluation methods have tried to account for allowable wording differences either in terms of syntactic structure or synonyms/paraphrases. This paper primarily considers syntactic structure, combining scores from partial syntactic dependency matches with standard local n-gram matches using a statistical parser, and taking advantage of N-best parse probabilities. The new scoring metric, expected dependency pair match (EDPM), is shown to outperform BLEU and TER in terms of correlation to human judgments and as a predictor of HTER. Further, we combine the syntactic features of EDPM with the alternative wording features of TERp, showing a benefit to accounting for syntactic structure on top of semantic equivalency features.

Comparison of Stemming and N-gram Matching for Term Conflation in Arabic Text

A stem in Arabic is a root verb form combined with derivational morphemes but with affixes removed. One part of this paper is to repeat the Mixed Stemming experiment, that chooses the use of a word, a stem, or a root for a query term based upon which form has the highest average inverse document frequency value, using more documents and queries. A consistency result has been obtained. The root with weighting method was the superior. The mixed stemming improved binary weighting search results in all cases but did not increase performance over weighted stems or roots. N-gram matching is widely used for term conflation when searching the World Wide Web for something without knowing whether it is stored as singular or plural, as a compound or in the form of several words, in old or new spelling or possibly in wrong orthography. Part two of this paper reports a comparison of Stemming and N-gram matching. The Stem-NGram-Stem Search outperforms (at recall 0.4 and above) the Digrams method when both the queries and documents words are not stemmed. It also suggests that the (Stem-NGram-Stem) Search approach outperforms (at recall 0.4 and above) the Trigrams method (Stem-NGram-Word) when the queries words are stemmed and documents words are not stemmed. The results also suggest that the Trigrams method (Stem-NGram-Word) outperforms the Digrams method when both the queries and documents words are not stemmed. All of the proposed N-gram methods outperform the Word, Stem, and Root index methods for Binary Weighting scheme.

Character contiguity in N-gram-based word matching: the case for Arabic text searching

This work assesses the performance of two N-gram matching techniques for Arabic root-driven string searching: contiguous N-grams and hybrid N-grams, combining contiguous and non-contiguous. The two techniques were tested using three experiments involving different levels of textual word stemming, a textual corpus containing about 25 thousand words (with a total size of about 160KB), and a set of 100 query textual words. The results of the hybrid approach showed significant performance improvement over the conventional contiguous approach, especially in the cases where stemming was used. The present results and the inconsistent findings of previous studies raise some questions regarding the efficiency of pure conventional N-gram matching and the ways in which it should be used in languages other than English.

Applying query structuring in cross-language retrieval

We will explore various ways to apply query structuring in cross-language information retrieval. In the first test, English queries were translated into Finnish using an electronic dictionary, and were run in a Finnish newspaper database of 55,000 articles. Queries were structured by combining the Finnish translation equivalents of the same English query key using the syn-operator of the InQuery retrieval system. Structured queries performed markedly better than unstructured queries. Second, the effects of compound-based structuring using a proximity operator for the translation equivalents of query language compound components were tested. The method was not useful in syn-based queries but resulted in decrease in retrieval effectiveness. Proper names are often non-identical spelling variants in different languages. This allows n-gram based translation of names not included in a dictionary. In the third test, a query structuring method where the Boolean and-operator was used to assign more weight to keys translated through n-gram matching gave good results.

A Comparison of Spelling-Correction Methods for the Identification of Word Forms in Historical Text Databases

This paper discusses the application of algorithmic spelling-correction techniques to the identification of those words in databases of sixteenth-, seventeenth, and eighteenth- century English texts that are most similar to a query word in modern English. The experiments involved the n-gram matching, phonetic and non-phonetic coding, and dynamic-programming methods for spelling correction, and demonstrate the general effectiveness of this approach to the identification of historical word forms