Occurrence Matrix Research Articles

From Feature Engineering and Topics Models to Enhanced Prediction Rates in Phishing Detection

Phishing is a type of fraud attempt in which the attacker, usually by e-mail, pretends to be a trusted person or entity in order to obtain sensitive information from a target. Most recent phishing detection researches have focused on obtaining highly distinctive features from the metadata and text of these e-mails. The obtained attributes are then used to feed classification algorithms in order to determine whether they are phishing or legitimate messages. In this paper, it is proposed an approach based on machine learning to detect phishing e-mail attacks. The methods that compose this approach are performed through a feature engineering process based on natural language processing, lemmatization, topics modeling, improved learning techniques for resampling and cross-validation, and hyperparameters configuration. The first proposed method uses all the features obtained from the Document-Term Matrix (DTM) in the classification algorithms. The second one uses Latent Dirichlet Allocation (LDA) as a operation to deal with the problems of the “curse of dimensionality”, the sparsity, and the text context portion included in the obtained representation. The proposed approach reached marks with an F1-measure of 99.95% success rate using the XGBoost algorithm. It outperforms state-of-the-art phishing detection researches for an accredited data set, in applications based only on the body of the e-mails, without using other e-mail features such as its header, IP information or number of links in the text.

Marathi Document: Similarity Measurement using Semantics-based Dimension Reduction Technique

Textual data is increasing exponentially and to extract the required information from the text, different techniques are being researched. Some of these techniques require the data to be presented in the tabular or matrix format. The proposed approach designs the Document Term Matrix for Marathi (DTMM) corpus and converts unstructured data into a tabular format. This approach has been called DTMM in this paper and it fails to consider the semantics of the terms. We propose another approach that forms synsets and in turn reduces dimensions to formulate a Document Synset Matrix for Marathi (DSMM) corpus. This also helps in better capturing the semantics and hence is context-based. We abbreviate and call this approach as DSMM and carry out experiments for document-similarity measurement on a corpus consisting of more than 1200 documents, consisting of both verses as well as proses, of Marathi language of India. Marathi text processing has been largely an untouched area. The precision, recall, accuracy, F1-score and error rate are used to prove the betterment of the proposed technique.

A Goal Programming Model for BWM

In this paper we deal with the problem of addition of new documents in collection when documents are represented in lower dimensional space by concept indexing. Concept indexing (CI) is a method of feature construction that is relying on concept decomposition of term-document matrix. By using CI original representations of documents are projected on the space spread by centroids of clusters, which are called concept vectors. This problem is especially interesting for application on World Wide Web. Proposed methods are tested for the task of information retrieval. Vectors on which the projection is done in the process of dimension reduction are constructed on the basis of representations of all documents in the collection, and computation of the new representations in the space of reduced dimension demands recomputation of concept decomposition. The solution to this problem is the development of methods which will give approximate representation of newly added documents in the space of reduced dimension. In the paper are introduced two methods for addition of new documents in the space of reduced dimension. In the first method there no addition of new index terms and added documents are represented by existing list of index terms, while in the second method list of index terms is extended and representations of documents and concept vectors are extended in dimensions of newly added terms. It is shown that representation of documents by extended list of index terms does not improve performance of information retrieval significantly. Povzetek: Predstavljeni sta dve metodi konceptualnega indeksiranja dokumentov.

Lemon Leaf Disease Detection and Classification using SVM and CNN

India is a developing country and agriculture has always played a major role in bolstering the country’s economic growth. Due to various factors like industrialization, mechanization and globalization, the green fields are facing complications. So, identifying the plant disease incorrectly will lead to a huge loss of both quantity and quality of the product and it will also incur loss in time and money. Hence, identifying the condition of the plant plays a major role for successful cultivation. Now a day’s image processing technique is being employed as a focal technique for diagnosing the various features of the crop. The image processing techniques can be used for identification of the plant disease and hence classify the plant disease. Generally, the symptoms of the disease are observed on leaves, stems, flowers etc. Here, the leaves of the affected plant are used for the identification and classification of the disease. Leaf image is captured using a smart phone as the first step and then they are processed to determine the condition of the plant. Identification of plant disease follows the steps like loading the image of the plant leaf, histogram equalization for enhancing contrast of the image, segmentation process by using Lab color space model, extracting features of the segmented image using GLCM (Grey Level Co- occurrence Matrix) and finally classification of leaf disease by using MCSVM (Multi Class Support Vector Machine).This procedure obtained an accuracy percentage of 83.6%.Also, it takes long training time for large datasets. To improve the accuracy of the detection and the classification of the plants, Convolutional Neural Network (CNN) is used. The main advantage of CNN is that it automatically detects the main features of the input without any supervision of human. In CNN identification of disease follow the steps like loading the image as the input image, convolution of the feature map and finally max pooling the layers to calculate the features of the image in detail. The plant diseases are classified with an accuracy of 93.8 %.

INNV-15. CLINICAL DATA THAT MATTERS: A DISTILLATION OF NEURO-ONCOLOGY CLINICAL TRIAL INCLUSION CRITERIA USING MACHINE LEARNING

Abstract INTRODUCTION Neuro-oncologic conditions have dismal outcomes, ineffective treatments, poor access to clinical trials, and variability in care. Clinical trials do not capture a patient’s complete journey and are restricted to select populations. ‘Real-world-evidence’ (RWE) attempts to inform point of care decisions through routine collection of data with a clinical-trial-like rigor. RWE complements existing knowledge through broad patient participation, collection throughout disease course, and creation of large multidimensional datasets “knowledge network of disease” 1,2. RWE implementation is hindered by unstructured data, uncertainty of relevant features, and semantic heterogeneity. Clinical attributes were selected from trial inclusion criteria and prioritized for structuring in clinic notes for abstraction. METHOD We queried Clinicaltrials.gov from 1/1/2018-12/31/2018, refined to North America, recruiting, interventional, and adult. Meningioma, pituitary, glioblastoma, astrocytoma, oligodendroglioma, and ependymoma were chosen based on incidence3. Lymphoma and nerve sheath tumors were omitted. “Brain tumor” and “glioma” were added. ‘K-nearest-neighbor’ tokenization parsed inclusion criteria4. Document term matrix (n-gram) converted text to vectors5. A generative probabilistic model using ‘Latent Dirichlet Allocation’ plotted words into 10 clusters6. Hierarchal clustering was used to compare histology with terms. RESULTS 401 trials parsed into 3676 statements and 4008 keywords. 10 clusters of terms were similarly distributed amongst histologies, suggesting generalizability across tumor types. Cluster revealed 8 categories: 1) Time: enrollment; 2) Performance status: KPS; 3) Testing: mutations, upper limit of normal, routine hematologic laboratory assays; 4) Imaging: extent of surgery; 5) Pregnancy/childbearing; 6) Tumor grade; 7) Treatment history: recurrence, chemotherapy, radiation, time; 8) Informed consent CONCLUSIONS Dissecting the compendium of clinical trials using machine learning can identify general parameters for trial enrollment to guide RWE clinical collection. Using practical definitions of the most germane trial data, specific information can be sought after and defined to improve research quality, maximize research yields and improve patient care whilst minimizing wasted research and clinical endeavors.

Relative Spectral Difference Occurrence Matrix: A Metrological Spectral-Spatial Feature for Hyperspectral Texture Analysis

We develop a spectral-spatial feature, Relative Spectral Difference Occurrence Matrix (RSDOM) for hyperspectral texture recognition. Inspired by Julesz's conjecture, the proposed feature is based on spectral difference approach and respects the metrological constraints. It simultaneously considers the distribution of spectra and their spatial arrangement in the hyperspectral image. The feature is generic and adapted for any number of spectral bands and range. We validate our proposition by applying a classification scheme on the HyTexiLa database. An accuracy comparable to local binary pattern approach is obtained, but at a much reduced cost due to the extremely small feature size.

Fake news detection within online social media using supervised artificial intelligence algorithms

Along with the development of the Internet, the emergence and widespread adoption of the social media concept have changed the way news is formed and published. News has become faster, less costly and easily accessible with social media. This change has come along with some disadvantages as well. In particular, beguiling content, such as fake news made by social media users, is becoming increasingly dangerous. The fake news problem, despite being introduced for the first time very recently, has become an important research topic due to the high content of social media. Writing fake comments and news on social media is easy for users. The main challenge is to determine the difference between real and fake news. In this paper, a two-step method for identifying fake news on social media has been proposed, focusing on fake news. In the first step of the method, a number of pre-processing is applied to the data set to convert un-structured data sets into the structured data set. The texts in the data set containing the news are represented by vectors using the obtained TF weighting method and Document-Term Matrix. In the second step, twenty-three supervised artificial intelligence algorithms have been implemented in the data set transformed into the structured format with the text mining methods. In this work, an experimental evaluation of the twenty-three intelligent classification methods has been performed within existing public data sets and these classification models have been compared depending on four evaluation metrics.

Handheld Device Based on Image Processing Technique for Detecting Multiple Diseases of Apple Leaves

The present invention discloses a handheld device for multiple disease detection from apple leaf and method thereof. An algorithm is developed in combination with image processing and gray level co occurrence matrix for the classification of normal leaf and diseased apple leaf. The device performs image processing by segmentation of the image and then by using extracted features. The classification and detection of these diseases impart an early solution to the farmers leading less harm to the apple crops. Conventional Detection of the diseases through naked eye can sometimes be faulty. Therefore the device of present invention helps farmers to detect the accurate diseases and provide timely solutions for the same. The present invention increases the throughput and reduces subjectiveness of the previously used conventional methods by proving early and precise disease detection from apple leaves.

EXTRACTION OF FETAL FEATURES FROM B MODE ULTRASONOGRAMS FOR EFFICIENT DIAGNOSIS OF DOWN SYNDROME IN FIRST AND SECOND TRIMESTER

Segmentation of ultrasound images has been found to be a tedious task due to the presence of speckle and other artifacts. The random nature of the multiplicative speckle noise and lack of demarcation of information in ultrasound images makes the segmentation a highly complex one. In this paper a modified watershed based method has been proposed for segmentation of features from Ultrasound images towards efficient diagnosis of Down Syndrome in first and second trimester. The pixels are grouped based on the pixel differences and the co- occurrence matrix is formed based on the energy and contrast. If global scheme is adopted for segmentation the high frequency edges may appear as artifacts. Hence to overcome this the wavelet transform of co-occurrence matrix is obtained and each decomposed band is subjected to an averaging filter. The process bands are thresholded using Otsu’s method and a binary image is obtained. The isolated pixels are removed by using suitable morphological operations. Then inverse wavelet transform is performed to obtain the image skeleton. The resultant image is subjected to watershed segmentation using gradients. Using the above mentioned approach we can see that the regions of interest are clearly segmented and is producing reproducible results.

Human Behavior Understanding in Big Multimedia Data Using CNN based Facial Expression Recognition

Human behavior analysis from big multimedia data has become a trending research area with applications to various domains such as surveillance, medical, sports, and entertainment. Facial expression analysis is one of the most prominent clues to determine the behavior of an individual, however, it is very challenging due to variations in face poses, illuminations, and different facial tones. In this paper, we analyze human behavior using facial expressions by considering some famous TV-series videos. Firstly, we detect faces using Viola-jones algorithm followed by tracking through Kanade-Lucas-Tomasi (KLT) algorithm. Secondly, we use histogram of oriented gradients (HOG) features with support vector machine (SVM) classifier for facial recognition. Next, we recognize facial expressions using the proposed light-weight convolutional neural network (CNN). We utilize data augmentation techniques to overcome the issue of appearance of faces from different views and lightening conditions in video data. Finally, we predict human behaviors using an occurrence matrix acquired from facial recognition and expressions. The subjective and objective experimental evaluations prove better performance for both facial expression recognition and human behavior understanding.

Using Text Mining and Data Mining Techniques for Applied Learning Assessment

In a society where first hand work experience is greatly valued many universities or institutions of higher education have designed their Quality enhancement plan (QEP) to address student applied learning. This paper is the results of a university’s QEP plan, called Experiencing Transformative Education Through Applied Learning or ETEAL. This paper will highlight the research that was conducted using text mining and data mining techniques to analyze a dataset of 672 student evaluations collected from 40 different applied learning courses from fall 2013 to spring 2015, in order to evaluate the impact on instructional practice and student learning. Text mining techniques are applied through the NVivo text mining software to find the 100 most frequent terms to create a document-term matrix in Excel. Then, the document-term matrix is merged with the manual interpretation scores received to create the applied learning assessment data. Lastly, data mining techniques are applied to evaluate the performance, including Random Forest, K-nearest neighbors, Support Vector Machines (with linear and radial kernel), and 5-fold cross-validation. Our results show that the proposed text mining and data mining approach can provide prediction rates of around 67% to 85%, while the decision fusion approach can provide an improvement of 69% to 86%. Our study demonstrates that automatic quantitative analysis of student evaluations can be an effective approach to applied learning assessment.

An Automatic Diagnosis Scheme for Gliomas Using Gray Level Co- occurrence Matrix, Histogram of Oriented Gradient,Discrete Wavelet Transformation, Intensity-based Features andHierarchical PSO-SVM

An Automatic Diagnosis Scheme for Gliomas Using Gray Level Co- occurrence Matrix, Histogram of Oriented Gradient,Discrete Wavelet Transformation, Intensity-based Features andHierarchical PSO-SVM

The analysis of source code plagiarism in basic programming course

The purpose of this research is to recognize the types of plagiarism that occurred in basic programming courses as a preliminary research in developing an intelligent system to detect source plagiarism in programming class. This research use a hierarchical clustering method which is implemented in term-document matrices. In this research, we propose the use of keywords in the programming language informing the term-document matrix. The result is normalized and then we use Euclidean distance to normalize data for similarity measurement. The result shows that this method is quite effective to classify objects of data indicated by plagiarism. Some types of plagiarism which are mostly occurred are formatting source code, changing comment, renaming identifier, and adding statements or redundant variables.

Task recommender system using semantic clustering to identify the right personnel

PurposeText mining is growing in importance proportionate to the growth of unstructured data and its applications are increasing day by day from knowledge management to social media analysis. Mapping skillset of a candidate and requirements of job profile is crucial for conducting new recruitment as well as for performing internal task allocation in the organization. The automation in the process of selecting the candidates is essential to avoid bias or subjectivity, which may occur while shuffling through thousands of resumes and other informative documents. The system takes skillset in the form of documents to build the semantic space and then takes appraisals or resumes as input and suggests the persons appropriate to complete a task or job position and employees needing additional training. The purpose of this study is to extend the term-document matrix and achieve refined clusters to produce an improved recommendation. The study also focuses on achieving consistency in cluster quality in spite of increasing size of data set, to solve scalability issues.Design/methodology/approachIn this study, a synset-based document matrix construction method is proposed where semantically similar terms are grouped to reduce the dimension curse. An automated Task Recommendation System is proposed comprising synset-based feature extraction, iterative semantic clustering and mapping based on semantic similarity.FindingsThe first step in knowledge extraction from the unstructured textual data is converting it into structured form either as Term frequency–Inverse document frequency (TF-IDF) matrix or synset-based TF-IDF. Once in structured form, a range of mining algorithms from classification to clustering can be applied. The algorithm gives a better feature vector representation and improved cluster quality. The synset-based grouping and feature extraction for resume data optimizes the candidate selection process by reducing entropy and error and by improving precision and scalability.Research limitations/implicationsThe productivity of any organization gets enhanced by assigning tasks to employees with a right set of skills. Efficient recruitment and task allocation can not only improve productivity but also cater to satisfy employee aspiration and identifying training requirements.Practical implicationsIndustries can use the approach to support different processes related to human resource management such as promotions, recruitment and training and, thus, manage the talent pool.Social implicationsThe task recommender system creates knowledge by following the steps of the knowledge management cycle and this methodology can be adopted in other similar knowledge management applications.Originality/valueThe efficacy of the proposed approach and its enhancement is validated by carrying out experiments on the benchmarked dataset of resumes. The results are compared with existing techniques and show refined clusters. That is Absolute error is reduced by 30 per cent, precision is increased by 20 per cent and dimensions are lowered by 60 per cent than existing technique. Also, the proposed approach solves issue of scalability by producing improved recommendation for 1,000 resumes with reduced entropy.

Development of Sindhi text corpus

Sindhi language is a rich language with plenty of literary and general texts. There are number of books, newspapers, magazines and internet material available to develop Sindhi text corpus but yet proper and useful text corpus could not be developed and presented online for research, language features analysis, linguistics analysis and information retrieval systems. The lack of resources for research on computational linguistics and NLP applications for Sindhi language are challenging tasks at this stage. However, we have developed Sindhi text corpora in order to provide text resources to computational linguists, Natural Languages process (NLP) experts and researchers. Online books, newspapers, magazines, blogs and social websites are utilized to build Sindhi text corpus. Sindhi sentiment based text corpus is developed and analyzed with Document Term Matrix and TF-IDF models using 2-gram technique of n-gram model. The corpus may be useful for research on language variation analysis, sentiment analysis, aspect based sentiment analysis, semantic analysis, machine translation, information retrieval, Word2Vec, topic modeling and cluster analysis.

Perception d’espèces agroforestières et de leurs services écosystémiques par trois groupes ethniques du bassin versant de Boura, zone soudanienne du Burkina Faso

En zone tropicale, les services écosystémiques associés aux espèces agroforestières sont peu connus. Une enquête ethnobotanique a été conduite dans le bassin versant de Boura pour appréhender la perception des espèces agroforestières et leurs services écosystémiques. Des interviews semi-structurées et des observations directes ont été réalisées auprès de 214 chefs d'exploitation (CE) appartenant aux groupes ethniques Sissala, Dagara et Mossi. Les caractéristiques socio-économiques des CE, les espèces ligneuses conservées dans les champs et leurs services écosystémiques ont été recensés. L’importance des espèces a été déterminée selon la méthode de l’indice culturel d’importance (ICT). Des tests de comparaison des réponses et des modèles linéaires généralisés ont été réalisés à l’aide du logiciel R.3.3.2. Au total, 64 espèces ligneuses appartenant à 59 genres et 30 familles ont été recensées. Ces espèces fournissent aux communautés 17 services écosystémiques répartis en quatre catégories. Vitellaria paradoxa (ICT = 10,45) et Parkia biglobosa (ICT = 7,80) sont deux espèces agroforestières clés pourvoyeuses de services écosystémiques à l’ensemble des communautés. L’ordination non métrique (NMS) de la matrice d’occurrence des espèces agroforestières indique qu’en dépit des similitudes de connaissances ethnobotaniques les groupes ethniques Dagara et Sissala manifestent différentes préférences dans la conservation d’essences ligneuses spécifiques dans les champs. Le groupe ethnique, le genre, la taille du ménage, le niveau d’éducation et l’expérience du chef d’exploitation sont les principaux facteurs déterminant la perception et la conservation des espèces agroforestières. La sélection d’essences végétales pour les interventions agroforestières doit tenir compte des facteurs socio-économiques déterminant les préférences des communautés.

Detecting and mapping Gonipterus scutellatus induced vegetation defoliation using WorldView-2 pan-sharpened image texture combinations and an artificial neural network

Defoliation induced by the weevil Gonipterus scutellatus is causing significant damage to South Africa’s eucalyptus plantations. Therefore, the ability of remote sensing to detect and map G. scutellatus defoliation is essential for monitoring the spread of the weevil so that precautionary measures are set in place. In our study, an integrated approach using image texture in various processing combinations and an artificial neural network (ANN) were developed to detect and map G. scutellatus induced vegetation defoliation. A 0.5-m WorldView-2 pan-sharpened image was used to compute texture parameters from the gray-level occurrence matrix and gray-level co-occurrence matrix, using optimal moving windows for specific levels of G. scutellatus induced vegetation defoliation. In order to improve the accuracy of detecting and mapping G. scutellatus induced vegetation defoliation, a method involving a three-band texture processing combination was proposed and tested. Using a sequential forward selection algorithm allowed for the selection of optimal texture combinations, which were subsequently input into a backpropagation ANN. The results showed an improvement in detecting vegetation defoliation using single texture bands [R2 = 0.82, root mean square error (RMSE) = 0.95 (1.82% of the mean measured defoliation)] when compared to single spectral reflectance bands [R2 = 0.60, RMSE = 1.79 (3.43% of the mean measured defoliation)], two-band spectral reflectance combination model [R2 = 0.74, RMSE = 1.48 (2.83% of the mean measured defoliation)], and the three-band spectral reflectance combination model [R2 = 0.80, RMSE = 1.35 (2.59% of the mean measured defoliation)]. Further improvements were obtained using the two-band texture combination model [R2 = 0.85, RMSE = 1.05 (2.01% of the mean measured defoliation)] and the most promising result was obtained using the proposed three-band texture combination model [R2 = 0.90, RMSE = 0.85 (1.63% of the mean measured defoliation)]. Overall, our study highlights the potential of image texture combinations in improving the detection and mapping of vegetation defoliation.

Alternate Low-Rank Matrix Approximation in Latent Semantic Analysis

The latent semantic analysis (LSA) is a mathematical/statistical way of discovering hidden concepts between terms and documents or within a document collection (i.e., a large corpus of text). Each document of the corpus and terms are expressed as a vector with elements corresponding to these concepts to form a term-document matrix. Then, the LSA uses a low-rank approximation to the term-document matrix in order to remove irrelevant information, to extract more important relations, and to reduce the computational time. The irrelevant information is called as “noise” and does not have a noteworthy effect on the meaning of the document collection. This is an essential step in the LSA. The singular value decomposition (SVD) has been the main tool obtaining the low-rank approximation in the LSA. Since the document collection is dynamic (i.e., the term-document matrix is subject to repeated updates), we need to renew the approximation. This can be done via recomputing the SVD or updating the SVD. However, the computational time of recomputing or updating the SVD of the term-document matrix is very high when adding new terms and/or documents to preexisting document collection. Therefore, this issue opened the door of using other matrix decompositions for the LSA as ULV- and URV-based decompositions. This study shows that the truncated ULV decomposition (TULVD) is a good alternative to the SVD in the LSA modeling.

Applying Text-mining Techniques to Global Supply Chain Region Selection: Considering Regional Differences

The concept of the global supply chain is built upon the mindset of competitive advantages, and it results in several benefits to the manufacturing process, such as increasing flexibility and operational cost-down. Nevertheless, there are still plenty of risks differing among different regions or nations under a global supply chain, and this increases the uncertainty of the supply chain and the possibility of disruption. This paper tackles this issue through a text-mining based global supply chain regional risk identification framework that analyzes literature to identify potential regionalglobal supply chain risks in a supplier region selection process. Text mining was implemented to collect and analyze the existing previous studies related to regional-related global supply chain risks. The results of text mining are represented in a document-term matrix that can further calculate the term-frequency (TF) and term frequency-inverse document frequency (TF-IDF) for the correlation analysis of terms related to supply chain risks. Those indicators provided information about whether the term-related content is vital in the studied literature, and further identified potential regional risk factors. A total of eight regional risk factors were extracted and organized. The proposed framework can objectively analyze geographical differences and provide enterprises with a global supply chain with intuitive guidance on region selection.

DeepStar: Detecting Starring Characters in Movies

Recent advances in the film industry have given rise to exponential growth in movie/drama production and adaptation of the Big Data concept. Automatic identification and classification of movie characters have received tremendous attention from researchers due to its applications in video semantic analysis, video summarization, and personalized video retrieval for which several methods have been recently presented. However, these methods cannot detect main characters properly due to their variation in pose and style in different scenes of a movie. To address this problem we present DeepStar, a novel framework for starring character identification based on deep high-level robust features. The proposed framework is threefold: the extraction of shots with clear faces from the input video; face clustering using discriminative deep features; and the occurrence matrix generation, helping in the selection of starring characters. The promising results obtained using representative Hollywood movies demonstrate the effectiveness of our method in detecting starring characters over the state-of-the-art methods.