Data Mining Problems Research Articles

A survey on big data: an emerging imparity and revolution in digital world

Size of the heterogeneous data is increasing rapidly at an electrifying speed. But we cannot handle this massive amount of unstructured data in traditional database, most of the data in the digital universe is unstructured. Big data analytics provides better computational power and efficient mechanism to handle this situation. This paper attempts to offer a survey report of big data, which changes rapidly of high volume, velocity, verity. In this paper we discuss some of the major challenges with a focus on big data characteristics followed by a conceptual framework of big data. This survey paper also focuses on the complexity of medical images and data mining problems.

Behavior Action Mining

The actionable behavioral rules suggest specific actions that may influence certain behavior in the stakeholders’ best interest. In mining such rules, it was assumed previously that all attributes are categorical while the numerical attributes have been discretized in advance. However, this assumption significantly reduces the solution space, and thus hinders the potential of mining algorithms, especially when the numerical attributes are prevalent. As the numerical data are ubiquitous in business applications, there is a crucial need for new mining methodologies that can better leverage such data. To meet this need, in this paper, we define a new data mining problem, named behavior action mining, as a problem of continuous variable optimization of expected utility for action. We then develop three approaches to solving this new problem, which uses regression as a technical basis. The experimental results based on a marketing dataset demonstrate the validity and superiority of our proposed approaches.

Fast approximate nearest neighbor search with the navigating spreading-out graph

Approximate nearest neighbor search (ANNS) is a fundamental problem in databases and data mining. A scalable ANNS algorithm should be both memory-efficient and fast. Some early graph-based approaches have shown attractive theoretical guarantees on search time complexity, but they all suffer from the problem of high indexing time complexity. Recently, some graph-based methods have been proposed to reduce indexing complexity by approximating the traditional graphs; these methods have achieved revolutionary performance on million-scale datasets. Yet, they still can not scale to billion-node databases. In this paper, to further improve the search-efficiency and scalability of graph-based methods, we start by introducing four aspects: (1) ensuring the connectivity of the graph; (2) lowering the average out-degree of the graph for fast traversal; (3) shortening the search path; and (4) reducing the index size. Then, we propose a novel graph structure called Monotonic Relative Neighborhood Graph (MRNG) which guarantees very low search complexity (close to logarithmic time). To further lower the indexing complexity and make it practical for billion-node ANNS problems, we propose a novel graph structure named Navigating Spreading-out Graph (NSG) by approximating the MRNG. The NSG takes the four aspects into account simultaneously. Extensive experiments show that NSG outperforms all the existing algorithms significantly. In addition, NSG shows superior performance in the E-commercial scenario of Taobao (Alibaba Group) and has been integrated into their billion-scale search engine.

Short-text learning in social media: a review

AbstractSocial networks occupy a ubiquitous and pervasive place in the life of their users. The substantial amount of content generated and shared by social networking users offers new research opportunities across a wide variety of disciplines, including media and communication studies, linguistics, sociology, psychology, information and computer sciences, or education. This situation, in combination with the continuous growth of social media data, creates an imperative need for content organisation. Thus, large-scale text learning tasks in social environments arise as one of the most relevant problems in machine learning and data mining. Interestingly, social media data pose several challenges due to its sparse, high-dimensional and large-volume characteristics. This survey reviews the field of social media data learning, focusing on classification and clustering techniques, as they are two of the most frequent learning tasks. It reviews not only new techniques that have been developed to tackle the new challenges posed by short-texts, but also how traditional techniques can be adapted to overcome such challenges. Then, open issues and research opportunities for social media data learning are discussed.

A New Application of Optimized Random Forest Algorithms in Intelligent Fault Location of Rudders

During the movements of aircraft, missiles, and ships, the rudder plays an important role in their direction control. In order to test the parameters of the rudders, we have to manually measure each item one by one in traditional production and manufacturing of rudders, which waste a great quantity of manpower and time. In this paper, we present a new application in rudder fault test by using machine learning technology and recommend a new intelligent method for fault location. The main subject revolves around prediction-oriented problems of multi-dimensional performance parameters data mining and the modeling of classification, including the analysis and processing of data features and the solution of fault location based on classification model. In addition, to improve the accuracy of the classification model, we optimized the random forest (RF) algorithm with the shuffled frog leaping algorithm (SFLA), which we call shuffled frog leaping algorithm-based random forest (SFLA-RF). It effectively solves the problem of voting competition among each tree, which makes the decision of the model more efficient and accurate. In a word, by means of automatic test and intelligent analysis, this new method breaks through the technical bottleneck of low efficiency of parameters test and the shortcomings of traditional rudder fault location.

Usage of the rough set theory for generating decision rules of number of traffic vehicles

Abstract Often, it is difficult to interpret and use the large size of data obtained from the experiment. In addition, the generated information can be unprecise. The rough set theory besides probability theory, fuzzy set theory and many others in recent years is very often used by scientists to solve problems of data mining. In the paper the data mining of the traffic vehicles with rough set theory was made. With this theory it was shown that it is possible to generate the decision rules of the number of vehicles at the specific points in the city. On the basis of 120 objects 16 well-defined linguistic decision rules were obtained.

IEEE Access Special Section Editorial: Data Mining and Granular Computing in Big Data and Knowledge Processing

Data mining has actively contributed to solving many real-world problems with a variety of techniques. Traditional approaches in this field are classification, clustering and regression. During the last few years a number of chal-lenges have emerged, such as imbalanced data, multi-label and multi-instance problems, low quality and/or noisy data or semi-supervised learning, among others [item 1) in the Appendix]. When these non-standard scenarios are encountered in the realm of big data, it remains an uncharted research territory, although a growing effort has been made to break the limits. The current trend is to address the classical and newly emerging data mining problems in big data and knowledge processing. Granular computing provides a powerful tool for multiple granularity and multiple-view data analysis at differ-ent granularity levels, which has demonstrated strong capabil-ities and advantages in intelligent data analysis, pattern recog-nition, machine learning and uncertain reasoning [item 2) inthe Appendix]. Big data often contains a significant amount of unstructured, uncertain and imprecise data. There are new challenges regarding the scalability of granular computing when addressing very big data sets [item 3) in the Appendix]. Big data mining relies on distributed computational strate-gies; it is often impossible to store and process data on one single computing node. The exploration of data mining and granular computing in big data and knowledge processing is an emerging field which crosses multiple research disciplines and industry domains, including transportation, communications, social network, medical health, and so on.

Location-Based Parallel Sequential Pattern Mining Algorithm

Given a data sequence, sequential pattern mining, which finds frequent sequence patterns among them, is an important data mining problem. However, in the existing sequential pattern mining, only the purchase order of the items is considered, and the position where the item is purchased is not considered. In this paper, we developed a sequential pattern mining algorithm using Apache spark. The proposed algorithm finds frequent sequential patterns in parallel by distributing data to several machines. Experimentally, we performed a comprehensive performance study on the proposed algorithm by varying various parameter values using various synthetic data. Experimental results show that the proposed algorithm shows a linear speed improvement over the number of machines.

Record linkage based on a three-way decision with the use of granular descriptors

Record linkage is a typical two-class recognition problem in data mining. To improve its classification performance of the problem, this paper proposes to apply three-way classification to identify uncertain points (regions) for further clerical investigation in decision-making. The detailed three-way decision process is realized by a two-phase approach. During the first phase, an information granule is constructed to describe the uncertain region in the data space. In the second phase, the constructed granule is utilized to discriminate between certain points (those with a high likelihood of belonging to one of the classes) and uncertain points (viz. those requiring clerical attention). For uncertain points, manual investigation is realized; for certain points, the generic binary classifier is applied for classification. Synthetic data and publicly available data are used to demonstrate the performance of the proposed approach. Finally, the proposed approach is shown effective in applications involving real-world record linkage data.

Exploiting Data Mining for Fast Inter Prediction Mode Decision in HEVC

HEVC has made significant progress in coding efficiency. Compared with the previous generation H.264/AVC video coding standard, HEVC can save 50% of code rate under the same visual quality. However, this drop in bit rate is accompanied by an increase in computational complexity, which poses a great challenge to the application and development of HEVC. After analyzing the great correlation between the partitioning mode of coding unit in the encoder and several parameters, the partitioning problem of the CU is modeled as a classification problem in data mining, and it is decided to use a decision tree to solve this classification problem. We implement the decision tree on the official software HM16.2 and test the algorithm on the test set. Experiments show that fast algorithm based on decision tree can improve the encoding speed effectively with little influence on the encoding performance.

High Utility Infrequent Itemset Mining Using a Customized Ant Colony Algorithm

Itemset mining is a popular extension to the frequent pattern mining problem in data mining. Finding infrequent patterns, however, has gained its importance due to proven utility in the areas of web mining, bioinformatics and others. High utility mining refines the problem focus to identifying business-relevant transaction patterns that take purchase quantities and monetary considerations into account, like unit price and cost, typically to identify patterns of profit potential. High utility infrequent itemset mining unveils rare cases of highly profitable itemsets. This paper proposes a customized Ant colony algorithm for the efficient discovery of high utility infrequent itemsets. The mining performance of proposed algorithm is analyzed on four real time datasets namely chess, food mart, mushroom and retail.

Distributed Data Mining for Multiple Sourced Heterogeneous Datasets: A Survey

In the information age of the 21st century, a large amount of information is collected and applied. However, due to the heterogeneity of system environment for data storage and computing, how to mine these distributed data sources has become a valuable research topic that attracted more and more attention. In this paper, we firstly presented the problem scenario and main challenges confronting with the problem of distributed data mining on multiple sourced heterogeneous data sets. Then, we surveyed research works related to the problem and elicited their main features on different technology domains to show current distributed solutions for different data mining algorithm categories. Finally, we reviewed in detail the research works and discussed the challenges remained in the distributed data mining problem for multiple sourced heterogeneous data sets.

PyHIVE, a health-related image visualization and engineering system using Python

BackgroundImaging is one of the major biomedical technologies to investigate the status of a living object. But the biomedical image based data mining problem requires extensive knowledge across multiple disciplinaries, e.g. biology, mathematics and computer science, etc.ResultspyHIVE (a Health-related Image Visualization and Engineering system using Python) was implemented as an image processing system, providing five widely used image feature engineering algorithms. A standard binary classification pipeline was also provided to help researchers build data models immediately after the data is collected. pyHIVE may calculate five widely-used image feature engineering algorithms efficiently using multiple computing cores, and also featured the modules of Principal Component Analysis (PCA) based preprocessing and normalization.ConclusionsThe demonstrative example shows that the image features generated by pyHIVE achieved very good classification performances based on the gastrointestinal endoscopic images. This system pyHIVE and the demonstrative example are freely available and maintained at http://www.healthinformaticslab.org/supp/resources.php.

A novel distance measure for time series: Maximum shifting correlation distance

Time series distance or similarity measurement is one of the most important problems in time series data mining, including representation, clustering, classification, and outlier detection. The existing distance measures may not efficiently deal with the drifts in the series in both the phase and amplitude. In this study, a novel measurement, maximum shifting correlation distance (MSCD), is proposed to improve the accuracy and efficiency of the time series distance measure. By integrating the curve registration and correlation, the misalignments or drifts in the phase and amplitude can be eliminated, respectively. In addition, the second distance of the MSCD (MSCD-2nd), which has the “shrinkage effect”, enhances the similarity of the samples within a cluster. The second distance has the same effect on the dynamic time warping (DTW) distance. The experimental results demonstrate that MSCD-2nd is the preferred measure in terms of the accuracy and efficiency for time series clustering and classification.

Data Mining Approach in Retail Knowledge Discovery and Internet Technologies

Online shopping has a shopping channel or purchasing various items through online medium. Data mining is defined as a process used to extract usable data from a larger set of any raw data. The data set extraction from the demographic profiles and Questionnaire to investigate the gathered based by association. The method for shopping was totally changed with the happening to internet Technology. Association rule mining is one of the important problems of data mining has been used here. The goal of the association rule mining is to detect relationships or associations between specific values of categorical variables in large data sets.

Evolutionary Propositionalization of Multi-Relational Data — Research Notes

Propositionalization has been proven to be a very effective solution for multi-relational data mining problems. The approaches usually follow a two-step principle: transforming the relational data into a single, flat table and applying a propositional learning algorithm. During the transformation, the target table gets expanded by adding many new features summarizing the information of the non-target tables. Based on the used feature construction strategy, this leads to a table of very high dimensionality with a lot of irrelevant and/or redundant features that can negatively affect the predictive performance. In this paper, we propose a modification of the traditional two-step framework to overcome such problems. The proposed approach evaluates the features during the construction phase and reports only a subset of highly predictive features to the propositional learner. We present an implementation of this approach using a genetic algorithm to search for an optimal feature subset. Our experiments on a number of benchmark datasets suggest that the modified framework can help propositionalization methods to significantly improve their predictive performance.

A Novel Honey-Bees Mating Optimization Approach with Higher order Neural Network for Classification

In the recent past, several biological and natural phenomena have extensively attracted researchers towards the rapid development of science and engineering. Basically solving the optimization problems in various Engineering discipline is a popular topic among the other problem solving strategies. Most of the biological processes include the swarm intelligence research areas where the activity and the behavior of real insects have been studied. One of the recently developed Swarm algorithms is the Honey Bee Mating Optimization (HBMO) algorithm which is based on the mating behavior of bees. In this work, a hybrid metaheuristic honey bee mating based Pi-Sigma Neural Network (PSNN) have been proposed to successfully solve the classification problem of data mining. The proposed approach combines HBMO with the PSNN and is compared with other techniques like GA (Genetic Algorithm), DE (Differential Evolution), and PSO (Particle Swarm Optimization). Experimental results reveal that the proposed approach is steady as well as reliable and provides better classification accuracy than others.

SPPC: a new tree structure for mining erasable patterns in data streams

Discovering Erasable Patterns (EPs) consists of identifying product parts that will produce a small profit loss if their production is stopped. It is a data mining problem that has attracted the attention of numerous researchers in recent years due to the possibility of using EPs to reduce profit loss of manufacturers. Though, many algorithms have been designed to mine EPs, an important limitation of state-of-the-art EP mining algorithms is that they are batch algorithms, that is, they are designed to be applied on static databases. But in real-life applications, databases are dynamic, as they are constantly updated by adding or removing products and parts. To be informed about EPs in real-time, traditional EP mining algorithms must be applied over and over again on a database. This is inefficient as those algorithms are always applied from scratch without taking advantage of results generated by previous executions. Considering this important drawback of previous work for handling real-life dynamic data, this paper proposes an efficient algorithm named MSPPC for mining EPs in data streams. It relies on a novel tree structure named SPPC (Streaming Pre-Post Code) tree, which extends the WPPC tree structure for maintaining a compact tree representation of EPs in a data stream. Experimental results show that the designed MSPPC algorithm outperforms the state-of-the-art batch MERIT and dMERIT algorithms when they are run in batch mode using a sliding-window. Besides, the proposed algorithm is also faster than the state-of-the-art algorithms for mining EPs, namely MERIT, dMERIT + , MEI and EIFDD.

A high-performance clustering algorithm based on searched experiences

Clustering is a traditional data mining problem that has attracted researchers from different disciplines because its solution can be applied to many useful problems in our daily life. Since the era of big data is coming, how to “reduce the computing time” of an “effective clustering algorithm” has been a promising research issue in recent years. Thus, this paper presents an effective clustering algorithm, by using the so-called searched information to determine later search directions, and then has it implemented on Spark to accelerate its response time for analyzing large-scale datasets. Simulation results show that the proposed algorithm provides a better result than the other clustering algorithms compared in this paper because it is less sensitive to the initial solutions. The simulation results further show that cloud computing platform is capable of enhancing the performance of the proposed algorithm.

Spatio-Temporal Data Mining

Large volumes of spatio-temporal data are increasingly collected and studied in diverse domains, including climate science, social sciences, neuroscience, epidemiology, transportation, mobile health, and Earth sciences. Spatio-temporal data differ from relational data for which computational approaches are developed in the data-mining community for multiple decades in that both spatial and temporal attributes are available in addition to the actual measurements/attributes. The presence of these attributes introduces additional challenges that needs to be dealt with. Approaches for mining spatio-temporal data have been studied for over a decade in the data-mining community. In this article, we present a broad survey of this relatively young field of spatio-temporal data mining. We discuss different types of spatio-temporal data and the relevant data-mining questions that arise in the context of analyzing each of these datasets. Based on the nature of the data-mining problem studied, we classify literature on spatio-temporal data mining into six major categories: clustering, predictive learning, change detection, frequent pattern mining, anomaly detection, and relationship mining. We discuss the various forms of spatio-temporal data-mining problems in each of these categories.