Real-world Data Mining Research Articles

Emphatic Constraints Support Vector Machine

In this paper, a new support vector machine, ESVM, with more emphasis on constraints is presented. The constraints are fuzzy inequalities. With this scheme, two problems are solved: training samples with some degree of uncertainty, and samples with tolerance. Also, the fuzzy SVM (FSVM) model is modified with emphasis constraints. The new model is called fuzzy ESVM (FESVM), in this paper. With this scheme we will able to consider importance degree for samples both in the cost function and constraints, simultaneously. Necessary experiments are performed and the results show the superiority of the proposed methods. ESVM and fuzzy ESVM are strongly recommended to the researchers who work on data sets with noisy or low degree of certainty samples. Support vector machines (SVMs) are one of the most powerful methods that deliver state-of-the-art performance in real world pattern recognition and data mining applications. In the SVM solution, a pattern recognition problem is converted to a constraint quadratic programming. Support vector machine is originally introduced by Vapnik (1) within the area of statistical learning theory. It is bases on structural risk minimization (SRM) principle and finds a classifier with minimized Vapnik-Chervonenkis (VC) dimension. In pattern recognition, SVMs have been developed for data classification, feature reduction, and function estimation (2,3). The SVM classifier is very sensitive to outliers and noisy samples since the penalty term of SVM treats every data point equally in the training process. Increasing the values of slack variables, help in reducing the effect of noisy support vectors. Without introducing these slack variables, in the presence of noisy data, SVM may not be able to determine a hyperplane between two classes. The generalized formulation of the SVM model which can tolerate noisy data close to separating hyperplane is as follows:

A framework for modeling positive class expansion with single snapshot

In many real-world data mining tasks, the connotation of the target concept may change as time goes by. For example, the connotation of “learned knowledge” of a student today may be different from his/her “learned knowledge” tomorrow, since the “learned knowledge” of the student is expanding everyday. In order to learn a model capable of making accurate predictions, the evolution of the concept must be considered, and thus, a series of data sets collected at different time is needed. In many tasks, however, there is only a single data set instead of a series of data sets. In other words, only a single snapshot of the data along the time axis is available. In this paper, we formulate the Positive Class Expansion with single Snapshot (PCES) problem and discuss its difference with existing problem settings. To show that this new problem is addressable, we propose a framework which involves the incorporation of desirable biases based on user preferences. The resulting optimization problem is solved by the Stochastic Gradient Boosting with Double Target approach, which achieves encouraging performance on PCES problems in experiments.

Boosting support vector machines for imbalanced data sets

Real world data mining applications must address the issue of learning from imbalanced data sets. The problem occurs when the number of instances in one class greatly outnumbers the number of instances in the other class. Such data sets often cause a default classifier to be built due to skewed vector spaces or lack of information. Common approaches for dealing with the class imbalance problem involve modifying the data distribution or modifying the classifier. In this work, we choose to use a combination of both approaches. We use support vector machines with soft margins as the base classifier to solve the skewed vector spaces problem. We then counter the excessive bias introduced by this approach with a boosting algorithm. We found that this ensemble of SVMs makes an impressive improvement in prediction performance, not only for the majority class, but also for the minority class.

Performance Comparative in Classification Algorithms Using Real Datasets

Classification is one of the most common data mining tasks, used frequently for data categorization and analysis in the industry and research. In real-world data mining sometimes it mainly deals with noisy information sources, because of data collection inaccuracy, device limitations, data transmission and discretization errors, or man-made perturbations frequently result in imprecise or vague data which is called as noisy data. This noisy data may decrease performance of any classification algorithms. This paper deals with the performance of different classification algorithms and the impact of feature selection algorithm on Logistic Regression Classifier, How it controls False Discovery Rate (FDR) and thus improves the efficiency of Logistic Regression classifier.

USING INSTANCE CLONING TO IMPROVE NAIVE BAYES FOR RANKING

Improving naive Bayes (simply NB)15,28 for classification has received significant attention. Related work can be broadly divided into two approaches: eager learning and lazy learning.1 Different from eager learning, the key idea for extending naive Bayes using lazy learning is to learn an improved naive Bayes for each test instance. In recent years, several lazy extensions of naive Bayes have been proposed. For example, LBR,30 SNNB,27 and LWNB.8 All these algorithms aim to improve naive Bayes' classification performance. Indeed, they achieve significant improvement in terms of classification, measured by accuracy. In many real-world data mining applications, however, an accurate ranking is more desirable than an accurate classification. Thus a natural question is whether they also achieve significant improvement in terms of ranking, measured by AUC (the area under the ROC curve).2,11,17 Responding to this question, we conduct experiments on the 36 UCI data sets18 selected by Weka12 to investigate their ranking performance and find that they do not significantly improve the ranking performance of naive Bayes. Aiming at scaling up naive Bayes' ranking performance, we present a novel lazy method ICNB (instance cloned naive Bayes) and develop three ICNB algorithms using different instance cloning strategies. We empirically compare them with naive Bayes. The experimental results show that our algorithms achieve significant improvement in terms of AUC. Our research provides a simple but effective method for the applications where an accurate ranking is desirable.

Mining With Noise Knowledge: Error-Aware Data Mining

Real-world data mining deals with noisy information sources where data collection inaccuracy, device limitations, data transmission and discretization errors, or man-made perturbations frequently result in imprecise or vague data. Two common practices are to adopt either data cleansing approaches to enhance the data consistency or simply take noisy data as quality sources and feed them into the data mining algorithms. Either way may substantially sacrifice the mining performance. In this paper, we consider an error-aware (EA) data mining design, which takes advantage of statistical error information (such as noise level and noise distribution) to improve data mining results. We assume that such noise knowledge is available in advance, and we propose a solution to incorporate it into the mining process. More specifically, we use noise knowledge to restore original data distributions, which are further used to rectify the model built from noise- corrupted data. We materialize this concept by the proposed EA naive Bayes classification algorithm. Experimental comparisons on real-world datasets will demonstrate the effectiveness of this design.

DDDM2007

Real-world data mining generally must consider and involve domain and business oriented factors such as human knowledge, constraints and business expectations. This encourages the development of a domain driven methodology to strengthen data-centered pattern mining. This report presents a review of the ACM SIGKDD Workshop on Domain Driven Data Mining (DDDM2007), held in conjunction with the Thirteenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD07), which was held in San Jose, USA on 12 August, 2007. The aims and objectives of this workshop were to provide a premier forum for sharing innovative findings, knowledge, insights, experiences and lessons in tackling challenges met in domain driven, actionable knowledge discovery in the real world.

Revisiting the Foundations of Artificial Immune Systems for Data Mining

This paper advocates a problem-oriented approach for the design of artificial immune systems (AIS) for data mining. By problem-oriented approach we mean that, in real-world data mining applications the design of an AIS should take into account the characteristics of the data to be mined together with the application domain: the components of the AIS - such as its representation, affinity function, and immune process - should be tailored for the data and the application. This is in contrast with the majority of the literature, where a very generic AIS algorithm for data mining is developed and there is little or no concern in tailoring the components of the AIS for the data to be mined or the application domain. To support this problem-oriented approach, we provide an extensive critical review of the current literature on AIS for data mining, focusing on the data mining tasks of classification and anomaly detection. We discuss several important lessons to be taken from the natural immune system to design new AIS that are considerably more adaptive than current AIS. Finally, we conclude this paper with a summary of seven limitations of current AIS for data mining and ten suggested research directions.

Framework for efficient feature selection in genetic algorithm based data mining

We present the design of more effective and efficient genetic algorithm based data mining techniques that use the concepts of feature selection. Explicit feature selection is traditionally done as a wrapper approach where every candidate feature subset is evaluated by executing the data mining algorithm on that subset. In this article we present a GA for doing both the tasks of mining and feature selection simultaneously by evolving a binary code along side the chromosome structure used for evolving the rules. We then present a wrapper approach to feature selection based on Hausdorff distance measure. Results from applying the above techniques to a real world data mining problem show that combining both the feature selection methods provides the best performance in terms of prediction accuracy and computational efficiency.

Exploiting idle cycles to execute data mining applications on clusters of PCs

In this paper we present and evaluate Inhambu, a distributed object-oriented system that supports the execution of data mining applications on clusters of PCs and workstations. This system provides a resource management layer, built on the top of Java/RMI, that supports the execution of the data mining tool called Weka. We evaluate the performance of Inhambu by means of several experiments in homogeneous, heterogeneous and non-dedicated clusters. The obtained results are compared with those achieved by a similar system named Weka-Parallel. Inhambu outperforms its counterpart for coarse grain applications, mainly for heterogeneous and non-dedicated clusters. Also, our system provides additional advantages such as application checkpointing, support for dynamic aggregation of hosts to the cluster, automatic restarting of failed tasks, and a more effective usage of the cluster. Therefore, Inhambu is a promising tool for efficiently executing real-world data mining applications. The software is delivered at the project’s web site available at http://incubadora.fapesp.br/projects/inhambu/.

Introduction to the special issue on successful real-world data mining applications

Since its inception, the field of Data Mining and Knowledge Discovery from Databases has been driven by the need to solve practical problems [4]. From scaling to large databases and handling noisy and high-dimensional data to finding associational patterns in grocery store transaction data, data mining is a research area rich in application [1]. Despite its practical roots few case studies of data mining applications have been published. The industrial track of the annual SIGKDD conference has provided one such forum, but rarely do these papers present complete descriptions of deployed systems [2]. This special issue attempts to address the gap by showcasing the choices, strategies, and lessons learned from building a real-world data mining application. In a sense this collection is a follow-up to the first workshop on data mining case studies held during ICDM-2006 [3]. This issue however introduces several new papers. Of the 29 papers reviewed 10 papers were accepted. The papers come from a broad range of application areas including Customer Relationship Management, Medicine, Taxation, and Software Development.

Making SVMs Scalable to Large Data Sets using Hierarchical Cluster Indexing

Support vector machines (SVMs) have been promising methods for classification and regression analysis due to their solid mathematical foundations, which include two desirable properties: margin maximization and nonlinear classification using kernels. However, despite these prominent properties, SVMs are usually not chosen for large-scale data mining problems because their training complexity is highly dependent on the data set size. Unlike traditional pattern recognition and machine learning, real-world data mining applications often involve huge numbers of data records. Thus it is too expensive to perform multiple scans on the entire data set, and it is also infeasible to put the data set in memory. This paper presents a method, Clustering-Based SVM (CB-SVM), that maximizes the SVM performance for very large data sets given a limited amount of resource, e.g., memory. CB-SVM applies a hierarchical micro-clustering algorithm that scans the entire data set only once to provide an SVM with high quality samples. These samples carry statistical summaries of the data and maximize the benefit of learning. Our analyses show that the training complexity of CB-SVM is quadratically dependent on the number of support vectors, which is usually much less than that of the entire data set. Our experiments on synthetic and real-world data sets show that CB-SVM is highly scalable for very large data sets and very accurate in terms of classification.

KBA: kernel boundary alignment considering imbalanced data distribution

An imbalanced training data set can pose serious problems for many real-world data mining tasks that employ SVMs to conduct supervised learning. In this paper, we propose a kernel-boundary-alignment algorithm, which considers THE training data imbalance as prior information to augment SVMs to improve class-prediction accuracy. Using a simple example, we first show that SVMs can suffer from high incidences of false negatives when the training instances of the target class are heavily outnumbered by the training instances of a nontarget class. The remedy we propose is to adjust the class boundary by modifying the kernel matrix, according to the imbalanced data distribution. Through theoretical analysis backed by empirical study, we show that our kernel-boundary-alignment algorithm works effectively on several data sets.

Mining Skewed and Sparse Transaction Data for Personalized Shopping Recommendation

A good shopping recommender system can boost sales in a retailer store. To provide accurate recommendation, the recommender needs to accurately predict a customer's preference, an ability difficult to acquire. Conventional data mining techniques, such as association rule mining and collaborative filtering, can generally be applied to this problem, but rarely produce satisfying results due to the skewness and sparsity of transaction data. In this paper, we report the lessons that we learned in two real-world data mining applications for personalized shopping recommendation. We learned that extending a collaborative filtering method based on ratings (e.g., GroupLens) to perform personalized shopping recommendation is not trivial and that it is not appropriate to apply association-rule based methods (e.g., the IBM SmartPad system) for large scale prediction of customers' shopping preferences. Instead, a probabilistic graphical model can be more effective in handling skewed and sparse data. By casting collaborative filtering algorithms in a probabilistic framework, we derived HyPAM (Hybrid Poisson Aspect Modelling), a novel probabilistic graphical model for personalized shopping recommendation. Experimental results show that HyPAM outperforms GroupLens and the IBM method by generating much more accurate predictions of what items a customer will actually purchase in the unseen test data. The data sets and the results are made available for download at http://chunnan.iis.sinica.edu.tw/hypam/HyPAM.html.

Editorial: Data Mining Lessons Learned

IntroductionData mining is concerned with finding interesting patterns in data.Many techniques have emerged for analyzing and visualizing large vol-umes of data. What one finds in the technical literature are mostlysuccess stories of these techniques. Researchers rarely report on stepsleading to success, failed attempts, or critical representation choicesmade; and rarely do papers include expert evaluations of achieved re-sults. An interesting point of investigation is also why some particularsolutions, despite good performance, were never used in practice orrequired additional treatment before they could be used. Insightfulanalyses of successful and unsuccessful applications are crucial for in-creasing our understanding of machine learning techniques and theirlimitations.The UCI Repository of Machine Learning Databases (Blake &Merz, 1998) hasserved the machine learning community for manyyearsas a valuable resource. It has benefited the community by allowingresearchers to compare algorithm performance on a common set ofbenchmark datasets, most taken from real-world domains. However,its existence has indirectly promoted a very narrow view of real-worlddata mining. Performance comparisons, which typically focus on clas-sification accuracy, neglect important data mining issues such as dataunderstanding, data preparation, selection of appropriate performancemetrics,andexpertevaluation ofresults.Furthermore,becausetheUCIrepository has been used so extensively, some researchers have claimedthat our algorithms may be “overfitting the UCI repository”.Challenge problems such as the KDD Cup, CoIL and PTE chal-lenges have also become popular in recent years and have attractednumerous participants. Contrary to the “UCI challenge” of achieving

A new approach to hierarchical clustering and structuring of data with Self-Organizing Maps

The Self-Organizing Map (SOM) is a powerful tool for exploratory data analysis which has been employed in a wide range of data mining applications. We present a novel approach to reveal the inherent hierarchical structure of data using multiple SOMs together with heuristics which optimize the stability. In particular, we address shortcomings of the Growing Hierarchical Self-Organizing Map (GHSOM) regarding the decision which areas in the hierarchical structure need to be represented by a finer granularity and which areas do not. We introduce the Tension and Mapping Ratio extension to exploit specific characteristics of the SOM based on the topology preservation. As a main result, in contrast to the GHSOM, the inherent hierarchical structure of the data is revealed without requiring the user to define a threshold parameter which controls the map sizes of the individual SOMs. We evaluate our approach using data from real-world data mining projects in the music domain.

A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problems

Categorical data fields characterized by a large number of distinct values represent a serious challenge for many classification and regression algorithms that require numerical inputs. On the other hand, these types of data fields are quite common in real-world data mining applications and often contain potentially relevant information that is difficult to represent for modeling purposes.This paper presents a simple preprocessing scheme for high-cardinality categorical data that allows this class of attributes to be used in predictive models such as neural networks, linear and logistic regression. The proposed method is based on a well-established statistical method (empirical Bayes) that is straightforward to implement as an in-database procedure. Furthermore, for categorical attributes with an inherent hierarchical structure, like ZIP codes, the preprocessing scheme can directly leverage the hierarchy by blending statistics at the various levels of aggregation.While the statistical methods discussed in this paper were first introduced in the mid 1950's, the use of these methods as a preprocessing step for complex models, like neural networks, has not been previously discussed in any literature.