Data Mining Projects Research Articles

Cross-Industry Standard Process for data mining is applicable to the lung cancer surgery domain, improving decision making as well as knowledge and quality management

The aim of this study was to assess the applicability of knowledge discovery in database methodology, based upon data mining techniques, to the investigation of lung cancer surgery. According to CRISP 1.0 methodology, a data mining (DM) project was developed on a data warehouse containing records for 501 patients operated on for lung cancer with curative intention. The modelling technique was logistic regression. The finally selected model presented the following values: sensitivity 9.68%, specificity 100%, global precision 94.02%, positive predictive value 100% and negative predictive value 93.98% for a cut-off point set at 0.5. A receiver operating characteristic (ROC) curve was constructed. The area under the curve (CI 95%) was 0.817 (0.740- 0.893) (p < 0.05). Statistical association with perioperative mortality was found for the following variables [odds ratio (CI 95%)]: age over 70 [2.3822 (1.0338-5.4891)], heart disease [2.4875 (1.0089-6.1334)], peripheral arterial disease [5.7705 (1.9296-17.2570)], pneumonectomy [3.6199 (1.4939-8.7715)] and length of surgery (min) [1.0067 (1.0008-1.0126)]. The CRISP-DM process model is very suitable for lung cancer surgery analysis, improving decision making as well as knowledge and quality management.

Collaborative research between academia and industry using a large clinical trial database: a case study in Alzheimer's disease

BackgroundLarge clinical trials databases, developed over the course of a comprehensive clinical trial programme, represent an invaluable resource for clinical researchers. Data mining projects sponsored by industry that use these databases, however, are often not viewed favourably in the academic medical community because of concerns that commercial, rather than scientific, goals are the primary purpose of such endeavours. Thus, there are few examples of sustained collaboration between leading academic clinical researchers and industry professionals in a large-scale data mining project. We present here a successful example of this type of collaboration in the field of dementia.MethodsThe Donepezil Data Repository comprised 18 randomised, controlled trials conducted between 1991 and 2005. The project team at Pfizer determined that the data mining process should be guided by a diverse group of leading Alzheimer's disease clinical researchers called the "Expert Working Group." After development of a list of potential faculty members, invitations were extended and a group of seven members was assembled. The Working Group met regularly with Eisai/Pfizer clinicians and statisticians to discuss the data, identify issues that were currently of interest in the academic and clinical communities that might lend themselves to investigation using these data, and note gaps in understanding or knowledge of Alzheimer's disease that these data could address. Leadership was provided by the Pfizer Clinical Development team leader; Working Group members rotated responsibility for being lead and co-lead for each investigation and resultant publication.ResultsSix manuscripts, each published in a leading subspecialty journal, resulted from the group's work. Another project resulted in poster presentations at international congresses and two were cancelled due to resource constraints.ConclusionsThe experience represents a particular approach to optimising the value of data mining of large clinical trial databases for the combined purpose of furthering clinical research and improving patient care. Fruitful collaboration between industry and academia was fostered while the donepezil data repository was used to advance clinical and scientific knowledge. The Expert Working Group approach warrants consideration as a blueprint for conducting similar research ventures in the future.

Data set preprocessing and transformation in a database system

In general, there is a significant amount of data mining analysis performed outside a database system, which creates many data management issues. This article presents a summary of our experience and recommendations to compute data set preprocessing and transformation inside a database system (i.e. data cleaning, record selection, summarization, denormalization, variable creation, coding), which is the most time-consuming task in data mining projects. This aspect is largely ignored in the literature. We present practical issues, common solutions and lessons learned when preparing and transforming data sets with the SQL language, based on experience from real-life projects. We then provide specific guidelines to translate programs written in a traditional programming language into SQL statements. Based on successful real-life projects, we present time performance comparisons between SQL code running inside the database system and external data mining programs. We highlight which steps in data mining projects become faster when processed by the database system. More importantly, we identify advantages and disadvantages from a practical standpoint based on data mining users feedback.

CMIN - a CRISP-DM-based case tool for supporting data mining projects

This paper introduces CMIN, an integrated computer aided software engineering (CASE) tool based on cross-industry standard process for data mining (CRISP-DM) 1.0 designed to support carrying out data mining projects. It is “integrated” in the sense that it supports all phases of a process. A general overview of how CMIN works is presented first, including a treatment of processes, templates and project management. CMIN´s capacity for easily and intuitively monitoring projects is highlighted, as is the manner in which CMIN allows a user to increase knowledge regarding using CRISP-DM or any other process defined in the CASE tool through the help and information presented in each step. Next, it is shown how CMIN can bind new data mining algorithms in runtime (without the need to recompile the tool) to support modelling tasks (based on a Workflow) and evaluate data mining projects. Finally, the results of two evaluations of the tool, some conclusions and suggestions for future work are presented.

A survey of data mining and knowledge discovery process models and methodologies

AbstractUp to now, many data mining and knowledge discovery methodologies and process models have been developed, with varying degrees of success. In this paper, we describe the most used (in industrial and academic projects) and cited (in scientific literature) data mining and knowledge discovery methodologies and process models, providing an overview of its evolution along data mining and knowledge discovery history and setting down the state of the art in this topic. For every approach, we have provided a brief description of the proposed knowledge discovery in databases (KDD) process, discussing about special features, outstanding advantages and disadvantages of every approach. Apart from that, a global comparative of all presented data mining approaches is provided, focusing on the different steps and tasks in which every approach interprets the whole KDD process. As a result of the comparison, we propose a new data mining and knowledge discovery process namedrefined data mining processfor developing any kind of data mining and knowledge discovery project. The refined data mining process is built on specific steps taken from analyzed approaches.

Partitional Cluster for Educational Institution Quality Assessment

The demand of data mining in educational institution has increased depending on the development of information systems. However, data mining needs high quality data while there are no sufficient methods to get quality data in the institutional zone for data mining to get a credible conclusion, which prevents its implementation. Commonly data cleaning and Extraction-Transformation-Loading tools or tolerance algorithms have been used to mine low quality data. But these methods only can improve the current data quality for mining, newly created data from institutional information system will make all data dirty again. By analyzing the reasons of low data quality on System Engineering theory, a new method called data mining consulting has been established, which solves the data quality problem by meta synthesis method including software designing, management and data mining testing, etc. Its application shows that it has good practicality which can do data mining project in low quality data and increase the institution’s decision levels. However, the main objective of an educational institution is to impart quality education. One way to reach the highest level of quality in education systems is by improving the decision making procedures on various processes such as assessment, evaluation, counseling and so on which requires knowledge. The proposal made in this research work evaluates the institutional quality in imparting education to the students using data mining techniques such as clusters, and predictive mining. It develops a clustering scheme to segment the student grades based on the knowledge acquired. In addition, it applies predictive mining with decision support tree mechanism to evaluate the quality of the institution with the hidden data attributes of the student profile, grade, faculty profile, curriculum, etc. extracted from educational data bases. The data mining evaluation provides the actual status on the institutional quality and guides the management in improving its performance by concentrating on the lacking areas. The system framework comprises clustering ensembles with the goal of creating an optimal framework for categorizing educational resources. It proposes both non adaptive and adaptive re-sampling schemes for the integration of multiple clustering (independent and dependent). Experimental results show improved stability and accuracy for clustering structures obtained via bootstrapping, sub sampling, and adaptive techniques. These improvements offer insights into specific decision within the data sets.

Rising Standards for Data Mining

This paper presents an overview of data mining, then discusses standards of existing and proposed that are relevant to data mining. This includes standards that affect several stages of a data mining project. Summaries of several emerging standards are given, as well as proposals that have the potential to change the way data mining tools are built. Keywords: Data Mining; SQL; OLE DB; PMML; CRISP-DM

Improving a rule evaluation support method based on objective indices

It is a key for the successes of data mining projects in practical situations to evaluate the obtained so many patterns as valuable knowledge effectively. In order to provide an effective support, we have been developing a rule evaluation support method based on the learning models of objective rule evaluation indices. In this paper, we report two improvements of this method and their evaluations. One is improved the learning algorithm selection in the proposed method by introducing a constructive meta-learning scheme. The other is improved the sorting efficiency of objective rule evaluation indices by combining them.

Development and user experiences of an open source data cleaning, deduplication and record linkage system

Record linkage, also known as database matching or entity resolution, is now recognised as a core step in the KDD process. Data mining projects increasingly require that information from several sources is combined before the actual mining can be conducted. Also of increasing interest is the deduplication of a single database. The objectives of record linkage and deduplication are to identify, match and merge all records that relate to the same real-world entities. Because real-world data is commonly 'dirty', data cleaning is an important first step in many deduplication, record linkage, and data mining project. In this paper, an overview of the Febrl (Freely Extensible Biomedical Record Linkage) system is provided, and the results of a recent survey of Febrl users is discussed. Febrl includes a variety of functionalities required for data cleaning, deduplication and record linkage, and it provides a graphical user interface that facilitates its application for users who do not have programming experience.

On Comparing Neuronal Morphologies with the Constrained Tree-edit-distance

The constrained tree-edit-distance provides a computationally practical method for comparing morphologies directly without first extracting distributions of other metrics. The application of the constrained tree-edit-distance to hippocampal dendrites by Heumann and Wittum is reviewed and considered in the context of other applications and potential future uses. The method has been used on neuromuscular projection axons for comparisons of topology as well as on trees for comparing plant architectures with particular parameter sets that may inform future efforts in comparing dendritic morphologies. While clearly practical on a small scale, testing and extrapolation of run-times raise questions as to the practicality of the constrained tree-edit-distance for large-scale data mining projects. However, other more efficient algorithms may make use of it as a gold standard for direct morphological comparison.

A new approach for health insurance customer relationship support through hybrid mining based system

The information and knowledge gained by hybrid mining can be used for applications. In health insurance domain, hybrid mining more than just helps management to identify healthcare fraud and abuse, instead addressing the broader application to make a wise decision with respect to improved customer relationship. While data mining methods are so powerful to master by users to facilitate customer relationship, this study introduces an approach which integrate decision support system with hybrid mining approach in health insurance customers' management to synthesise what is known about business value and guide future research. Meanwhile, we investigate the characteristics of the approach to show how it can aid in customer' management and provide policy information for insurance decision making. The approach can be used as a template for such evaluation by researcher and practicer before, during and after the completion of their data mining projects in customer relationship management (CRM).

Intelligent cluster algorithms for changing data structures

Many real life applications are characterised by changing data structures. For example, the buying patterns of retail customers may change due to changing economical parameters (increasing oil prices motivate to buy smaller cars) or a technological break-through (replacement of analogue by digital cameras). In such dynamic environments the parameters obtained in data mining projects need to be updated to adequately describe the actual real life situation. Dynamic data mining addresses such situations. It has been applied successfully in many projects, like in traffic data analysis. In our paper, we apply the concepts of dynamic data mining to rough k-means.

Cross-Chip Probe Matching Tool: A Web-Based Tool for Linking Microarray Probes within and across Plant Species

The CCPMT is a free, web-based tool that allows plant investigators to rapidly determine if a given gene is present across various microarray platforms, which, of a list of genes, is present on array(s), and which gene a probe or probe set queries and vice versa, and to compare and contrast the gene contents of arrays. The CCPMT also maps a probe or probe sets to a gene or genes within and across species, and permits the mapping of the entire content from one array to another. By using the CCPMT, investigators will have a better understanding of the contents of arrays, a better ability to link data between experiments, ability to conduct meta-analysis and combine datasets, and an increased ability to conduct data mining projects.

Webservices oriented data mining in knowledge architecture

Massive parallelism is required for an efficient solution to data mining tasks, considering the proliferation of data and the need for high computational effort. The DisDaMin (Distributed Data Mining) project addressed distributed discovery and knowledge discovery through parallelization of data mining tasks. DisDaMin algorithms are based on the DG-ADAJ (Desktop-Grid Adaptive Application in Java), a middleware platform for Desktop Grid used as a deployment mechanism for DisDaMin algorithms. On the top of DG-ADAJ, SOA specific components could be employed to provide additional features and to improve current operation. An ESB (Enterprise Service Bus) built on the top of DG-ADAJ is going to provide improved availability, solve interoperability issues by exposing services through well established interfaces, and offer a loosely coupled infrastructure. An additional enactment layer is going to improve the performance of intelligent fragmentation of data, offering at the same time the necessary support for the execution of data workflows.

Privacy-preserving decision trees over vertically partitioned data

Privacy and security concerns can prevent sharing of data, derailing data-mining projects. Distributed knowledge discovery, if done correctly, can alleviate this problem. We introduce a generalized privacy-preserving variant of the ID3 algorithm for vertically partitioned data distributed over two or more parties. Along with a proof of security, we discuss what would be necessary to make the protocols completely secure. We also provide experimental results, giving a first demonstration of the practical complexity of secure multiparty computation-based data mining.

Decision analysis of data mining project based on Bayesian risk

Data mining, an efficient method of business intelligence, is a process to extract knowledge from large scale data. As the augment of the size of enterprise and the data, data mining as a way to make use of the data become more and more necessary. But now most of the literatures only focus on the algorithm itself. Few literatures research what qualification to fulfill before the decision doing data mining from the perspective of the company manager. This paper discusses the factors affect the data mining project. Based on the Bayesian risk, we build a model taking the risk attitude of the top executive in account to help them make decision whether to do data mining or not.

Toward data mining engineering: A software engineering approach

The number, variety and complexity of projects involving data mining or knowledge discovery in databases activities have increased just lately at such a pace that aspects related to their development process need to be standardized for results to be integrated, reused and interchanged in the future. Data mining projects are quickly becoming engineering projects, and current standard processes, like CRISP-DM, need to be revisited to incorporate this engineering viewpoint. This is the central motivation of this paper that makes the point that experience gained about the software development process over almost 40 years could be reused and integrated to improve data mining processes. Consequently, this paper proposes to reuse ideas and concepts underlying the IEEE Std 1074 and ISO 12207 software engineering model processes to redefine and add to the CRISP-DM process and make it a data mining engineering standard.

Framework for formal implementation of the business understanding phase of data mining projects

Various data mining methodologies have been proposed in the literature to provide guidance towards the process of implementing data mining projects. The methodologies describe a data mining project as comprised of a sequence of phases and highlight the particular tasks and their corresponding activities to be performed during each of the phases. It seems that the large number of tasks and activities, often presented in a checklist manner, are cumbersome to implement and may explain why all the recommended tasks are not always formally implemented. Additionally, there is often little guidance provided towards how to implement a particular task. These issues seem to be especially dominant in case of the business understanding phase which is the foundational phase of any data mining project. In this paper, we present an organizationally grounded framework to formally implement the business understanding phase of data mining projects. The framework serves to highlight the dependencies between the various tasks of this phase and proposes how and when each task can be implemented. An illustrative example of a credit scoring application from the financial sector is used to exemplify the tasks discussed in the proposed framework.

Integration of Learning Methods, Medical Literature and Expert Inspection in Medical Data Mining

From lessons learned in medical data mining projects we show that integration of advanced computation techniques and human inspection is indispensable in medical data mining. We proposed an integrated approach that merges data mining and text mining methods plus visualization support for expert evaluation. We also appropriately developed temporal abstraction and text mining methods to exploit the collected data. Furthermore, our visual discovery system D2MS allowed to actively and effectively working with physicians. Significant findings in hepatitis study were obtained by the integrated approach.

A cost model to estimate the effort of data mining projects (DMCoMo)

CRISP-DM is the standard to develop Data Mining projects. CRISP-DM proposes processes and tasks that you have to carry out to develop a Data Mining project. A task proposed by CRISP-DM is the cost estimation of the Data Mining project. In software development a lot of methods are described to estimate the costs of project development (SLIM, SEER-SEM, PRICE-S and COCOMO). These methods are not appropriate in the case of Data Mining projects because in Data Mining software development is not the first goal. Some methods have been proposed to estimate some phases of a Data Mining project, but there is no method to estimate the global cost of a generic Data Mining project. The lack of Data Mining project estimation methods is because of many real-life project failures due to the non-realistic estimation at the beginning of the projects. Consequently, in this paper we propose to design and validate a parametric cost estimation model, similar to COCOMO or SLIM in software development, for Data Mining projects (DMCoMo 1 1 The work presented in this paper has been partially supported by UPM project ERDM ref.14589. ). The drivers of the model will be proposed first and later the equation of the model will be proposed.