Issue In Data Mining Research Articles

An outlier analysis framework for impedance-based structural health monitoring

This paper presents the use of statistically rigorous algorithms combined with active-sensing impedance methods for damage identification in engineering systems. In particular, we propose to use statistical pattern recognition methods to address damage classification and data mining issues associated with the examination of large numbers of impedance signals for health monitoring applications. In this paper, in order to diagnose damage with statistical confidence, the impedance-based monitoring is cast in the context of an outlier detection framework. A modified auto-regressive model with exogenous inputs (ARX) in the frequency domain is developed and the damage-sensitive feature is obtained by quantifying the differences between the measured impedance and the output of the ARX model. Furthermore, because of the non-Gaussian nature of the feature distribution tails, extreme value statistics is employed to develop a robust damage classifier. This paper concludes with a numerical example on a 5 degree-of-freedom system and an experimental investigation on a multi-story building model to demonstrate the performance of the proposed concept.

Discovering Golden Nuggets: Data Mining in Financial Application

With the increase of economic globalization and evolution of information technology, financial data are being generated and accumulated at an unprecedented pace. As a result, there has been a critical need for automated approaches to effective and efficient utilization of massive amount of financial data to support companies and individuals in strategic planning and investment decision-making. Data mining techniques have been used to uncover hidden patterns and predict future trends and behaviors in financial markets. The competitive advantages achieved by data mining include increased revenue, reduced cost, and much improved marketplace responsiveness and awareness. There has been a large body of research and practice focusing on exploring data mining techniques to solve financial problems. In this paper, we describe data mining in the context of financial application from both technical and application perspectives. In addition, we compare different data mining techniques and discuss important data mining issues involved in specific financial applications. Finally, we highlight a number of challenges and trends for future research in this area.

Editorial: Data Mining Lessons Learned

IntroductionData mining is concerned with finding interesting patterns in data.Many techniques have emerged for analyzing and visualizing large vol-umes of data. What one finds in the technical literature are mostlysuccess stories of these techniques. Researchers rarely report on stepsleading to success, failed attempts, or critical representation choicesmade; and rarely do papers include expert evaluations of achieved re-sults. An interesting point of investigation is also why some particularsolutions, despite good performance, were never used in practice orrequired additional treatment before they could be used. Insightfulanalyses of successful and unsuccessful applications are crucial for in-creasing our understanding of machine learning techniques and theirlimitations.The UCI Repository of Machine Learning Databases (Blake &Merz, 1998) hasserved the machine learning community for manyyearsas a valuable resource. It has benefited the community by allowingresearchers to compare algorithm performance on a common set ofbenchmark datasets, most taken from real-world domains. However,its existence has indirectly promoted a very narrow view of real-worlddata mining. Performance comparisons, which typically focus on clas-sification accuracy, neglect important data mining issues such as dataunderstanding, data preparation, selection of appropriate performancemetrics,andexpertevaluation ofresults.Furthermore,becausetheUCIrepository has been used so extensively, some researchers have claimedthat our algorithms may be “overfitting the UCI repository”.Challenge problems such as the KDD Cup, CoIL and PTE chal-lenges have also become popular in recent years and have attractednumerous participants. Contrary to the “UCI challenge” of achieving

On-Line Unsupervised Outlier Detection Using Finite Mixtures with Discounting Learning Algorithms

Outlier detection is a fundamental issue in data mining, specifically in fraud detection, network intrusion detection, network monitoring, etc. SmartSifter is an outlier detection engine addressing this problem from the viewpoint of statistical learning theory. This paper provides a theoretical basis for SmartSifter and empirically demonstrates its effectiveness. SmartSifter detects outliers in an on-line process through the on-line unsupervised learning of a probabilistic model (using a finite mixture model) of the information source. Each time a datum is input SmartSifter employs an on-line discounting learning algorithm to learn the probabilistic model. A score is given to the datum based on the learned model with a high score indicating a high possibility of being a statistical outlier. The novel features of SmartSifter are: (1) it is adaptive to non-stationary sources of datas (2) a score has a clear statistical/information-theoretic meanings (3) it is computationally inexpensives and (4) it can handle both categorical and continuous variables. An experimental application to network intrusion detection shows that SmartSifter was able to identify data with high scores that corresponded to attacks, with low computational costs. Further experimental application has identified a number of meaningful rare cases in actual health insurance pathology data from Australia's Health Insurance Commission.

Ethical issues in web data mining

Web mining refers to the whole of data mining and related techniques that are used to automatically discover and extract information from web documents and services. When used in a business context and applied to some type of personal data, it helps companies to build detailed customer profiles, and gain marketing intelligence. Web mining does, however, pose a threat to some important ethical values like privacy and individuality. Web mining makes it difficult for an individual to autonomously control the unveiling and dissemination of data about his/her private life. To study these threats, we distinguish between `content and structure mining' and `usage mining.' Web content and structure mining is a cause for concern when data published on the web in a certain context is mined and combined with other data for use in a totally different context. Web usage mining raises privacy concerns when web users are traced, and their actions are analysed without their knowledge. Furthermore, both types of web mining are often used to create customer files with a strong tendency of judging and treating people on the basis of group characteristics instead of on their own individual characteristics and merits (referred to as de-individualisation). Although there are a variety of solutions to privacy-problems, none of these solutions offers sufficient protection. Only a combined solution package consisting of solutions at an individual as well as a collective level can contribute to release some of the tension between the advantages and the disadvantages of web mining. The values of privacy and individuality should be respected and protected to make sure that people are judged and treated fairly. People should be aware of the seriousness of the dangers and continuously discuss these ethical issues. This should be a joint responsibility shared by web miners (both adopters and developers), web users, and governments.

Mining interesting association rules from customer databases and transaction databases

In this paper, we examine a new data mining issue of mining association rules from customer databases and transaction databases. The problem is decomposed into two subproblems: identifying all the large itemsets from the transaction database and mining association rules from the customer database and the large itemsets identified. For the first subproblem, we propose an efficient algorithm to discover all the large itemsets from the transaction database. Experimental results show that by our approach, the total execution time can be reduced significantly. For the second subproblem, a relationship graph is constructed according to the identified large itemsets from the transaction database and the priorities of condition attributes from the customer database. Based on the relationship graph, we present an efficient graph-based algorithm to discover interesting association rules embedded in the transaction database and the customer database.

Analysis of existing databases at the logical level

Whereas physical database tuning has received a lot of attention over the last decade, logical database tuning seems to be under-studied. We have developed a project called DBA Companion devoted to the understanding of logical database constraints from which logical database tuning can be achieved.In this setting, two main data mining issues need to be addressed: the first one is the design of efficient algorithms for functional dependencies and inclusion dependencies inference and the second one is about the interestingness of the discovered knowledge. In this paper, we point out some relationships between database analysis and data mining. In this setting, we sketch the underlying themes of our approach. Some database applications that could benefit from our project are also described, including logical database tuning.

Privacy Preserving Data Mining

In this paper we address the issue of privacy preserving data mining. Specifically, we consider a scenario in which two parties owning confidential databases wish to run a data mining algorithm on the union of their databases, without revealing any unnecessary information. Our work is motivated by the need both to protect privileged information and to enable its use for research or other purposes. The above problem is a specific example of secure multi-party computation and, as such, can be solved using known generic protocols. However, data mining algorithms are typically complex and, furthermore, the input usually consists of massive data sets. The generic protocols in such a case are of no practical use and therefore more efficient protocols are required. We focus on the problem of decision tree learning with the popular ID3 algorithm. Our protocol is considerably more efficient than generic solutions and demands both very few rounds of communication and reasonable bandwidth.

시간 데이타마이닝 프레임워크

시간 데이타마이닝은 기존 데이타마이닝에 시간 개념을 추가하여 "시간값을 가진 대용량 데이타로부터 이전에 잘 알려지지는 않았지만, 묵시적이고 잠재적으로 유용한 시간 지식을 탐사하는 기술"로 정의된다. 시간 지식이란 주기적 패턴, 캘린더 패턴, 경향 등과 같이 시간 의미와 시간 관계를 가진 지식을 말한다. 실세계에서는 환자의 병력, 상품 구매 이력, 웹 로그 등과 같은 다양한 시간 데이타가 존재하며 이로부터 여러 형태의 유용한 시간 지식을 찾아낼 수 있다. 데이타마이닝에 대한 연구가 진행되면서 순차 패턴, 유사 시계열 탐사, 주기적 연관규칙 탐사 등과 같이 시간 지식을 탐사하고자 하는 시간 데이타마이닝에 대한 부분적인 연구가 수행되었다. 그러나 기존 연구는 단순히 데이타의 발생 순서 및 유사한 패턴을 찾아내는데 중점을 두고 있어 데이타가 포함하고 있는 시간 의미와 시간 관계를 탐사하는데 부족하며, 시간 지식의 전체적인 측면보다는 연관 규칙과 같은 일부분만을 다루고 있다는 문제점을 가지고 있다. 따라서 이 논문에서는 시간 데이타마이닝에 대한 체계적인 연구를 위하여 시간 데이타마이닝에 대한 기존 연구 내용과 해결해야 할 문제점을 분석하고 이를 바탕으로 전체적인 프레임워크를 제시하였다. 또한 그 구현 방안 및 적용평가를 수행하였다. 프레임워크에서는 시간 데이타마이닝 모델을 제안하고, 이를 바탕으로 시간 데이타마이닝 질의어와 시간 지식을 탐사할 수 있는 시간 데이타마이닝 시스템을 설계하였다. Temporal data mining, the incorporation of temporal semantics to existing data mining techniques, refers to a set of techniques for discovering implicit and useful temporal knowledge from large quantities of temporal data. Temporal knowledge, expressible in the form of rules, is knowledge with temporal semantics and relationships, such as cyclic pattern, calendric pattern, trends, etc. There are many examples of temporal data, including patient histories, purchaser histories, and web log that it can discover useful temporal knowledge from. Many studies on data mining have been pursued and some of them have involved issues of temporal data mining for discovering temporal knowledge from temporal data, such as sequential pattern, similar time sequence, cyclic and temporal association rules, etc. However, all of the works treated data in database at best as data series in chronological order and did not consider temporal semantics and temporal relationships containing data. In order to solve this problem, we propose a theoretical framework for temporal data mining. This paper surveys the work to date and explores the issues involved in temporal data mining. We then define a model for temporal data mining and suggest SQL-like mining language with ability to express the task of temporal mining and show architecture of temporal mining system.

Distributed Mining of Classification Rules

Many successful data-mining techniques and systems have been developed. These techniques usually apply to centralized databases with less restricted requirements on learning and response time. Not so much effort has yet been put into mining distributed databases and real-time issues. In this paper, we investigate issues of fast-distributed data mining. We assume that merging the distributed databases into a single one would either be too costly (distributed case) or the individual fragments would be non-uniform so that mining only one fragment would bias the result (fragmented case). The goal is to classify the objects O of the database into one of several mutually exclusive classes Ci. Our approach to make mining fast and feasible is as follows. From each data site or fragment dbk, only a single rule rik is generated for each class Ci. A small subset {ri1,.....,rih} of these individual rules is selected to form a rule set Ri for each class Ci. These rule subsets represent adequately the hidden knowledge of the entire database. Various selection criteria to form Ri are discussed, both theoretically and experimentally.

Computation of the physio‐chemical properties and data mining of large molecular collections

Very large data sets of molecules screened against a broad range of targets have become available due to the advent of combinatorial chemistry. This information has led to the realization that ADME (absorption, distribution, metabolism, and excretion) and toxicity issues are important to consider prior to library synthesis. Furthermore, these large data sets provide a unique and important source of information regarding what types of molecular shapes may interact with specific receptor or target classes. Thus, the requirement for rapid and accurate data mining tools became paramount. To address these issues Pharmacopeia, Inc. formed a computational research group, The Center for Informatics and Drug Discovery (CIDD).* In this review we cover the work done by this group to address both in silico ADME modeling and data mining issues faced by Pharmacopeia because of the availability of a large and diverse collection (over 6 million discrete compounds) of drug-like molecules. In particular, in the data mining arena we discuss rapid docking tools and how we employ them, and we describe a novel data mining tool based on a ID representation of a molecule followed by a molecular sequence alignment step. For the ADME area we discuss the development and application of absorption, blood-brain barrier (BBB) and solubility models. Finally, we summarize the impact the tools and approaches might have on the drug discovery process.

Report on the workshop on research issues in data mining and knowledge discovery workshop (DMKD 2001)

This short article summarizes the program of the Sixth Workshop on Research Issues in Data Mining and Knowledge Discovery Workshop (DMKD 2001).

High dimensional wavelet smoothing

A fundamental issue in Data Mining is the development of algorithms to extract some useful information from very large databases. One important technique is to estimate a smooth surface approximating the data. However, the number of observations can be of the order of millions and there may be hundreds of variables recorded so one has to deal with the so-called "curse of dimensionality". The algorithmic complexity of this process is of the order N 3d-2 where N is the number of grid points in each dimension and d is the number of dimensions. We propose a method for approximating a high dimensional surface by computing a projection onto multiresolution spaces of low density and we demonstrate that the algorithmic complexity of the multiresolution method is proportional to ((log N ) d-1 N ) 3 --a substantial reduction in computational work. In addition, we show that the approximation error is proportional to d 2 J 2 -2J , the proportionality constant depending on the smoothness of the computed surface.

Data mining in brain imaging.

Data mining in brain imaging is proving to be an effective methodology for disease prognosis and prevention. This, together with the rapid accumulation of massive heterogeneous data sets, motivates the need for efficient methods that filter, clarify, assess, correlate and cluster brain-related information. Here, we present data mining methods that have been or could be employed in the analysis of brain images. These methods address two types of brain imaging data: structural and functional. We introduce statistical methods that aid the discovery of interesting associations and patterns between brain images and other clinical data. We consider several applications of these methods, such as the analysis of task-activation, lesion-deficit, and structure morphological variability; the development of probabilistic atlases; and tumour analysis. We include examples of applications to real brain data. Several data mining issues, such as that of method validation or verification, are also discussed.

ANTS: Agents on Networks, Trees, and Subgraphs

Efficient exploration of large networks is a central issue in data mining and network maintenance applications. In most existing work there is a distinction between the active ‘searcher’ which both executes the algorithm and holds the memory and the passive ‘searched graph’ over which the searcher has no control at all. Large dynamic networks like the Internet, where the nodes are powerful computers and the links have narrow bandwidth and are heavily-loaded, call for a different paradigm, in which a noncentralized group of one or more lightweight autonomous agents traverse the network in a completely distributed and parallelizable way. Potential advantages of such a paradigm would be fault tolerance against network and agent failures, and reduced load on the busy nodes due to the small amount of memory and computing resources required by the agent in each node. Algorithms for network covering based on this paradigm could be used in today’s Internet as a support for data mining and network control algorithms. Recently, a vertex ant walk ( VAW) method has been suggested [I.A. Wagner, M. Lindenbaum, A.M. Bruckstein, Ann. Math. Artificial Intelligence 24 (1998) 211–223] for searching an undirected, connected graph by an a(ge)nt that walks along the edges of the graph, occasionally leaving ‘pheromone’ traces at nodes, and using those traces to guide its exploration. It was shown there that the ant can cover a static graph within time nd, where n is the number of vertices and d the diameter of the graph. In this work we further investigate the performance of the VAW method on dynamic graphs, where edges may appear or disappear during the search process. In particular we prove that (a) if a certain spanning subgraph S is stable during the period of covering, then the VAW method is guaranteed to cover the graph within time nd s, where d s is the diameter of S, and (b) if a failure occurs on each edge with probability p, then the expected cover time is bounded from above by nd(( log Δ / log(1/p))+((1+p)/(1−p))) , where Δ is the maximum vertex degree in the graph. We also show that (c) if G is a static tree then it is covered within time 2 n.

Structural analysis of transition metal beta-X substituent interactions. Toward the use of soft computing methods for catalyst modeling

Fuzzy logic and neural network techniques are used to classify intramolecular interactions between transition metals (M) and beta-X substituents in the following structural motif (LnMC(alpha)(A1)(A2)-C(beta)(B1)(B2)X). These interactions are relevant to the direct polymerization of functionalized olefins by Ziegler-Natta (ZN) catalysis. The efficiency and effectiveness of different soft computing techniques are compared. These methods give not only encouraging results with respect to general data mining issues but also insight into the factors that effect interactions between transition metals and beta-X substituents.

Advances and Research Directions in Data-Warehousing Technology

Information is one of the most valuable assets of an organisation and when used properly can assist in intelligent decision making that can significantly improve the functioning of an organisation. Data Warehousing is a recent technology that allows information to be easily and efficiently accessed for decision-making activities by collecting data from many operational, legacy and possibly heterogeneous data sources. On-Line Analytical Processing (OLAP) tools are well-suited for complex data analysis, such as multi-dimensional data analysis, and to assist in decision support activities while data mining tools take the process one step further and actively search the data for patterns and hidden knowledge in the data held in the warehouse. Many organisations are building, or are planning to develop, a data warehouse for their operational and decision support needs. In this paper, we present an overview of data warehousing, multi-dimensional databases, OLAP and data mining technology and discuss the directions of current research in the area. We also discuss recent developments in data warehouse modelling, view selection and maintenance, indexing schemes, parallel query processing and data mining issues. A number of technical issues for exploratory research are presented and possible solutions are also discussed.

The corporate library on every desktop

Looks at the development of Telecom New Zealand’s library Internet service. Considers design, philosophy and architecture of the system, before outlining content, retrieval modules, data mining and monitoring issues. Outlines future directions for consideration.

Data mining using extensions of the rough set model

This article examines basic issues of data mining using the theory of rough sets, which is a recent proposal for generalizing classical set theory. The Pawlak rough set model is based on the concept of an equivalence relation. Recent research has shown that a generalized rough set model need not be based on equivalence relation axioms. The Pawlak rough set model has been used for deriving deterministic as well as probabilistic rules from a complete database. This article demonstrates that a generalized rough set model can be used for generating rules from incomplete databases. These rules are based on plausibility functions proposed by Shafer. The article also discusses the importance of rule extraction from incomplete databases in data mining. © 1998 John Wiley & Sons, Inc.