Sparse Databases Research Articles

MỘT THUẬT TOÁN HỮU ÍCH ĐỂ KHAI THÁC TẬP HỮU ÍCH CAO

High utility itemsets (HUIs) mining is the finding of itemsets that satisfy a user-defined minimum utility threshold. Many successful studies in this field have been carried out, however they are all reliant on Tidset techniques, which records the intersection of transactions in a data structure. This paper presents the DCHUIM algorithm which mines the high utility itemset based on the Diffset techniques. Essentially, this mechanism stores the subtraction set of transactions rather than the intersection set. In order to achieve this, a DUL data structure is proposed to store utilities information and subtraction transactions of an itemset. Furthermore, the algorithm also applies pruning strategies such as U-Prune, EUCS-Prune and the concept of closed utility to effectively compress data. Thus, in the mining process, the search space is greatly diminished. Experiment on large datasets including Accidents, Mushroom, Retail, Chainstore and compare the performance of DCHUIM algorithm with HMiner algorithm. The findings indicate that the DCHUIM method outperforms the HMiner algorithm in terms of memory utilization across all databases and outperforms it in terms of time on sparse databases.

Deep learning combined with singular value decomposition to reconstruct databases in fluid dynamics

Fluid dynamics problems are characterized by being multidimensional and nonlinear. Therefore, experiments and numerical simulations are complex and time-consuming. Motivated by this, the need arises to find new techniques to obtain data in a simpler way and in less time. In this article, we present a novel methodology based on physical principles to reconstruct databases with three, four and five dimensions, from sparse databases formed by sensor measurements. The methodology consists of combining Single Value Decomposition (SVD), which can extract the main flow dynamics, with neural networks. The neural network used is characterized by a simple architecture based on combining two autoencoders that work in parallel and are joined in the last layer. This new algorithm has been proved with three databases with different dimensions and complexities: in an Atmospheric Boundary Layer (ABL) with a turbulence model and in the flow past a two- and a three-dimensional cylinder. By applying this methodology, it has been achieved to reconstruct databases of different dimensions obtaining errors of the same order as those obtained in simulations. Summarizing, this work proposes a new hybrid physics-based machine learning model with a simple, robust and generalizable architecture, which allows reconstructing databases from very few sensors and with a very low computational cost.

Mining relevant partial periodic pattern of multi-source time series data

Traditional partial periodic pattern mining algorithms tend to work on a single time series or database. However, time series databases usually consist of interrelated multivariate time series from different sources in real application scenarios. A new pattern–relevant partial periodic patterns and its corresponding mining algorithm (PMMS-Eclat) are designed, which can effectively reflect and mine the correlations of multi-source time series data. And PMMS-Eclat introduces the contribution difference of various patterns in the mining process to quickly identify important patterns. Firstly, PMMS-Eclat, an improved mining algorithm based on Ecalt, is used to obtain frequent partial periodic patterns from single time serie, which avoids the expensive tree structure adjustment cost. In this process, two new metrics are defined to characterize the correlation among sequential data effectively, namely density ratio and average periodic rate, which respectively reflect the density and occurrence rate of patterns. Secondly, the frequent partial periodic patterns excavated above are used as input matrices, and the ultimate relevant partial periodic patterns are obtained based on LSH principle. Finally, experiments on various datasets are conducted to verify the effectiveness of our proposed algorithm, and the results show that PMMS-Eclat is more competent for sparse databases.

Evaluating the performance of Bayesian and frequentist approaches for longitudinal modeling: application to Alzheimer’s disease

Linear mixed effects (LME) modelling under both frequentist and Bayesian frameworks can be used to study longitudinal trajectories. We studied the performance of both frameworks on different dataset configurations using hippocampal volumes from longitudinal MRI data across groups—healthy controls (HC), mild cognitive impairment (MCI) and Alzheimer’s disease (AD) patients, including subjects that converted from MCI to AD. We started from a big database of 1250 subjects from the Alzheimer’s disease neuroimaging initiative (ADNI), and we created different reduced datasets simulating real-life situations using a random-removal permutation-based approach. The number of subjects needed to differentiate groups and to detect conversion to AD was 147 and 115 respectively. The Bayesian approach allowed estimating the LME model even with very sparse databases, with high number of missing points, which was not possible with the frequentist approach. Our results indicate that the frequentist approach is computationally simpler, but it fails in modelling data with high number of missing values.

Efficient mining of cross-level high-utility itemsets in taxonomy quantitative databases

In contrast to frequent itemset mining (FIM) algorithms that focus on identifying itemsets with high occurrence frequency, high-utility itemset mining algorithms can reveal the most profitable sets of items in transaction databases. Several algorithms were proposed to perform the task efficiently. Nevertheless, most of them ignore the item categorizations. This useful information is provided in many real-world transaction databases. Previous works, such as CLH-Miner and ML-HUI Miner were proposed to solve this limitation to discover cross-level and multi-level HUIs. However, the CLH-Miner has a long runtime and high memory usage. To address these drawbacks, this study extends tight upper bounds to propose effective pruning strategies. A novel algorithm named FEACP (Fast and Efficient Algorithm for Cross-level high-utility Pattern mining) is introduced, which adopts the proposed strategies to efficiently identify cross-level HUIs in taxonomy-based databases. It can be seen from a thorough performance evaluation that FEACP can identify useful itemsets of different abstraction levels in transaction databases with high efficiency, that is up to 8 times faster than the state-of-the-art algorithm on the tested sparse databases and up to 177 times on the tested dense databases. FEACP reduces memory usage by up to half over the CLH-Miner algorithm.

Efficient algorithms for mining closed high utility itemsets in dynamic profit databases

The problem of discovering high-utility itemsets (HUIs) in transaction databases, which is an extension of Frequent Itemset Mining, is a commonly encountered mining task. Researchers have proposed algorithms to efficiently mine highly profitable itemsets in customer transaction databases, in which the unit profits of items are fixed. However, this assumption does not reflect the true nature of the utility measure of items in real-life transaction databases, which might vary over time. Moreover, since this important characteristic is ignored by all the current HUI mining algorithms, they are either not applicable to this type of database, or they generate inaccurate results. In addition, the HUI mining algorithms’ traditional limitation is that they produce a huge number of HUIs for users. In this paper, we define the problem of mining a lossless, concise and compact representation of HUIs, called closed HUIs (CHUIs), in dynamic unit profit databases. Based on newly defined of utility measure, a novel algorithm, called iEFIM-Closed, is introduced. This relies on this new utility measure, a novel compact database format to reduce the cost of database scans and increase the efficiency of the mining process. Experimental evaluations show that iEFIM-Closed significantly outperforms state-of-the-art algorithms for mining CHUIs on sparse databases with dynamic profit, and has competitive performance in dense databases in terms of runtime, the cost of database scans and scalability.

Minimizing information loss in shared data: Hiding frequent patterns with multiple sensitive support thresholds

AbstractPrivacy preserving data mining (PPDM) is the process of protecting sensitive knowledge from being discovered by data mining techniques in case of data sharing. Privacy preserving frequent itemset mining (PPFIM) is a subtask and NP‐hard problem of PPDM. Its objective is to modify a given database in such a way that none of the sensitive itemsets of the database owner can be obtained by any frequent itemset mining technique from the modified database. The main challenge of PPFIM is to minimize the distortion given to the data and nonsensitive knowledge while sanitizing all given sensitive itemsets. Distortion‐based sensitive itemset hiding algorithms decrease the support of each sensitive itemset under a predefined sensitive threshold through sanitization. Most of the distortion‐based itemset hiding algorithms allow database owner to define a single sensitive threshold for each sensitive itemset. However, this is a limitation to the database owner since the importance of each sensitive itemset varies. In this paper we propose a distortion‐based itemset hiding algorithm that allows database owner to assign multiple sensitive thresholds, namely itemset oriented pseudo graph based sanitization (IPGBS) algorithm. The purpose of IPGBS algorithm is to give minimum distortion to the nonsensitive knowledge and data while hiding all sensitive itemsets. For this reason, the IPGBS algorithm modifies least amount of transaction and transaction content. The performance evaluation of the IPGBS algorithm is conducted by using two different counterparts on four different databases. The results show that the IPGBS algorithm is more efficient in terms of nonsensitive frequent itemset loss on both dense and sparse databases. It has considerable good results in terms of number of transactions modified, number of items deleted, execution time and total memory allocation as well.

Deep learning in the built environment: automatic detection of rooftop solar panels using Convolutional Neural Networks

Mapping the location and size of solar installations in urban areas can be a valuable input for policymakers and for investing in distributed energy infrastructures. Machine Learning techniques, combined with satellite and aerial imagery, allow to overcome the limitations of surveys and sparse databases in providing this mapping at large scale. In this paper we apply a supervised method based on convolutional neural networks to delineate rooftop solar panels and to detect their sizes by means of pixel-wise image segmentation. As input to the algorithm, we rely on high resolution aerial photos provided by the Swiss Federal Office of Topography. We explore different data augmentation and we vary network parameters in order to maximize model performance. Preliminary results show that we are able to automatically detect in test images the area of a set of solar panels at pixel level with an accuracy of about 0.94 and an Intersection over Union index of up to 0.64. The scalability of the trained model allows to predict the existing solar panels deployment at the Swiss national scale. The correlation with local environmental and socio-economic variables would allow to extract predictive models to foster future adoption of solar technology in urban areas.

Synthesizing High-Utility Patterns from Different Data Sources

In large organizations, it is often required to collect data from the different geographic branches spread over different locations. Extensive amounts of data may be gathered at the centralized location in order to generate interesting patterns via mono-mining the amassed database. However, it is feasible to mine the useful patterns at the data source itself and forward only these patterns to the centralized company, rather than the entire original database. These patterns also exist in huge numbers, and different sources calculate different utility values for each pattern. This paper proposes a weighted model for aggregating the high-utility patterns from different data sources. The procedure of pattern selection was also proposed to efficiently extract high-utility patterns in our weighted model by discarding low-utility patterns. Meanwhile, the synthesizing model yielded high-utility patterns, unlike association rule mining, in which frequent itemsets are generated by considering each item with equal utility, which is not true in real life applications such as sales transactions. Extensive experiments performed on the datasets with varied characteristics show that the proposed algorithm will be effective for mining very sparse and sparse databases with a huge number of transactions. Our proposed model also outperforms various state-of-the-art distributed models of mining in terms of running time.

A weighted N-list-based method for mining frequent weighted itemsets

Mining frequent itemsets (FIs) is an important problem in the field of data mining, and thus there have been many different methods proposed to solve this problem. However, mining FIs usually works on binary databases and has a limitation that is only concerned with the appearance of items regardless of their importance. In practical applications, items often have different importance depending on their values or meanings, and that leads to the emergence of weighted databases. In this paper, we propose a new method for mining frequent weighted itemsets (FWIs) from a weighted database by using the weighted N-list structure (WN-list), an extension of the N-list. Some theorems are proposed to calculate the weighted supports of itemsets fast, and then an algorithm is built based on these theorems for efficiently mining FWIs. The experimental results show that the proposed method outperforms existing methods, especially when run on very large and sparse databases.

Novel parallel method for association rule mining on multi-core shared memory systems

ShaFEM: a novel association rule mining method for multi-core shared memory systems.ShaFEM self-adapts to data characteristic to run fast on sparse and dense databases.ShaFEM uses two mining strategies and dynamically switching between them.ShaFEM applies its new lock free solution, new data structure named XFP-tree.ShaFEM is up to 5.8 times faster and 7.1 times less memory than the compared method. Association rule mining (ARM) is an important task in data mining with many practical applications. Current methods for association rule mining have shown unstable performance for different database types and under-utilize the benefits of multi-core shared memory machines. In this paper, we address these issues by presenting a novel parallel method for finding frequent patterns, the most computational intensive phase of ARM. Our proposed method, named ShaFEM, combines two mining strategies and applies the most appropriate one to each data subset of the database to efficiently adapt to the data characteristics and run fast on both sparse and dense databases. In addition, our newlock-free design minimizes the synchronization needs and maximizes the data independence to enhance the scalability. The new structure lends itself well to dynamic job scheduling resulting in a well-balanced load on the new multi-core shared memory architectures. We have evaluated ShaFEM on 12-core multi-socket servers and found that our method run up to 5.8 times faster and consumes memory up to 7.1 times less than the state-of-the-art parallel method. For some test cases, ShaFEM can save up to 4.9days of execution time over the compared method.

Efficient Algorithms for Mining Frequent Patterns from Sparse and Dense Databases

AbstractIn this article, we present a new approach for frequent pattern mining (FPM) that runs fast for both sparse and dense databases. Two algorithms, FEM and DFEM, based on our approach are also introduced. FEM applies a fixed threshold as the condition for switching between the two mining strategies; meanwhile, DFEM adopts this threshold dynamically at runtime to best fit the characteristics of the database during the mining process, especially when minimum support threshold is low. Additionally, we present optimization techniques for the proposed algorithms to speed the mining process, reduce the memory usage, and optimize the I/O cost. We also analyze in depth the performance of FEM and DFEM and compare them with several existing algorithms. The experimental results show that FEM and DFEM achieve a significant improvement in execution time and consume less memory than many popular FPM algorithms including the well-known Apriori, FP-growth, and Eclat.

An FPGA-Based Accelerator for Frequent Itemset Mining

In this article we describe a Field Programmable Gate Array (FPGA)-based coprocessor architecture for Frequent Itemset Mining (FIM). FIM is a common data mining task used to find frequently occurring subsets amongst a database of sets. FIM is a nonnumerical, data intensive computation and is used in machine learning and computational biology. FIM is particularly expensive---in terms of execution time and memory---when performed on large and/or sparse databases or when applied using a low appearance frequency threshold. Because of this, the development of increasingly efficient FIM algorithms and their mapping to parallel architectures is an active field. Previous attempts to accelerate FIM using FPGAs have relied on performance-limiting strategies such as iterative database loading and runtime logic unit reconfiguration. In this article, we present a novel architecture to implement Eclat, a well-known FIM algorithm. Unlike previous efforts, our technique does not impose limits on the maximum set size as a function of available FPGA logic resources and our design scales well to multiple FPGAs. In addition to a novel hardware design, we also present a corresponding compression scheme for intermediate results that are stored in on-chip memory. On a four-FPGA board, experimental results show up to 68X speedup compared to a highly optimized software implementation.

An adaptive approach to mining frequent itemsets efficiently

The mining of frequent itemsets is a fundamental and important task of data mining. To improve the efficiency in mining frequent itemsets, many researchers developed smart data structures to represent the database, and designed divide-and-conquers approaches to generate frequent itemsets from these data structures. However, the features of real databases are diversified and the features of local databases in the mining process may also change. Consequently, different data structures may be utilized in the mining process to enhance efficiency. This study presents an adaptive mechanism to select suitable data structures depending on database densities: the Frequent Pattern List (FPL) for sparse databases, and the Transaction Pattern List (TPL) for dense databases. Experimental results verified the effectiveness of this approach.

Signal Detection in the Pharmaceutical Industry

Drug safety profiles are dynamic and established over time using multiple, complimentary datasets and tools. The principal concern of pharmacovigilance is the detection of adverse drug reactions that are novel by virtue of their clinical nature, severity and/or frequency as soon as possible with minimum patient exposure. A key step in the process is the detection of 'signals' that direct safety reviewers to associations that might be worthy of further investigation. Although the 'prepared mind' remains the cornerstone of signal detection safety reviewers seeking potential signals by scrutinising very large, sparse databases may find themselves 'drowning in data but thirsty for knowledge'. Understandably, health authorities, pharmaceutical companies and academic centres are developing, testing and/or deploying computer-assisted database screening tools (also known as data-mining algorithms [DMAs]) to assist human reviewers. The most commonly used DMAs involve disproportionality analysis that project high-dimensional data onto two-dimensional (2 x 2) contingency tables in the context of an independence model. The objective of this paper is to extend the discussion of the evaluation, potential utility and limitations of the commonly used DMAs by providing a 'holistic' perspective on their use as one component of a comprehensive suite of signal detection strategies incorporating clinical and statistical approaches to signal detection -- a marriage of technology and the 'prepared mind'. Data-mining exercises involving spontaneous reports submitted to the US FDA will be used for illustration. Potential pitfalls and obstacles to the acceptance and implementation of data mining will be considered and suggestions for future research will be offered.

Closed constrained gradient mining in retail databases

Incorporating constraints into frequent itemset mining not only improves data mining efficiency, but also leads to concise and meaningful results. In this paper, a framework for closed constrained gradient itemset mining in retail databases is proposed by introducing the concept of gradient constraint into closed itemset mining. A tailored version of CLOSET+, LCLOSET, is first briefly introduced, which is designed for efficient closed itemset mining from sparse databases. Then, a newly proposed weaker but antimonotone measure, top-X average measure, is proposed and can be adopted to prune search space effectively. Experiments show that a combination of LCLOSET and the top-X average pruning provides an efficient approach to mining frequent closed gradient itemsets

Data categorization using decision trellises

We introduce a probabilistic graphical model for supervised learning on databases with categorical attributes. The proposed belief network contains hidden variables that play a role similar to nodes in decision trees and each of their states either corresponds to a class label or to a single attribute test. As a major difference with respect to decision trees, the selection of the attribute to be tested is probabilistic. Thus, the model can be used to assess the probability that a tuple belongs to some class, given the predictive attributes. Unfolding the network along the hidden states dimension yields a trellis structure having a signal flow similar to second order connectionist networks. The network encodes context specific probabilistic independencies to reduce parametric complexity. We present a custom tailored inference algorithm and derive a learning procedure based on the expectation-maximization algorithm. We propose decision trellises as an alternative to decision trees in the context of tuple categorization in databases, which is an important step for building data mining systems. Preliminary experiments on standard machine learning databases are reported, comparing the classification accuracy of decision trellises and decision trees induced by C4.5. In particular, we show that the proposed model can offer significant advantages for sparse databases in which many predictive attributes are missing.

Exposure Assessment Using Analytical and Numerical Models: Case Study

Exposure assessment studies attempt to link environmental pollution with increased health risk to humans. Most occurrences of past exposure have sparse databases from which to estimate health risks. Thus, analytical and numerical models can be used to help quantify estimates of past exposures. In this paper, we present a case study describing a situation in which tetrachloroethylene (PCE), a volatile organic chemical, has been detected in ground-water supplies in the Osborn Connecticut Correctional Institution (OCCI) area, including the Rye Hill Circle neighborhood, in Somers, Connecticut. Contaminant concentrations based on measured ground-water samples on the OCCI property were found to range from 2,553 parts per billion (ppb) in glacial till to 1,860 ppb in the underlying bedrock aquifer. In the residential wells tapping the same bedrock aquifer, PCE concentrations ranged from 545 ppb (well RHC-082) to below detection limits (<1 ppb). Analysis of the site included applying simplified analytical and numerical models for flow and transport in one and two dimensions and a parameter uncertainty analysis for vertical migration of contaminants. Estimates based on results of modeling analyses indicate that residential wells in the Rye Hill Circle area have probably been contaminated since their installation in 1978 throug 1981. Thus, citizens have probably been exposed to PCE-contaminated water for 16 years from 1978 through 1993, when carbon-activated filters were installed on each well. However, no acute or long-term adverse health effects, including cancer, are likely to occur as a result of these exposures. The important lesson that should be derived from this study is that the use of simplified one- and two-dimensional fate and transport models in an appropriately simplified hydrogeologic setting can yield meaningful and useful results.