Uncertain Databases Research Articles

A weighted frequent itemset mining algorithm for intelligent decision in smart systems

Intelligent decision is the key technology of smart systems. Data mining technology has been playing an increasingly important role in decision-making activities. Frequent itemset mining (FIM), as an important step of association rule analysis, is becoming one of the most important research fields in data mining. Weighted FIM in uncertain databases should take both existential probability and importance of items into account in order to find frequent itemsets of great importance to users. However, the introduction of weight makes the weighted frequent itemsets not satisfy the downward closure property any longer. As a result, the search space of frequent itemsets cannot be narrowed according to downward closure property which leads to a poor time efficiency. In this paper, the weight judgment downward closure property for the weighted frequent itemsets and the existence property of weighted frequent subsets are introduced and proved first. Based on these two properties, the Weight judgment downward closure property-based FIM (WD-FIM) algorithm is proposed to narrow the searching space of the weighted frequent itemsets and improve the time efficiency. Moreover, the completeness and time efficiency of WD-FIM algorithm are analyzed theoretically. Finally, the performance of the proposed WD-FIM algorithm is verified on both synthetic and real-life data sets.

Mining Weighted Frequent Itemsets without Candidate Generation in Uncertain Databases

Frequent itemset mining (FIM) is a fundamental set of techniques used to discover useful and meaningful relationships between items in transaction databases. In recent decades, extensions of FIM such as weighted frequent itemset mining (WFIM) and frequent itemset mining in uncertain databases (UFIM) have been proposed. WFIM considers that items may have different weight/importance. It can thus discover itemsets that are more useful and meaningful by ignoring irrelevant itemsets with lower weights. UFIM takes into account that data collected in a real-life environment may often be inaccurate, imprecise, or incomplete. Recently, these two ideas have been combined in the HEWI-Uapriori algorithm. This latter considers both item weights and transaction uncertainty to mine the high expected weighted itemsets (HEWIs) using a two-phase Apriori-based approach. Although the upper-bound proposed in HEWI-Uapriori can reduce the size of the search space, it still generates a large amount of candidates and uses a level-wise search. In this paper, a more efficient algorithm named HEWI-Utree is developed to efficiently mine HEWIs without performing multiple database scans and without generating candidates. This algorithm relies on three novel structures named element (E)-table, weighted-probability (WP)-table and WP-tree to maintain the information required for identifying and pruning unpromising itemsets early. Experimental results show that the proposed algorithm is generally much more efficient than traditional methods for WFIM and UFIM, as well as the state-of-the-art HEWI-Uapriori algorithm, in terms of runtime, memory consumption, and scalability.

Reducing Uncertainty of Probabilistic Top-$k$ Ranking via Pairwise Crowdsourcing

Probabilistic top- $k$ ranking is an important and well-studied query operator in uncertain databases. However, the quality of top- $k$ results might be heavily affected by the ambiguity and uncertainty of the underlying data. Uncertainty reduction techniques have been proposed to improve the quality of top- $k$ results by cleaning the original data. Unfortunately, most data cleaning models aim to probe the exact values of the objects individually and therefore do not work well for subjective data types, such as user ratings, which are inherently probabilistic. In this paper, we propose a novel pairwise crowdsourcing model to reduce the uncertainty of top- $k$ ranking using a crowd of domain experts. Given a crowdsourcing task of limited budget, we propose efficient algorithms to select the best object pairs for crowdsourcing that will bring in the highest quality improvement. Extensive experiments show that our proposed solutions outperform a random selection method by up to 30 times in terms of quality improvement of probabilistic top- $k$ ranking queries. In terms of efficiency, our proposed solutions can reduce the elapsed time of a brute-force algorithm from several days to one minute.

Probabilistic group nearest neighbor query optimization based on classification using ELM

Abstract The probabilistic group nearest neighbor(PGNN) query , which returns all the uncertain objects whose probabilities of being the group nearest neighbor (GNN) results exceed a user-specified threshold, is widely used in uncertain database. Most existing work for answering PGNN queries adopted a general framework which consist of three phases: spatial pruning, probabilistic pruning, refinement . In the probabilistic pruning phase, dividing the uncertain regions into many partitions to derive a tighter probabilities bounds is a common method. However, there is a tradeoff between the computational cost of probabilistic pruning phase and refinement phase controlled by the granularity of the partitions. In this paper, we study the problem of setting the optimal granularity of the partitions for uncertain objects, and propose a new framework for PGNN queries based on granularity classification using ELM such that the overall cost is minimized. In addition, to improve the accuracy of classification and make the classifier applicable to the dynamic environment, a plurality voting method and a dynamic classification strategy are proposed respectively. Extensive experiments shows that compared with the default granularities of the partitions, the granularities chosen by ELM classifiers are more proper, which further improves the performance of PGNN query algorithm. In addition, ELM outperforms SVM with regard to both the response time and classification accuracy.

FIDOOP: PARALLEL MINING OF FREQUENT ITEM SETS USING MAPREDUCE

Due to the exponential increase of real-time data monitoring systems, the extraction of frequent item (Frequent item set mining) set from large uncertain database is the challenging task. The existing parallel mining algorithm for frequent item sets includes the limitations in terms of more memory usage and excessive run time even for less amount of data. To overcome this problem, the FiDoop based item set mining algorithm is proposed by using map reduce framework. It is used to improve the performance of load balancing operation in an uncertain database for computing frequent patterns. This system includes data uploading, preprocessing, threshold, find support and confidence, merge and result. Initially, the data is selected from the dataset and uploaded in the server. Afterwards, the preprocessing stage removes columns which contains unwanted entries. The information is analyzed and partitioned to compute threshold value. These data are classified depends on threshold values and the clustering algorithm is used to find high support and confidence values among clusters to discover frequent item. Finally, those frequent item sets are merged to acquire a frequent pattern. The proposed system is mainly developed for improving the accuracy and it is evaluated based on the performance measures of accuracy, memory usage and execution time.

A forward scan based plane sweep algorithm for parallel interval joins

The interval join is a basic operation that finds application in temporal, spatial, and uncertain databases. Although a number of centralized and distributed algorithms have been proposed for the efficient evaluation of interval joins, classic plane sweep approaches have not been considered at their full potential. A recent piece of related work proposes an optimized approach based on plane sweep (PS) for modern hardware, showing that it greatly outperforms previous work. However, this approach depends on the development of a complex data structure and its parallelization has not been adequately studied. In this paper, we explore the applicability of a largely ignored forward scan (FS) based plane sweep algorithm, which is extremely simple to implement. We propose two optimizations of FS that greatly reduce its cost, making it competitive to the state-of-the-art single-threaded PS algorithm while achieving a lower memory footprint. In addition, we show the drawbacks of a previously proposed hash-based partitioning approach for parallel join processing and suggest a domain-based partitioning approach that does not produce duplicate results. Within our approach we propose a novel breakdown of the partition join jobs into a small number of independent mini-join jobs with varying cost and manage to avoid redundant comparisons. Finally, we show how these mini-joins can be scheduled in multiple CPU cores and propose an adaptive domain partitioning, aiming at load balancing. We include an experimental study that demonstrates the efficiency of our optimized FS and the scalability of our parallelization framework.

Mining of high utility-probability sequential patterns from uncertain databases.

High-utility sequential pattern mining (HUSPM) has become an important issue in the field of data mining. Several HUSPM algorithms have been designed to mine high-utility sequential patterns (HUPSPs). They have been applied in several real-life situations such as for consumer behavior analysis and event detection in sensor networks. Nonetheless, most studies on HUSPM have focused on mining HUPSPs in precise data. But in real-life, uncertainty is an important factor as data is collected using various types of sensors that are more or less accurate. Hence, data collected in a real-life database can be annotated with existing probabilities. This paper presents a novel pattern mining framework called high utility-probability sequential pattern mining (HUPSPM) for mining high utility-probability sequential patterns (HUPSPs) in uncertain sequence databases. A baseline algorithm with three optional pruning strategies is presented to mine HUPSPs. Moroever, to speed up the mining process, a projection mechanism is designed to create a database projection for each processed sequence, which is smaller than the original database. Thus, the number of unpromising candidates can be greatly reduced, as well as the execution time for mining HUPSPs. Substantial experiments both on real-life and synthetic datasets show that the designed algorithm performs well in terms of runtime, number of candidates, memory usage, and scalability for different minimum utility and minimum probability thresholds.

Consistent Query Answering for Self-Join-Free Conjunctive Queries Under Primary Key Constraints

A relational database is said to be uncertain if primary key constraints can possibly be violated. A repair (or possible world) of an uncertain database is obtained by selecting a maximal number of tuples without ever selecting two distinct tuples with the same primary key value. For any Boolean query q , CERTAINTY( q ) is the problem that takes an uncertain database db as input and asks whether q is true in every repair of db . The complexity of this problem has been particularly studied for q ranging over the class of self-join-free Boolean conjunctive queries. A research challenge is to determine, given q , whether CERTAINTY( q ) belongs to complexity classes FO , P , or coNP -complete. In this article, we combine existing techniques for studying this complexity classification task. We show that, for any self-join-free Boolean conjunctive query q , it can be decided whether or not CERTAINTY( q ) is in FO . We additionally show how to construct a single SQL query for solving CERTAINTY( q ) if it is in FO . Further, for any self-join-free Boolean conjunctive query q , CERTAINTY( q ) is either in P or coNP -complete and the complexity dichotomy is effective. This settles a research question that has been open for 10 years.

Reporting l most influential objects in uncertain databases based on probabilistic reverse top-k queries

Reverse top-k queries are proposed from the perspective of a product manufacturer, which are essential for manufacturers to assess the potential market. However, the existing approaches for reverse top-k queries are all based on the assumption that the underlying data are exact (or certain). Due to the intrinsic differences between uncertain and certain data, these methods cannot be applied to process uncertain data sets directly. Motivated by this, in this paper, we firstly model the probabilistic reverse top-k queries over uncertain data. Moreover, we formulate a probabilistic top-l influential query, that reports the l most influential objects having the largest impact factors, where the impact factor of an object is defined as the cardinality of its probabilistic reverse top-k query result set. We present effective pruning heuristics for speeding up the queries. Particularly, we exploit several properties of probabilistic threshold top-k queries and probabilistic skyline queries to reduce the search space of this problem. In addition, an upper bound of the potential users is estimated to reduce the cost of computing the probabilistic reverse top-k queries for the candidate objects. Finally, efficient query algorithms are presented seamlessly with integration of the proposed pruning strategies. Extensive experiments using both real-world and synthetic data sets demonstrate the efficiency and effectiveness of our proposed algorithms.

Selecting skyline stars over uncertain databases: Semantics and refining methods in the evidence theory setting

In recent years, a great attention has been paid to skyline computation over uncertain data. In this paper, we study how to conduct advanced skyline analysis over uncertain databases where uncertainty is modeled thanks to the evidence theory (a.k.a., belief functions theory). We particularly tackle an important issue, namely the skyline stars (denoted by SKY2) over the evidential data. This kind of skyline aims at retrieving the best evidential skyline objects (or the stars). Efficient algorithms have been developed to compute the SKY2. Extensive experiments have demonstrated the efficiency and effectiveness of our proposed approaches that considerably refine the huge skyline. In addition, the conducted experiments have shown that our algorithms significantly outperform the basic skyline algorithms in terms of CPU and memory costs.

Two new Top-k queries in uncertain database

Two new Top-k queries in uncertain database

Discovering Top-k Probabilistic Frequent Itemsets from Uncertain Databases

Probabilistic frequent itemset mining is to find the itemsets with support larger than the threshold with a given probabilistic confidence within an uncertain database. Nevertheless, when the threshold is smaller, the mining results will be massive, which are not easy to understand by the users. In this paper, we focus on this problem and propose a method to achieve the top-k probabilistic frequent itemsets, which, to our best knowledge, has never been addressed before. A scoring function is defined to evaluate the level of itemsets. We introduce a compacted data structure, named TopKPFITree, to maintain the mining results and some other information. Furthermore, an efficient algorithm TopKPFIM is proposed to build the TopKPFITree and get the results. Our experimental results over uncertain datasets show that our algorithm significantly outperform the Naive algorithm.

Framework for Managing Uncertain Distributed Categorical Data

In recent years, data has become uncertain due to the flourishing advanced technologies that participate continuously and increasingly in producing large amounts of incomplete data. Often, many modern applications where uncertainty occurs are distributed in nature, e.g., distributed sensor networks, information extraction, data integration, social network, etc. Consequently, even though the data uncertainty has been studied in the past for centralized behavior, it is still a challenging issue to manage uncertainty over the data in situ. In this paper, we propose a framework to managing uncertain categorical data over distributed environments that is built upon a hierarchical indexing technique based on inverted index, and a distributed algorithm to efficiently process queries on uncertain data in distributed environment. Leveraging this indexing technique, we address two kinds of queries on the distributed uncertain databases 1) a distributed probabilistic thresholds query, where its answers satisfy the probabilistic threshold requirement; and 2) a distributed top-k-queries, optimizing, the transfer of the tuples from the distributed sources to the coordinator site and the time treatment. Extensive experiments are conducted to verify the effectiveness and efficiency of the proposed method in terms of communication costs and response time.

A new efficient approach for mining uncertain frequent patterns using minimum data structure without false positives

The concept of uncertain pattern mining was recently proposed to fulfill the demand for processing databases with uncertain data, and various relevant methods have been devised. However, previous approaches have the following limitations. State-of-the-art methods based on tree structure can cause fatal problems in terms of runtime and memory usage according to the characteristics of uncertain databases and threshold settings because their own tree data structures can become excessively large and complicated in their mining processes. Various approximation approaches have been suggested in order to overcome such problems; however, they are methods that increase their own mining performance at the cost of accuracy of the mining results. In order to solve the problems, we propose an exact, efficient algorithm for mining uncertain frequent patterns based on novel data structures and mining techniques, which can also guarantee the correctness of the mining results without any false positives. The newly proposed list-based data structures and pruning techniques allow a complete set of uncertain frequent patterns to be mined more efficiently without pattern losses. We also demonstrate that the proposed algorithm outperforms previous state-of-the art approaches in both theoretical and empirical aspects. Especially, we provide analytical results of performance evaluation for various types of datasets to show efficiency of runtime, memory usage, and scalability in our method.

Bounds on skyline probability for databases with uncertain preferences

For determining skyline objects for an uncertain database with uncertain preferences, it is necessary to compute the skyline probability of a given object with respect to other objects. The problem boils down to computing the probability of the union of events from the probabilities of all possible joint probabilities. Linear Bonferroni bound is concerned with computing the bounds on the probability of the union of events with partial information. We use this technique to estimate the skyline probability of an object and propose a polynomial-time algorithm for computing sharp upper bound. We show that the use of partial information does not affect the quality of solution but helps in improving the efficiency. We formulate the problem as a Linear Programming Problem (LPP) and characterize a set of feasible points that is believed to contain all extreme points of the LPP. The maximization of the objective function over this set of points is equivalent to a bi-polar quadratic optimization problem. We use a spectral relaxation technique to solve the bi-polar quadratic optimization problem. The proposed algorithm is of O(n3) time complexity and is the first ever polynomial-time algorithm to determine skyline probability. We show that the bounds computed by our proposed algorithm determine almost the same set of skyline objects as that with the deterministic algorithm. Experimental results are presented to corroborate this claim.

Efficient pruning for top-K ranking queries on attribute-wise uncertain datasets

Top-K ranking queries in uncertain databases aim to find the top-K tuples according to a ranking function. The interplay between score and uncertainty makes top-K ranking in uncertain databases an intriguing issue, leading to rich query semantics. Recently, a unified ranking framework based on parameterized ranking functions (PRFs) has been formulated, which generalizes many previously proposed ranking semantics. Under the PRFs based ranking framework, efficient pruning approach for Top-K ranking on datasets with tuple-wise uncertainty has been well studied in the literature. However, this cannot be applied to top-K ranking on datasets with attribute-wise uncertainty, which are often natural and useful in analyzing uncertain data in many applications. This paper aims to develop efficient pruning techniques for top-K ranking on datasets with attribute-wise uncertainty under the PRFs based ranking framework, which has not been well studied in the literature. We first develop a Tuple Insertion Based Algorithm for computing each tuple's PRF value, which reduce the time cost from the state of the art cubic order of magnitude to quadratic order of magnitude. Based on the Tuple Insertion Based Algorithm, three pruning strategies are developed to further reduce the time cost. The mathematics of deriving the Tuple Insertion Based Algorithm and corresponding pruning strategies are also presented. At last, we show that our pruning algorithms can also be applied to the computation of the top-k aggregate queries. The experimental results on both real and synthetic data demonstrate the effectiveness and efficiency of the proposed pruning techniques.

Mining interesting patterns from uncertain databases

Due to a growing demand for efficient algorithms for mining frequent itemsets from uncertain databases, several approaches have been proposed in recent years, but all of them use support-based constraints to prune the combinatorial search space. Most real life databases contain data whose correctness is uncertain. The support-based constraint alone is not enough, because the frequent itemsets may have weak affinity. Even a very high minimum support is not effective for finding correlated patterns with increased weight or support affinity. There are a few approaches in precise databases that propose new measures to mine correlated patterns, but they are not applicable in uncertain databases because certain and uncertain databases differ both semantically and computationally. In this paper, we propose a new strategy: Weighted Uncertain Interesting Pattern Mining (WUIPM), in which a tree structure (WUIP-tree) and several new measures (e.g., uConf, wUConf) are suggested to mine correlated patterns from uncertain databases. To our knowledge, ours is the first work specifically to consider weight or importance of an individual item alongside correlation between items of patterns in uncertain databases. Additionally, we propose a new metric, prefix proxy value, pProxy for our WUIP-tree that helps improve the mining performance. A comprehensive performance study shows that our strategy (a) generates fewer but valuable patterns and (b) is faster than existing approaches even when affinity measures are not applied.

Efficient algorithms for mining high-utility itemsets in uncertain databases

High-utility itemset mining (HUIM) is a useful set of techniques for discovering patterns in transaction databases, which considers both quantity and profit of items. However, most algorithms for mining high-utility itemsets (HUIs) assume that the information stored in databases is precise, i.e., that there is no uncertainty. But in many real-life applications, an item or itemset is not only present or absent in transactions but is also associated with an existence probability. This is especially the case for data collected experimentally or using noisy sensors. In the past, many algorithms were respectively proposed to effectively mine frequent itemsets in uncertain databases. But mining HUIs in an uncertain database has not yet been proposed, although uncertainty is commonly seen in real-world applications. In this paper, a novel framework, named potential high-utility itemset mining (PHUIM) in uncertain databases, is proposed to efficiently discover not only the itemsets with high utilities but also the itemsets with high existence probabilities in an uncertain database based on the tuple uncertainty model. The PHUI-UP algorithm (potential high-utility itemsets upper-bound-based mining algorithm) is first presented to mine potential high-utility itemsets (PHUIs) using a level-wise search. Since PHUI-UP adopts a generate-and-test approach to mine PHUIs, it suffers from the problem of repeatedly scanning the database. To address this issue, a second algorithm named PHUI-List (potential high-utility itemsets PU-list-based mining algorithm) is also proposed. This latter directly mines PHUIs without generating candidates, thanks to a novel probability-utility-list (PU-list) structure, thus greatly improving the scalability of PHUI mining. Substantial experiments were conducted on both real-life and synthetic datasets to assess the performance of the two designed algorithms in terms of runtime, number of patterns, memory consumption, and scalability.

Multi-algorithmic fingerprint model for cancellable template generation

In this paper we present a multi-algorithmic biometric system comprising the fingerprint biometrics to tackle highly uncertain databases. Two methods were proposed namely, spiral-based method and block-based method to generate cancellable fingerprint templates. Spiral-based method can be used to exploit elements in each contiguous right angled triangle, that are drawn using the invariant distances between reference minutia to every other minutia. In the block-based method, fingerprint image can be tessellated into blocks to get a set of features by considering Euclidean distance from each minutia in reference block to other minutia in blocks which are horizontal, vertical, and diagonal in direction to reference block, and orientation of minutiae. Two biometric methods are fused at score level based on Yager's triangular norm. The experimental evaluations and analysis are performed using the FVC2002 databases. The results show better performance and separation between genuine and imposter distributions.

Calibration of catalyst temperature in automotive engines over coldstart operation in the presence of different random noises and uncertainty: Implementation of generalized Gaussian process regression machine

The main scope of the current study is to develop a systematic stochastic model to capture the undesired uncertainty and random noises on the key parameters affecting the catalyst temperature over the coldstart operation of automotive engine systems. In the recent years, a number of articles have been published which aim at the modeling and analysis of automotive engines’ behavior during coldstart operations by using regression modeling methods. Regarding highly nonlinear and uncertain nature of the coldstart operation, calibration of the engine system’s variables, for instance the catalyst temperature, is deemed to be an intricate task, and it is unlikely to develop an exact physics-based nonlinear model. This encourages automotive engineers to take advantage of knowledge-based modeling tools and regression approaches. However, there exist rare reports which propose an efficient tool for coping with the uncertainty associated with the collected database. Here, the authors introduce a random noise to experimentally derived data and simulate an uncertain database as a representative of the engine system’s behavior over coldstart operations. Then, by using a Gaussian process regression machine (GPRM), a reliable model is used for the sake of analysis of the engine’s behavior. The simulation results attest the efficacy of GPRM for the considered case study. The research outcomes confirm that it is possible to develop a practical calibration tool which can be reliably used for modeling the catalyst temperature.