Set-valued Attributes Research Articles

A Study on Improving Metrics of Information Loss in Anonymization of Semi-structured Transaction Data with Set-valued Attributes

A Study on Improving Metrics of Information Loss in Anonymization of Semi-structured Transaction Data with Set-valued Attributes

Selectivity Estimation for Queries Containing Predicates over Set-Valued Attributes

Selectivity estimation aims to estimate the size of query results size accurately and efficiently. Despite being a well-researched area for decades, most existing estimators are designed to handle comparison predicates over numeric and categorical data. Nevertheless, Set-valued data are ubiquitous in many applications such as information retrieval and recommendation systems. However, these estimators may not be effective for handling predicates over set-valued data. In this work, we presents novel techniques for selectivity estimation on queries involving predicates over set-valued attributes. We first propose the set-valued column factorization problem, whereby each each set-valued column is converted to multiple numeric subcolumns, and set containment predicates are converted to numeric comparison predicates. This enables us to leverage any existing estimator to perform selectivity estimation. We then develop two methods for column factorization and query conversion, namely ST and STH. We integrate ST and STH into three estimators, Postgres, Neurocard, and DeepDB. We then conduct a comprehensive empirical analysis by comparing our approach against three baselines across three different datasets. The experimental results demonstrate that our methods exhibit superior estimation accuracy while maintaining high efficiency compared to the baseline techniques.

Internal and external memory set containment join

A set containment join operates on two set-valued attributes with a subset ( $$\subseteq $$ ) relationship as the join condition. It has many real-world applications, such as in publish/subscribe services and inclusion dependency discovery. Existing solutions can be broadly classified into union-oriented and intersection-oriented methods. Based on several recent studies, union-oriented methods are not competitive as they involve an expensive subset enumeration step. Intersection-oriented methods build an inverted index on one attribute and perform inverted list intersection on another attribute. Existing intersection-oriented methods intersect inverted lists one-by-one. In contrast, in this paper, we propose to intersect all the inverted lists simultaneously while skipping many irrelevant entries in the lists. To share computation, we utilize the prefix tree structure and extend our novel list intersection method to operate on the prefix tree. To further improve the efficiency, we propose to partition the data and process each partition separately. Each partition will be associated with a much smaller inverted index, and the set containment join cost can be significantly reduced. Moreover, to support large-scale datasets that are beyond the available memory space, we develop a novel adaptive data partition method that is designed to fully leverage the available memory and achieve high I/O efficiency, and thereby exhibiting outstanding performance for external memory set containment join. We evaluate our methods using both real-world and synthetic datasets. Experimental results demonstrate that our method outperforms state-of-the-art methods by up to 10 $$\times $$ when the dataset is completely resided in memory. Furthermore, our approach achieves up to two orders of magnitude improvement on I/O efficiency compared with a baseline method when the dataset size exceeds the main memory space.

Designing efficient algorithms for querying large corpora

I describe several new efficient algorithms for querying large annotated corpora. The search algorithms as they are implemented in several popular corpus search engines are less than optimal in two respects: regular expression string matching in the lexicon is done in linear time, and regular expressions over corpus positions are evaluated starting in those corpus positions that match the constraints of the initial edges of the corresponding network. To address these shortcomings, I have developed an algorithm for regular expression matching on suffix arrays that allows fast lexicon lookup, and a technique for running finite state automata from edges with lowest corpus counts. The implementation of the lexicon as suffix array also lends itself to an elegant and efficient treatment of multi-valued and set-valued attributes. The described techniques have been implemented in a fully functional corpus management system and are also used in a treebank query system.

FreshJoin: An Efficient and Adaptive Algorithm for Set Containment Join

This paper revisits set containment join (SCJ) problem, which uses the subset relationship (i.e., subseteq) as condition to join set-valued attributes of two relations and has many fundamental applications in commercial and scientific fields. Existing in-memory algorithms for SCJ are either signature-based or prefix-tree-based. The former incurs high CPU cost because of the enumeration of signatures, while the latter incurs high space cost because of the storage of prefix trees. This paper proposes a new adaptive parameter-free in-memory algorithm, named as frequency-hashjoin or {mathsf {FreshJoin}} in short, to evaluate SCJ efficiently. {mathsf {FreshJoin}} builds a flat index on-the-fly to record three kinds of signatures (i.e., two least frequent elements and a hash signature whose length is determined adaptively by the frequencies of elements in the universe set). The index consists of two sparse inverted indices and two arrays which record hash signatures of all sets in each relation. The index is well organized such that {mathsf {FreshJoin}} can avoid enumerating hash signatures. The rationality of this design is explained. And, the time and space cost of the proposed algorithm, which provide a rule to choose {mathsf {FreshJoin}} from existing algorithms, are analyzed. Experiments on 16 real-life datasets show that {mathsf {FreshJoin}} usually reduces more than 50% of space cost while remains as competitive as the state-of-the-art algorithms in running time.

A graph-based multifold model for anonymizing data with attributes of multiple types

Transactional data with attributes of multiple types may be extremely useful to secondary analysis (e.g., learning models and finding patterns). However, anonymization of such data is challenging because it contains multiple types of attributes (e.g., relational and set-valued attributes). Existing privacy-preserving techniques are not applicable to address this problem. In this paper, we propose a novel graph-based multifold model to anonymize data with attributes of multiple types. Under this model, such data are modelled as a graph, and multifold privacy is guaranteed through fuzzing on sensitive attributes and converting associations among items into an uncertain form. Specifically, we define a multi-objective attack model in a graph and devise a safety parameter and algorithm to prevent such attacks. Experiments have been performed on real-life data sets to evaluate the performance.

A fuzzy SV-k-modes algorithm for clustering categorical data with set-valued attributes

In this paper, we propose a fuzzy SV-k-modes algorithm that uses the fuzzy k-modes clustering process to cluster categorical data with set-valued attributes. In the proposed algorithm, we use Jaccard coefficient to measure the dissimilarity between two objects and represent the center of a cluster with set-valued modes. A heuristic update way of cluster prototype is developed for the fuzzy partition matrix. These extensions make the fuzzy SV-k-modes algorithm can cluster categorical data with single-valued and set-valued attributes together and the fuzzy k-modes algorithm is its special case. Experimental results on the synthetic data sets and the three real data sets from different applications have shown the efficiency and effectiveness of the fuzzy SV-k-modes algorithm.

Efficient parallel boolean matrix based algorithms for computing composite rough set approximations

In information systems, there may exist multiple different types of attributes like categorical attributes, numerical attributes, set-valued attributes, interval-valued attributes, missing attributes, etc. Such information systems are called as composite information systems. To process such attributes with rough set theory, composite rough set model and corresponding matrix methods were introduced in our previous study. Calculation of rough set approximations of a concept is the key step for rule acquisition and attribute reduction in rough set based methods. To accelerate the computation process of rough set approximations, this paper first presents the boolean matrix representation of the lower and upper approximations in the composite information system, then designs a parallel method for computing approximations based on matrix, and implements it on Multi-GPU. The experiments on data sets from UCI and user-defined data sets show that the proposed method can accelerate the computation process efficiently. The Multi-GPU implementation achieves up to a speedup of 334.9 over the CPU implementation.

A Locality Sensitive Hashing Technique for Categorical Data

The measured data may contain various types of attributes such as continuous, categorical, and set-valued attributes. Several locality-sensitive hashing techniques, which enable to find similar pairs of data in a fast and approximate way, have been developed for data with either numeric or set-valued attributes. This paper introduces a new locality sensitive-hashing technique applicable to data with categorical attributes.

On the Signature Tree Construction and Analysis

Advanced database application areas, such as computer aided design, office automation, digital libraries, data-mining, as well as hypertext and multimedia systems, need to handle complex data structures with set-valued attributes, which can be represented as bit strings, called signatures. A set of signatures can be stored in a file, called a signature file. In this paper, we propose a new method to organize a signature file into a tree structure, called a signature tree, to speed up the signature file scanning and query evaluation. In addition, the average time complexity of searching a signature tree is analyzed and how to maintain a signature tree on disk is discussed. We also conducted experiments, which show that the approach of signature trees provides a promising index structure

A performance study of four index structures for set-valued attributes of low cardinality

Abstract.The efficient retrieval of data items on set-valued attributes is an important research topic that has attracted little attention so far. We studied and modified four index structures (sequential signature files, signature trees, extendible signature hashing, and inverted files) for a fast retrieval of sets with low cardinality. We compared the index structures by implementing them and subjecting them to extensive experiments, investigating the influence of query set size, database size, domain size, and data distribution (synthetic and real). The results of the experiments clearly indicate that inverted files exhibit the best overall behavior of all tested index structures.

Adaptive algorithms for set containment joins

A set containment join is a join between set-valued attributes of two relations, whose join condition is specified using the subset (⊆) operator. Set containment joins are deployed in many database applications, even those that do not support set-valued attributes. In this article, we propose two novel partitioning algorithms, called the Adaptive Pick-and-Sweep Join (APSJ) and the Adaptive Divide-and-Conquer Join (ADCJ), which allow computing set containment joins efficiently. We show that APSJ outperforms previously suggested algorithms for many data sets, often by an order of magnitude. We present a detailed analysis of the algorithms and study their performance on real and synthetic data using an implemented testbed.

User profiling for the melvil knowledge retrieval system

Melvil is an ontology-based knowledge retrieval platform that provides a three-dimensional visualization of search results. The user can tailor the presentation of the search results to his or her preferences by changing the settings of various parameters on the screen. In this paper, we report on a prototype implementation of a user profiling device that learns to predict appropriate settings for these parameters for the current search results based on previous experiences. In a preliminary study, we evaluated several off-the-shelf machine learning algorithms on parts of the problem. The final implementation required the flexibility of handling both regression and classification problems, being able to deal with set-valued input and output attributes, as well as incorporating Melvil 's ontologies for the respective application domain. Thus, we selected a nearest-neighbor approach for the prototype implementation. An evaluation on off-line data collected from several users showed a satisfactory performance.

Signature-based structures for objects with set-valued attributes

Aiming at the efficient retrieval of objects with set-valued attributes, we introduce three variations of a new method in order to satisfy subset and superset queries. Our approach is to combine the advantages of two access methods, that of linear Hashing and of tree-shaped methods, on which other similar methods have been previously reported as well. Performance estimation analytical functions for each particular method are presented, followed by a thorough experimental comparison of all investigated structures, where analytical and experimental results deviate 10% on the average. Finally, the results of this performance evaluation are presented and discussed, clearly showing the superiority of the new methods reaching an improvement of up to 85%.

Induction of decision trees in numeric domains using set-valued attributes

Conventional algorithms for decision tree induction use an attribute-val ue representatio n scheme for instances. This paper explores the empirical consequences of using set-valued attributes. This simple representational extension, when used as a pre-processor for numeric data, is shown to yield significant gains in accuracy combined with attractive build times. It is also shown to improve the accuracy for the second best classification option, which has valuable ramifications for post-processing. To do so an intuitive and practical version of pre-pruning is employed. Moreover, the implementation of a simple pruning scheme serves as an example of pruning applicability over the resulted trees and also as an indication that the proposed discretization absorbs much of pruning potential. Finally, we construct several versions of the basic algorithm to examine the value of every component that comprises it.

A cost model for sort-domain traversal strategy in object-oriented databases

Some factors such as set-valued attributes, partial instantiation and clustering need to be considered to estimate the processing cost of an object-oriented query. In this paper, we propose a cost model which reflects the effect of such factors. The cost of a query is estimated using the number of objects participated in the instantiated paths and the number of tuples and the size of the reference relation which is an intermediate result of class traversais. The proposed cost model utilizes a small set of statistics maintained by the system, and estimates the costs more accurately.

Generalized union and project operations for pooling uncertain and imprecise information

We have previously proposed an extended relational data model with the objective of supporting uncertain information in a consistent and coherent manner. The model, which can represent both uncertainty and imprecision in data, is based on the Dempster-Shafer (D-S) theory of evidence, and it uses bel and pls functions of the theory, with their definitions extended somewhat for this purpose. Relational operations such as Select, Cartesian Product, Join, Project, Intersect, and Union have previously been defined [21]. In this paper we consider two data combination problems associated with the data model. These problems are believed to be inherent in most database models which handle uncertain information. The problems are: the potential existence in the database of identical tuples which have different respective degrees of belief (the redundancy problem), and the potential existence of different tuples with the same key values (the inconsistency problem). The redundancy problem was treated to some extent in an earlier paper, but the inconsistency problem has not been considered at all yet. Now the well-known orthogonal sum operation in the D-S theory, which performs the pooling of data for the purpose of making choices between hypotheses, may be viewed as a means of reducing inconsistency in data arising from different sources. This capability has not yet been exploited in our data model. So the idea here is to define a new combine operation as a primitive for handling inconsistency in relations. When data from a number of sources is being pooled — often in order to support decision making — the Union operation, and the Project operation, are very important. We are particularly interested in the case where tuples in operand relations match attribute-wise, but have different uncertainty and imprecision characteristics. The execution of both the Union and Project operations, which the new combine operation can help solve, is a means of dealing with the problem of information aggregation. We use the orthogonal sum, which generalizes results from traditional probability theory in a natural and correct manner, for pooling evidence during the combine computation. The paper also addresses the execution efficiency of our suggested approach. The orthogonal sum operation is exponentially complex if implemented naively. A linear time algorithm can readily be made available for Union and Project for the simple case where the attribute values to be combined are singletons (i.e., atomic values — as in the conventional relational model). However, many potential applications of the approach can exploit the new data model's facility of supporting set-valued attributes. In the method presented here we can combine data supporting non-singleton subsets in linear-time.

Fast algorithms for universal quantification in large databases

Universal quantification is not supported directly in most database systems despite the fact that it adds significant power to a system's query processing and inference capabilities, in particular for the analysis of many-to-many relationships and of set-valued attributes. One of the main reasons for this omission has been that universal quantification algorithms and their performance have not been explored for large databases. In this article, we describe and compare three known algorithms and one recently proposed algorithm for relational division, the algebra operator that embodies universal quantification. For each algorithm, we investigate the performance effects of explicit duplicate removal and referential integrity enforcement, variants for inputs larger than memory, and parallel execution strategies. Analytical and experimental performance comparisons illustrate the substantial differences among the algorithms. Moreover, comparisons demonstrate that the recently proposed division algorithm evaluates a universal quantification predicate over two relations as fast as hash (semi-) join evaluates an existential quantification predicate over the same relations. Thus, existential and universal quantification can be supported with equal efficiency by adding the recently proposed algorithm to a query evaluation system. A second result of our study is that universal quantification should be expressed directly in a database query language, because most query optimizers do not recognize the rather indirect formulations available in SQL as relational division and therefore produce very poor evaluation plans for many universal quantification queries.

Optimal block size for set-valued attributes

In a relational database management system, allowing tuples to be of variable length can complicate storage management considerably. As an example, consider an employee relation containing the employee’s name, social security number, and salary. We could store the names of the employee’s dependents in a separate relation, which would require an expensive join to access. As an alternative, we could add a set-valued attribute, called dependents, to the employee relation, with each tuple containing zero, one, or more values for that attribute. In non-first normal form relational databases, set-valued attributes are commonplace [3, 10, 11, 12, 13]. Variable-length tuples also arise in temporal databases, where sets of time intervals are associated with attributes or with tuples [1, 2, 5, 6, 8, 9]. One approach to representing such tuples on disk is to store the set-valued attribute values separately from the rest of the (now fixed-length) tuple. The sets can be stored as a linked list of fixed-size blocks, each storing one or several attribute values. If the sets exhibit a wide range of cardinalities, a linked list is an appropriate storage structure. The alternative of using a hash table to store the set of attribute values was rejected because such a table is also of variable size, complicating page space management. The question we address is, What is the optimal number of attributes in a block? If one attribute per block is employed, we may waste space on pointers. On the other hand, if large blocks are used, the last block of each list may be largely unfilled. The objective of this paper is to obtain an expression identifying the optimal block size, to achieve the best space utilization. In the next section, an analytical model is presented which can help determine the optimal block size. An example from temporal databases that exploits this model is described in Section 3. Section 4 provides a summary.

Improved reliability predictions for commerical computers : N. Keith Hergatt. Proc. A. Reliab. Maintainab. Symp., 357 (1991)

We have previously proposed an extended relational data model with the objective of supporting uncertain information in a consistent and coherent manner. The model, which can represent both uncertainty and imprecision in data, is based on the Dempster-Shafer (D-S) theory of evidence, and it uses bel and pls functions of the theory, with their definitions extended somewhat for this purpose. Relational operations such as Select, Cartesian Product, Join, Project, Intersect, and Union have previously been defined [21].In this paper we consider two data combination problems associated with the data model. These problems are believed to be inherent in most database models which handle uncertain information. The problems are: the potential existence in the database of identical tuples which have different respective degrees of belief (the redundancy problem), and the potential existence of different tuples with the same key values (the inconsistency problem). The redundancy problem was treated to some extent in an earlier paper, but the inconsistency problem has not been considered at all yet.Now the well-known orthogonal sum operation in the D-S theory, which performs the pooling of data for the purpose of making choices between hypotheses, may be viewed as a means of reducing inconsistency in data arising from different sources. This capability has not yet been exploited in our data model. So the idea here is to define a new combine operation as a primitive for handling inconsistency in relations.When data from a number of sources is being pooled — often in order to support decision making — the Union operation, and the Project operation, are very important. We are particularly interested in the case where tuples in operand relations match attribute-wise, but have different uncertainty and imprecision characteristics. The execution of both the Union and Project operations, which the new combine operation can help solve, is a means of dealing with the problem of information aggregation. We use the orthogonal sum, which generalizes results from traditional probability theory in a natural and correct manner, for pooling evidence during the combine computation.The paper also addresses the execution efficiency of our suggested approach. The orthogonal sum operation is exponentially complex if implemented naively. A linear time algorithm can readily be made available for Union and Project for the simple case where the attribute values to be combined are singletons (i.e., atomic values — as in the conventional relational model). However, many potential applications of the approach can exploit the new data model's facility of supporting set-valued attributes. In the method presented here we can combine data supporting non-singleton subsets in linear-time.