Minimal Functional Dependencies Research Articles

Composing Biases by Using CP to Decompose Minimal Functional Dependencies for Acquiring Complex Formulae

Given a table with a minimal set of input columns that functionally determines an output column, we introduce a method that tries to gradually decompose the corresponding minimal functional dependency (mfd) to acquire a formula expressing the output column in terms of the input columns. A first key element of the method is to create sub-problems that are easier to solve than the original formula acquisition problem, either because it learns formulae with fewer inputs parameters, or as it focuses on formulae of a particular class, such as Boolean formulae; as a result, the acquired formulae can mix different learning biases such as polynomials, conditionals or Boolean expressions. A second key feature of the method is that it can be applied recursively to find formulae that combine polynomial, conditional or Boolean sub-terms in a nested manner. The method was tested on data for eight families of combinatorial objects; new conjectures were found that were previously unattainable. The method often creates conjectures that combine several formulae into one with a limited number of automatically found Boolean terms.

Discovering Functional Dependencies through Hitting Set Enumeration

Functional dependencies (FDs) are among the most important integrity constraints in databases. They serve to normalize datasets and thus resolve redundancies, they contribute to query optimization, and they are frequently used to guide data cleaning efforts. Because the FDs of a particular dataset are usually unknown, automatic profiling algorithms are needed to discover them. These algorithms have made considerable advances in the past few years, but they still require a significant amount of time and memory to process datasets of practically relevant sizes. We present FDHits, a novel FD discovery algorithm that finds all valid, minimal FDs in a given relational dataset. FDHits is based on several discovery optimizations that include a hybrid validation approach, effective hitting set enumeration techniques, one-pass candidate validations, and parallelization. Our experiments show that FDHits, even without parallel execution, has a median speedup of 8.1 compared to state-of-the-art FD discovery algorithms while using significantly less memory. This allows the discovery of all FDs even on datasets that could not be processed by the current state-of-the-art.

FDTool: a Python application to mine for functional dependencies and candidate keys in tabular data.

Functional dependencies (FDs) and candidate keys are essential for table decomposition,database normalization, and data cleansing. In this paper, we present FDTool, a commandline Python application to discover minimal FDs in tabular datasets and infer equivalent attributesets and candidate keys from them. The runtime and memory costs associated withseven published FD discovery algorithms are given with an overview of their theoretical foundations.We conclude that FD_Mine is the most efficient FD discovery algorithm when appliedto datasets with many rows (> 100,000 rows) and few columns (< 14 columns). This putsit in a special position to rule mine clinical and demographic datasets, which often consistof long and narrow sets of participant records. The structure of FD Mine is described andsupplemented with a formal proof of the equivalence pruning method used. FDTool is are-implementation of FD Mine with additional features added to improve performance andautomate typical processes in database architecture. The experimental results of applyingFDTool to 12 datasets of different dimensions are summarized in terms of the number ofFDs checked, the number of FDs found, and the time it takes for the code to terminate. Wefind that the number of attributes in a dataset has a much greater effect on the runtime andmemory costs of FDTool than does row count. The last section explains in detail how theFDTool application can be accessed, executed, and further developed.

FastMFDs: a fast, efficient algorithm for mining minimal functional dependencies from large-scale distributed data with Spark

Minimal functional dependency is an important relationship in the relational database. It can describe some special relationships between complex and irregular attributes in the relational database. Extracting minimal functional dependencies (MFDs) from relational databases is an important database analysis technique. However, as the data grows larger and larger in size, even the most efficient stand-alone algorithms are exponential in the number of attributes of the relations. Discovering MFDs on a single computer is hard and slow, and it can only be applied to small centralized datasets. It is challenging to discover MFDs from big data, especially large-scale distributed data. Apache Spark is a unified analytics engine for big data processing; we present a new algorithm FastMFDs based on Spark for discovering all MFDs from large-scale distributed data in parallel. FastMFDs uses both the RDD framework and the DataFrame framework to store and process distributed data. FastMFDs deletes equivalent attributes. FastMFDs also provides two-way search algorithm for searching and pruning. We experimented our algorithm on real-life datasets, and our algorithm is more efficient and faster than the existing discovering methods.

Modeling Automatic Generator of Optimal and Minimal Covers of Functional Dependency

Functional Dependency (FD) rules arise from applications semantic and enforcing integrity of minimal cover of FDs by normalization or triggers is sufficient condition for guaranteeing integrity of database. Formal achievement of minimal and optimal FDs from initial FDs is accomplished in this paper. Process of computing optimal and minimal FDs is presented and modeled by Colored Petri Net (CPN). Proposed model iteratively applies a subset of Armstrong’s axioms and infers FDs that are origin of all other FDs. Execution of model automatically is stopped when markings of model’s places do not change in iteration. Minimal and optimal FDs are automatically computed and stored in two different places of the model. Although CPN is used in many areas but modeling automatic generation of optimal and minimal cover set using CPN is a novel work. Proposed CPN model in this paper can be considered as automatic proof generator too. Model is designed such that state space graph of model is very small and is generated quickly. Colour sets are declared such that starting from initial markings, step by step proof of inferring optimal and minimal cover FDs can be extracted automatically. Proposed model can be used as a simple tool for automatically computing optimal and minimal FDs of a set of initial FDs.

Discovery of Functional and Approximate Functional Dependencies in Relational Databases

This study develops the foundation for a simple, yet efficient method for uncovering functional and approximate functional dependencies in relational databases. The technique is based upon the mathematical theory of partitions defined over a relation's row identifiers. Using a levelwise algorithm the minimal non-trivial functional dependencies can be found using computations conducted on integers. Therefore, the required operations on partitions are both simple and fast. Additionally, the row identifiers provide the added advantage of nominally identifying the exceptions to approximate functional dependencies, which can be used effectively in practical data mining applications.

Inferring Minimal Functional Dependencies in Horn and q-Horn Theories

In this paper, we study various problems related to the inference of minimal functional dependencies in Horn and q-Horn theories. We show that if a Horn theory is represented by a Horn CNF, then there exists an incrementally polynomial algorithm for inferring all minimal functional dependencies. On the other hand, if a Horn theory is represented as the Horn envelope of a given set of models, then there exists a polynomial total time algorithm for this inference problem if and only if there exists such an algorithm for dualizing a positive CNF. Finally, we generalize our results to the case of q-Horn theories, and show that all the considered problems can be reduced in polynomial time to the corresponding problems for Horn theories.

Discovery of functional and approximate functional dependencies in relational databases

This study develops the foundation for a simple, yet efficient method for uncovering functional and approximate functional dependencies in relational databases. The technique is based upon the mathematical theory of partitions defined over a relation's row identifiers. Using a levelwise algorithm the minimal non-trivial functional dependencies can be found using computations conducted on integers. Therefore, the required operations on partitions are both simple and fast. Additionally, the row identifiers provide the added advantage of nominally identifying the exceptions to approximate functional dependencies, which can be used effectively in practical data mining applications.

On functional dependencies in q-Horn theories

This paper studies functional dependencies in q-Horn theories, and discusses their use in knowledge condensation. We introduce compact model-based representations of q-Horn theories, analyze the structure of functional dependencies in q-Horn theories, and show that every minimal functional dependency in a q-Horn theory Σ can be expressed either by a single term or by a single clause. We also prove that the set of all minimal functional dependencies in Σ is quasi-acyclic. We then develop polynomial time algorithms for recognizing whether a given functional dependency holds in a q-Horn theory, which is represented either by a q-Horn CNF or as the q-Horn envelope of a set of models. Finally, we show that every q-Horn theory has a unique condensation, and can be condensed in polynomial time.

Functional and embedded dependency inference: a data mining point of view

The issue of discovering functional dependencies from populated databases has received a great deal of attention because it is a key concern in database analysis. Such a capability is strongly required in database administration and design while being of great interest in other application fields such as query folding. Investigated for long years, the issue has been recently addressed in a novel and more efficient way by applying principles of data mining algorithms. The two algorithms fitting in such a trend are T ANE and Dep-Miner. They strongly improve previous proposals. In this paper, we propose a new approach adopting a data mining point of view. We define a novel characterization of minimal functional dependencies. This formal framework is sound and simpler than related work. We introduce the new concept of free set for capturing source of functional dependencies. By using the concepts of closure and quasi-closure of attribute sets, targets of such dependencies are characterized. Our approach is enforced through the algorithm F UN which is particularly efficient since it is comparable or improves the two best operational solutions (according to our knowledge): T ANE and Dep-Miner. It makes use of various optimization techniques and it can work on very large databases. Applying on real life or synthetic data more or less correlated, comparative experiments are performed in order to assess performance of F UN against T ANE and Dep-Miner. Moreover, our approach also exhibits (without significant additional execution time) embedded functional dependencies, i.e. dependencies captured in any subset of the attribute set originally considered. Embedded dependencies capture a knowledge specially relevant in all fields where materialized data sets are managed (e.g. materialized views widely used in data warehouses).

Functional dependencies in Horn theories

This paper studies functional dependencies in Horn theories, both when the theory is represented by its clausal form and when it is defined as the Horn envelope of a set of models. We provide polynomial algorithms for the recognition of whether a given functional dependency holds in a given Horn theory, as well as polynomial algorithms for the generation of some representative sets of functional dependencies. We show that some problems of inferring functional dependencies (e.g., constructing an irredundant FD-cover) are computationally difficult. We also study the structure of functional dependencies that hold in a Horn theory, showing that every such functional dependency is in fact a single positive term Boolean function, and prove that for any Horn theory the set of its minimal functional dependencies is quasi-acyclic. Finally, we consider the problem of condensing a Horn theory, prove that any Horn theory has a unique condensation, and develop an efficient polynomial algorithm for condensing Horn theories.