Monotonic Aggregation Research Articles

A new class of composite indicators: The penalized power mean

In this paper we propose a new aggregation method for constructing composite indicators based on a penalization of the power mean. The idea underlying this approach consists in multiplying the power mean by a factor that accounts for the horizontal heterogeneity among indicators while penalizing units with a larger heterogeneity. In line with the minimum loss of information principle, the penalization factor proposed is proven to be linked to the loss of information generated when the indicators are substituted with their power means. As a consequence, the aggregation approach gives rise to the class of penalized power means and the penalized Benefit of the Doubt aggregative approach. Including heterogeneity makes the aggregation approach more suitable for refined rankings. Interestingly, the penalized power mean of order one coincides with the Mazziotta Pareto Index. Some theoretical properties of the penalized power means are proven, thus supporting the Mazziotta Pareto index. An empirical analysis of the Human Development Index in 2019 is presented. Comparisons of the rankings induced by the penalized and non-penalized Benefit of the Doubt and power mean aggregation approaches are shown. There are three main findings: the penalized power means satisfy the properties characterizing weakly monotone aggregation functions; the penalization reduces ranking variations while differentiating units with close means; and the geometric mean provides composite indicators whose ranking is closest to those obtained with power means of different order.

Cache-Efficient Top-k Aggregation over High Cardinality Large Datasets

Top-k aggregation queries are widely used in data analytics for summarizing and identifying important groups from large amounts of data. These queries are usually processed by first computing exact aggregates for all groups and then selecting the groups with the top-k aggregate values. However, such an approach can be inefficient for high-cardinality large datasets where intermediate results may not fit within the local cache of multi-core processors leading to excessive data movement. To address this problem, we have developed Zippy, a new cache-conscious aggregation framework that leverages the skew in the data distribution to minimize data movements. This is achieved by designing cache-resident data structures and an adaptive multi-pass algorithm that quickly identifies candidate groups during processing, and performs exact aggregations for these groups. The non-candidate groups are pruned cheaply using efficient hashing and partitioning techniques without performing exact aggregations. We develop techniques to improve robustness over adversarial data distributions and have optimized the framework to reuse computations incrementally for rolling (or paginated) top-k aggregate queries. Our extensive evaluation using both real-world and synthetic datasets demonstrate that Zippy can achieve a median speed-up of more than 3× for monotonic aggregation functions across typical ranges of k values (e.g., 1 to 100) and 1.4× for non-monotonic functions when compared with state-of-the-art cache-conscious aggregation techniques.

A Differential Datalog Interpreter

The core reasoning task for datalog engines is materialization, the evaluation of a datalog program over a database alongside its physical incorporation into the database itself. The de-facto method of computing is through the recursive application of inference rules. Due to it being a costly operation, it is a must for datalog engines to provide incremental materialization; that is, to adjust the computation to new data instead of restarting from scratch. One of the major caveats is that deleting data is notoriously more involved than adding since one has to take into account all possible data that has been entailed from what is being deleted. Differential dataflow is a computational model that provides efficient incremental maintenance, notoriously with equal performance between additions and deletions, and work distribution of iterative dataflows. In this paper, we investigate the performance of materialization with three reference datalog implementations, out of which one is built on top of a lightweight relational engine, and the two others are differential-dataflow and non-differential versions of the same rewrite algorithm with the same optimizations. Experimental results suggest that monotonic aggregation is more powerful than ascenting merely the powerset lattice.

Aggregate Semantics for Propositional Answer Set Programs

AbstractAnswer set programming (ASP) emerged in the late 1990s as a paradigm for knowledge representation and reasoning. The attractiveness of ASP builds on an expressive high-level modeling language along with the availability of powerful off-the-shelf solving systems. While the utility of incorporating aggregate expressions in the modeling language has been realized almost simultaneously with the inception of the first ASP solving systems, a general semantics of aggregates and its efficient implementation have been long-standing challenges. Aggregates have been proposed and widely used in database systems, and also in the deductive database language Datalog, which is one of the main precursors of ASP. The use of aggregates was, however, still restricted in Datalog (by either disallowing recursion or only allowing monotone aggregates), while several ways to integrate unrestricted aggregates evolved in the context of ASP. In this survey, we pick up at this point of development by presenting and comparing the main aggregate semantics that have been proposed for propositional ASP programs. We highlight crucial properties such as computational complexity and expressive power, and outline the capabilities and limitations of different approaches by illustrative examples.

Ordered Completion for Logic Programs with Aggregates

In this paper, we show that first-order logic programs with monotone aggregates under the stable model semantics can be captured in classical first-order logic. More precisely, we extend the notion of ordered completion for logic programs with a large variety of aggregates so that every stable model of a program with aggregates corresponds to a classical model of its enhanced ordered completion, and vice versa.

Aggregation of indistinguishability operators

The paper studies the aggregation of pairs of T-indistinguishability operators. More concretely we address the question whether A(E,E′) is a T-indistinguishability operator if E,E′ are T-indistinguishability operators. The answer depends on the aggregation function, the t-norm T, and the chosen T-indistinguishability operators. It is well-known that an aggregation function preserves T-transitive relations if and only if it dominates the t-norm T. We show the important role of the minimum t-norm TM in this preservation problem. In particular we develop weaker forms of domination that are used to provide characterizations of TM-indistinguishability preservation under aggregation. We also prove that the existence of a single strictly monotone aggregation that satisfies the indistinguishability operator preservation property guarantees all aggregations to have the same preservation property.

Aggregation of fuzzy subgroups

Given an aggregation function, and two fuzzy subgroups of a group, this study investigates whether the aggregation of them is also a fuzzy subgroup. It is proven that if the cardinal of the group is a prime power, then it is always a fuzzy subgroup. For a general group, it is proven that the existence of a binary relation between the fuzzy subgroups determines if their aggregation is a fuzzy subgroup. Moreover, given two fuzzy subgroups, if the aggregation of them is a fuzzy subgroup for some strictly monotone aggregation function, then it is a fuzzy subgroup for any aggregation function. The present study also examines conditions on the aggregation function that characterise the case where the aggregation of two arbitrary fuzzy subgroups is also a fuzzy subgroup.

Efficient read monotonic data aggregation across shards on the cloud

Client-centric consistency models define the view of the data storage expected by a client in relation to the operations done by a client within a session. Monotonic reads is a client-centric consistency model which ensures that if a process has seen a particular value for the object, any subsequent accesses will never return any previous values. Monotonic reads are used in several applications like news feeds and social networks to ensure that the user always has a forward moving view of the data. The idea of Monotonic reads over multiple copies of the data and for lightly loaded systems is intuitive and easy to implement. For example, ensuring that a client session always fetches data from the same server automatically ensures that the user will never view old data. However, such a simplistic setup will not work for large deployments on the cloud, where the data is sharded across multiple high availability setups and there are several million clients accessing data at the same time. In such a setup it becomes necessary to ensure that the data fetched from multiple shards are logically consistent with each other. The use of trivial implementations, like sticky sessions, causes severe performance degradation during peak loads. This paper explores the challenges surrounding consistent monotonic reads over a sharded setup on the cloud and proposes an efficient architecture for the same. Performance of the proposed architecture is measured by implementing it on a cloud setup and measuring the response times for different shard counts. We show that the proposed solution scales with almost no change in performance as the number of shards increases.

Linguistic OWA and two time-windows based fault identification in wide plants

Fault detection and diagnosis in industrial processes are challenging tasks that demand effective and timely decision making procedures. The multivariate statistical approaches for fault detection based on data have been very useful. However, they are known to be less powerful for fault diagnosis because they normally require prior knowledge of the problem involved. In this context, this proposal is based on an on-line, distributed fault isolation approach to provide a scored rank of variables considered as responsible for the faults in a more robust and earlier way than usual approaches. The fault isolation is carried out considering some top Fault Isolation (FI) methods, without prior knowledge regarding faults, in a distributed and collaborative way by a linguistic based decision making. The isolation of faulty variables provided by each FI approach is aggregated to provide a fault identification based on a scored ranking at two time points: after the fault detection and when the plant has recovered. In both cases, the final fault isolation is provided as a scored ranking obtained by Ordered Weighted Average operators (OWA) and Regular Increasing Monotone (RIM) aggregation functions, which permit the implementation of linguistic aggregation functions. The risk aversion during this multicriteria isolation is tuned by the user and can provide several strategies or policies. The fault isolation at two key times searches for the origin of faults and evaluates the evolution of the system after the fault’s occurrence in the new working position of the plant. This is because faults in an industrial plant are propagated to different variables due to the actions of the process controllers. This method has been applied to two complex benchmark plants obtaining an earlier and more robust isolation.

Rewriting recursive aggregates in answer set programming: back to monotonicity

AbstractAggregation functions are widely used in answer set programming for representing and reasoning on knowledge involving sets of objects collectively. Current implementations simplify the structure of programs in order to optimize the overall performance. In particular, aggregates are rewritten into simpler forms known as monotone aggregates. Since the evaluation of normal programs with monotone aggregates is in general on a lower complexity level than the evaluation of normal programs with arbitrary aggregates, any faithful translation function must introduce disjunction in rule heads in some cases. However, no function of this kind is known. The paper closes this gap by introducing a polynomial, faithful, and modular translation for rewriting common aggregation functions into the simpler form accepted by current solvers. A prototype system allows for experimenting with arbitrary recursive aggregates, which are also supported in the recent version 4.5 of the groundergringo, using the methods presented in this paper.

Weakly Monotonic Averaging Functions

Monotonicity with respect to all arguments is fundamental to the definition of aggregation functions. It is also a limiting property that results in many important nonmonotonic averaging functions being excluded from the theoretical framework. This work proposes a definition for weakly monotonic averaging functions, studies some properties of this class of functions, and proves that several families of important nonmonotonic means are actually weakly monotonic averaging functions. Specifically, we provide sufficient conditions for weak monotonicity of the Lehmer mean and generalized mixture operators. We establish weak monotonicity of several robust estimators of location and conditions for weak monotonicity of a large class of penalty-based aggregation functions. These results permit a proof of the weak monotonicity of the class of spatial-tonal filters that include important members such as the bilateral filter and anisotropic diffusion. Our concept of weak monotonicity provides a sound theoretical and practical basis by which monotonic aggregation functions and nonmonotonic averaging functions can be related within the same framework, allowing us to bridge the gap between these previously disparate areas of research.

Perceived Ambiguity and Relevant Measures

We axiomatize preferences that can be represented by a monotonic aggregation of subjective expected utilities generated by a utility function and some set of i.i.d. probability measures over a product state space, S∞. For such preferences, we define relevant measures, show that they are treated as if they were the only marginals possibly governing the state space, and connect them with the measures appearing in the aforementioned representation. These results allow us to interpret relevant measures as reflecting part of perceived ambiguity, meaning subjective uncertainty about probabilities over states. Under mild conditions, we show that increases or decreases in ambiguity aversion cannot affect the relevant measures. This property, necessary for the conclusion that these measures reflect only perceived ambiguity, distinguishes the set of relevant measures from the leading alternative in the literature. We apply our findings to a number of well-known models of ambiguity-sensitive preferences. For each model, we identify the set of relevant measures and the implications of comparative ambiguity aversion.

Graph queries in a next-generation Datalog system

Recent theoretical advances have enabled the use of special monotonic aggregates in recursion. These special aggregates make possible the concise expression and efficient implementation of a rich new set of advanced applications. Among these applications, graph queries are particularly important because of their pervasiveness in data intensive application areas. In this demonstration, we present our Deductive Application Language (DeAL) System, the first of a new generation of Deductive Database Systems that support applications that could not be expressed using regular stratification, or could be expressed using XY-stratification (also supported in DeAL) but suffer from inefficient execution. Using example queries, we will (i) show how complex graph queries can be concisely expressed using DeAL and (ii) illustrate the formal semantics and efficient implementation of these powerful new monotonic constructs.

A declarative extension of horn clauses, and its significance for datalog and its applications

AbstractFS-rules provide a powerful monotonic extension for Horn clauses that supports monotonic aggregates in recursion by reasoning on the multiplicity of occurrences satisfying existential goals. The least fixpoint semantics, and its equivalent least model semantics, hold for logic programs with FS-rules; moreover, generalized notions of stratification and stable models are easily derived when negated goals are allowed. Finally, the generalization of techniques such as seminaive fixpoint and magic sets, make possible the efficient implementation of DatalogFS, i.e., Datalog with rules with Frequency Support (FS-rules) and stratified negation. A large number of applications that could not be supported efficiently, or could not be expressed at all in stratified Datalog can now be easily expressed and efficiently supported in DatalogFS and a powerful DatalogFS system is now being developed at UCLA.

Continuous Distributed Top-k Monitoring over High-Speed Rail Data Stream in Cloud Computing Environment

In the environment of cloud computing, real-time mass data about high-speed rail which is based on the intense monitoring of large scale perceived equipment provides strong support for the safety and maintenance of high-speed rail. In this paper, we focus on the Top- k algorithm of continuous distribution based on Multisource distributed data stream for high-speed rail monitoring. Specifically, we formalized Top- k monitoring model of high-speed rail and proposed DTMR that is the Top- k monitoring algorithm with random, continuous, or strictly monotone aggregation functions. The DTMR was proved to be valid by lots of experiments.

Distributed threshold querying of general functions by a difference of monotonic representation

The goal of athreshold queryis to detect all objects whose score exceeds a given threshold. This type of query is used in many settings, such as data mining, event triggering, and top-kselection. Often, threshold queries are performed overdistributed data. Given database relations that are distributed over many nodes, an object's score is computed by aggregating the value of each attribute, applying a given scoring function over the aggregation, and thresholding the function's value. However, joining all the distributed relations to a central database might incur prohibitive overheads in bandwidth, CPU, and storage accesses. Efficient algorithms required to reduce these costs exist only for monotonic aggregation threshold queries and certain specific scoring functions.We present a novel approach for efficiently performing general distributed threshold queries. To the best of our knowledge, this is the first solution to the problem of performing such queries with general scoring functions. We first present a solution for monotonic functions, and then introduce a technique to solve for other functions by representing them as a difference of monotonic functions. Experiments with real-world data demonstrate the method's effectiveness in achieving low communication and access costs.

Maskin monotonic aggregation rules and partial anonymity

In the case of two alternatives, we characterize the class of aggregation rules that satisfy Maskin monotonicity, neutrality, and partial anonymity.

Extension of Arrow's theorem to symmetric sets of tournaments

Arrow's impossibility theorem [K.J. Arrow, Social Choice and Individual Values, Wiley, New York, NY, 1951] shows that the set of acyclic tournaments is not closed to non-dictatorial Boolean aggregation. In this paper we extend the notion of aggregation to general tournaments and we show that for tournaments with four vertices or more any proper symmetric (closed to vertex permutations) subset cannot be closed to non-dictatorial monotone aggregation and to non-neutral aggregation. We also demonstrate a proper subset of tournaments that is closed to parity aggregation for an arbitrarily large number of vertices. This proves a conjecture of Kalai [Social choice without rationality, Reviewed NAJ Economics 3(4)] for the non-neutral and the non-dictatorial and monotone cases and gives a counter example for the general case.

Maskin monotonic aggregation rules

Given a society confronting two alternatives, we show that the set of anonymous, neutral and Maskin monotonic aggregation rules coincides with the family of absolute qualified majority rules. We also explore the effect of incorporating Pareto optimality in our characterization.

Optimal aggregation algorithms for middleware

Assume that each object in a database has m grades, or scores, one for each of m attributes. For example, an object can have a color grade, that tells how red it is, and a shape grade, that tells how round it is. For each attribute, there is a sorted list, which lists each object and its grade under that attribute, sorted by grade (highest grade first). Each object is assigned an overall grade, that is obtained by combining the attribute grades using a fixed monotone aggregation function, or combining rule, such as min or average. To determine the top k objects, that is, k objects with the highest overall grades, the naive algorithm must access every object in the database, to find its grade under each attribute. Fagin has given an algorithm (“Fagin's Algorithm”, or FA) that is much more efficient. For some monotone aggregation functions, FA is optimal with high probability in the worst case. We analyze an elegant and remarkably simple algorithm (“the threshold algorithm”, or TA) that is optimal in a much stronger sense than FA. We show that TA is essentially optimal, not just for some monotone aggregation functions, but for all of them, and not just in a high-probability worst-case sense, but over every database. Unlike FA, which requires large buffers (whose size may grow unboundedly as the database size grows), TA requires only a small, constant-size buffer. TA allows early stopping, which yields, in a precise sense, an approximate version of the top k answers. We distinguish two types of access: sorted access (where the middleware system obtains the grade of an object in some sorted list by proceeding through the list sequentially from the top), and random access (where the middleware system requests the grade of object in a list, and obtains it in one step). We consider the scenarios where random access is either impossible, or expensive relative to sorted access, and provide algorithms that are essentially optimal for these cases as well.