Content-based Publish/subscribe Systems Research Articles

Universal Cross-Cloud Communication

Integration of applications, data-centers, and programming abstractions in the cloud-of-clouds poses many challenges to system engineers. Different cloud providers offer different communication abstractions, and applications exhibit different communication patterns. By abstracting from hardware addresses and lower-level communication, the publish/subscribe paradigm seems like an adequate abstraction for supporting communication across clouds, as it supports many-to-many communication between publishers and subscribers, of which one-to-one or one-to-many can be viewed as special cases. In particular, content-based publish/subscribe (CPS) systems provide an expressive abstraction that matches well with the key-value pair model of many established cloud storage and computing systems, and decentralized overlay-based CPS implementations scale up well. However, CPS systems perform poorly at small scale, e.g., one-to-one or one-to-many communication. This holds especially for multi-send scenarios which we refer to as entourages that may range from a channel between a publisher and a single subscriber to a broadcast between a publisher and a handful of subscribers. These scenarios are common in cloud computing, where cheap hardware is exploited for parallelism (efficiency) and redundancy (fault-tolerance). With CPS, multi-send messages go over several hops before their destinations are even identified via predicate matching, resulting in increased latency, especially when destinations are located in different data-centers or zones. Topic-based publish/subscribe (TPS) systems support communication at small scale more efficiently, but still route messages over multiple hops and inversely lack the flexibility of CPS systems. In this paper, we propose CPS protocols for cloud-of-clouds communication that can dynamically identify entourages of publishers and corresponding subscribers. Our CPS protocols dynamically connect the publishers with their entourages through uberlays . These uberlays can transmit messages from a publisher to its corresponding subscribers with low latency. Our experiments show that our protocols make CPS abstraction viable and beneficial for many applications. We introduce a CPS system named Atmosphere that leverages out CPS protocols and illustrate how Atmosphere has allowed us to implement, with little effort, versions of the popular HDFS and ZooKeeper systems which operate efficiently across data-centers.

Delta

In content-based publish-subscribe (pub/sub) systems, users express their interests as queries over a stream of publications. Scaling up content-based pub/sub to very large numbers of subscriptions is challenging: users are interested in low latency , that is, getting subscription results fast, while the pub/sub system provider is mostly interested in scaling , i.e., being able to serve large numbers of subscribers, with low computational resources utilization. We present a novel approach for scalable content-based pub/sub in the presence of constraints on the available CPU and network resources, implemented within our pub/sub system Delta. We achieve scalability by off-loading some subscriptions from the pub/sub server, and leveraging view-based query rewriting to feed these subscriptions from the data accumulated in others. Our main contribution is a novel algorithm for organizing views in a multi-level dissemination network, exploiting view-based rewriting and powerful linear programming capabilities to scale to many views, respect capacity constraints, and minimize latency. The efficiency and effectiveness of our algorithm are confirmed through extensive experiments and a large deployment in a WAN.

Parametric Content-Based Publish/Subscribe

Content-based publish/subscribe (CPS) is an appealing abstraction for building scalable distributed systems, e.g., message boards, intrusion detectors, or algorithmic stock trading platforms. Recently, CPS extensions have been proposed for location-based services like vehicular networks, mobile social networking, and so on. Although current CPS middleware systems are dynamic in the way they support the joining and leaving of publishers and subscribers, they fall short in supporting subscription adaptations. These are becoming increasingly important across many CPS applications. In algorithmic high frequency trading, for instance, stock price thresholds that are of interest to a trader change rapidly, and gains directly hinge on the reaction time to relevant fluctuations rather than fixed values. In location-aware applications, a subscription is a function of the subscriber location (e.g. GPS coordinates), which inherently changes during motion. The common solution for adapting a subscription consists of a resubscription, where a new subscription is issued and the superseded one canceled. This incurs substantial overhead in CPS middleware systems, and leads to missed or duplicated events during the transition. In this article, we explore the concept of parametric subscriptions for capturing subscription adaptations. We discuss desirable and feasible guarantees for corresponding support, and propose novel algorithms for updating routing mechanisms effectively and efficiently in classic decentralized CPS broker overlay networks. Compared to resubscriptions, our algorithms significantly improve the reaction time to subscription updates without hampering throughput or latency under high update rates. We also propose and evaluate approximation techniques to detect and mitigate pathological cases of high frequency subscription oscillations, which could significantly decrease the throughput of CPS systems thereby affecting other subscribers. We analyze the benefits of our support through implementations of our algorithms in two CPS systems, and by evaluating our algorithms on two different application scenarios.

Routing algorithms for content-based publish/subscribe systems

In content-based publish/subscribe systems, messages target a dynamic group of participants whose expressed interests match the contents of the messages. In this generalization of multicasting communication, also dubbed content-based networking, naming, binding and communication are intertwined in the same substrate. Optimal content-based routing uses dissemination trees dynamically pruned to only cover the matching subscribers. It is a complex problem that has motivated significant research efforts. This paper presents a compilation of the main algorithms for routing messages in distributed content-based publish-subscribe systems proposed and published in the last decade. Discussion is focused on the content-based routing problem in respect to optimality, complexity and applicability. Moreover, whenever it is appropriate, the algorithms covered are also matched to similar algorithms familiar to the networking community, setting this paper apart from other surveys on the broad topic of publish/subscribe systems.

A Novel Approach to Adaptive Content-based Subscription Management in DHT-based Overlay Networks

Content-based Publish/Subscribe (CBPS) interaction paradigm is suitable for a variety of large scale dynamic applications: News delivery, stock quoting, air and metropolitan traffic control, on-line games, dissemination of multimedia contents, dissemination of auction bids, services and resources discovery, remote control of critical infrastructures and management of large scale systems. In contrast to their flexibility and expressiveness scalable CBPS systems are difficult to implement and the proposed solutions are not again mature. In this paper we propose a novel adaptive content-based subscription management system, relying on a Distributed Hash Table routing infrastructure. We define a model for the event space guaranteeing the expressiveness for any application domain. Also we provide mechanism to dynamically identify groups of users with similar preferences (multicast group). Two different approaches for multicast group identification are proposed and discussed: A dummy regular Grid partitioning of the event space and a more complex clustering of users preferences. When system evolves (e.g. joining and leaving of publisher and subscriber and/or nodes failure) multicast groups are reconfigured to maintain the desired level of system efficiency. System adaptation is supported by a scalable and dynamic application-level multicast infrastructure, and by a run time performance measurement mechanism. The paper gives a formal description of our solution and the main design guidelines. Also we discuss advantages and disadvantages of our proposal, compared with the existing solutions.

Exploiting IP multicast in content-based publish-subscribe systems

Exploiting IP multicast in content-based publish-subscribe systems