Content-based Publish/subscribe Research Articles

OEM: An operation-aware event matching algorithm for content-based Pub/Sub systems

Due to the continuous expansion of the user base and the heightened demands for quality of service (QoS), real-time data distribution has encountered significant challenges in event matching within large-scale content-based publish/subscribe (CPS) systems. Many efficient algorithms have been proposed to enhance the matching performance. However, most existing algorithms scarcely consider operation characteristics, which often leads to performance degradation due to a multitude of repetitive operations, such as comparisons, additions, and assignments. In this paper, we propose an Operation-aware Event Matching algorithm called OEM, which takes into account the efficiency of CPU operations, SIMD instructions, and multi-thread architecture. First, we introduce a bitset-based subscription cache mechanism (SCM), which enables the efficient execution of bitwise OR operations during the matching process. Furthermore, we propose two optimizations, namely an optimal skewness-aware cell grouping (OSCG) strategy and an attribute-based clustering mechanism (ACM), to enhance the efficiency of OEM. These optimizations effectively address the challenges posed by data skewness, low cache efficiency, and high dimensionality. In addition, we establish a performance analysis model that characterizes the trade-off between memory consumption and performance improvement. We conducted extensive experiments to evaluate the performance of OEM. Compared to six state-of-the-art algorithms, namely REIN, Ada-REIN, TAMA, OpIndex, PEM and fgPEM, OEM achieves an average reduction in matching time by 40.4×, 38.5×, 12.0×, 33.1×, 27.8×, and 25.5×, respectively.

Enabling Large-Scale IoT-Based Services through Elastic Publish/Subscribe.

In this paper, we report an algorithm that is designed to leverage the cloud as infrastructure to support Internet of Things (IoT) by elastically scaling in/out so that IoT-based service users never stop receiving sensors’ data. This algorithm is able to provide an uninterrupted service to end users even during the scaling operation since its internal state repartitioning is transparent for publishers or subscribers; its scaling operation is time-bounded and depends only on the dimension of the state partitions to be transmitted to the different nodes. We describe its implementation in E-SilboPS, an elastic content-based publish/subscribe (CBPS) system specifically designed to support context-aware sensing and communication in IoT-based services. E-SilboPS is a key internal asset of the FIWARE IoT services enablement platform, which offers an architecture of components specifically designed to capture data from, or act upon, IoT devices as easily as reading/changing the value of attributes linked to context entities. In addition, we discuss the quantitative measurements used to evaluate the scale-out process, as well as the results of this evaluation. This new feature rounds out the context-aware content-based features of E-SilboPS by providing, for example, the necessary middleware for constructing dashboards and monitoring panels that are capable of dynamically changing queries and continuously handling data in IoT-based services.

Design of High Performance, Scalable Content-based Publish-Subscribe System using MPI-CUDA Approach

International Journal of Computer Sciences and Engineering (A UGC Approved and indexed with DOI, ICI and Approved, DPI Digital Library) is one of the leading and growing open access, peer-reviewed, monthly, and scientific research journal for scientists, engineers, research scholars, and academicians, which gains a foothold in Asia and opens to the world, aims to publish original, theoretical and practical advances in Computer Science,Information Technology, Engineering (Software, Mechanical, Civil, Electronics & Electrical), and all interdisciplinary streams of Computing Sciences. It intends to disseminate original, scientific, theoretical or applied research in the field of Computer Sciences and allied fields. It provides a platform for publishing results and research with a strong empirical component. It aims to bridge the significant gap between research and practice by promoting the publication of original, novel, industry-relevant research.

PCP: A Privacy-Preserving Content-Based Publish–Subscribe Scheme With Differential Privacy in Fog Computing

Fog computing dramatically extends the cloud computing to the edge of the network and admirably solves the problem that the brokers (in publish-subscribe system) generally lack of computing capacity and energy power. However, brokers may be disguised, hacked, sniffed, and corrupted. The traditional security technology cannot protect the system privacy when facing a possible collusion attack. In this paper, we propose a privacy-preserving content-based publish/subscribe scheme with differential privacy in fog computing context, named PCP, where the fog nodes act as the brokers. Specifically, PCP firstly utilizes the U-Apriori algorithm to mine the top-K frequent itemsets (i.e., the attributes) from uncertain data sets, then applies the exponential and Laplace mechanism to ensure the differential privacy, and the broker uses the mined top-K itemsets to match appropriate publisher and subscriber finally. Security analysis shows that the PCP can guarantee differential privacy in theory. To evaluate the performance of PCP, we carry out experiments with real-world scenario data sets. The experimental results show that PCP efficiently achieves the tradeoff between the system cost and the privacy demand.

Universal Cross-Cloud Communication

Integration of applications, data-centers, and programming abstractions in the cloud-of-clouds poses many challenges to system engineers. Different cloud providers offer different communication abstractions, and applications exhibit different communication patterns. By abstracting from hardware addresses and lower-level communication, the publish/subscribe paradigm seems like an adequate abstraction for supporting communication across clouds, as it supports many-to-many communication between publishers and subscribers, of which one-to-one or one-to-many can be viewed as special cases. In particular, content-based publish/subscribe (CPS) systems provide an expressive abstraction that matches well with the key-value pair model of many established cloud storage and computing systems, and decentralized overlay-based CPS implementations scale up well. However, CPS systems perform poorly at small scale, e.g., one-to-one or one-to-many communication. This holds especially for multi-send scenarios which we refer to as entourages that may range from a channel between a publisher and a single subscriber to a broadcast between a publisher and a handful of subscribers. These scenarios are common in cloud computing, where cheap hardware is exploited for parallelism (efficiency) and redundancy (fault-tolerance). With CPS, multi-send messages go over several hops before their destinations are even identified via predicate matching, resulting in increased latency, especially when destinations are located in different data-centers or zones. Topic-based publish/subscribe (TPS) systems support communication at small scale more efficiently, but still route messages over multiple hops and inversely lack the flexibility of CPS systems. In this paper, we propose CPS protocols for cloud-of-clouds communication that can dynamically identify entourages of publishers and corresponding subscribers. Our CPS protocols dynamically connect the publishers with their entourages through uberlays . These uberlays can transmit messages from a publisher to its corresponding subscribers with low latency. Our experiments show that our protocols make CPS abstraction viable and beneficial for many applications. We introduce a CPS system named Atmosphere that leverages out CPS protocols and illustrate how Atmosphere has allowed us to implement, with little effort, versions of the popular HDFS and ZooKeeper systems which operate efficiently across data-centers.

On the performance improvement of gossip protocols for content-based publish-subscribe through caching

Gossip protocols are a common solution for content-based publish-subscribe. However, in spite of the advantages that these protocols present in comparison to other approaches, such as distributed hash tables, they also have inherent scalability issues. In this paper, CCPS (Caching Content-based Publish-Subscribe) is presented as a publish-subscribe protocol designed to address these scalability issues. Through caching, CCPS exploits search query repetition to address load balancing problems. It has been designed having in mind its use in combination with existing publish-subscribe solutions so as to improve their performance. In the paper, the protocol description together with the implementation details of its fundamental operations are provided, while CCPS performance is evaluated against standard gossip protocols.

Parametric Content-Based Publish/Subscribe

Content-based publish/subscribe (CPS) is an appealing abstraction for building scalable distributed systems, e.g., message boards, intrusion detectors, or algorithmic stock trading platforms. Recently, CPS extensions have been proposed for location-based services like vehicular networks, mobile social networking, and so on. Although current CPS middleware systems are dynamic in the way they support the joining and leaving of publishers and subscribers, they fall short in supporting subscription adaptations. These are becoming increasingly important across many CPS applications. In algorithmic high frequency trading, for instance, stock price thresholds that are of interest to a trader change rapidly, and gains directly hinge on the reaction time to relevant fluctuations rather than fixed values. In location-aware applications, a subscription is a function of the subscriber location (e.g. GPS coordinates), which inherently changes during motion. The common solution for adapting a subscription consists of a resubscription, where a new subscription is issued and the superseded one canceled. This incurs substantial overhead in CPS middleware systems, and leads to missed or duplicated events during the transition. In this article, we explore the concept of parametric subscriptions for capturing subscription adaptations. We discuss desirable and feasible guarantees for corresponding support, and propose novel algorithms for updating routing mechanisms effectively and efficiently in classic decentralized CPS broker overlay networks. Compared to resubscriptions, our algorithms significantly improve the reaction time to subscription updates without hampering throughput or latency under high update rates. We also propose and evaluate approximation techniques to detect and mitigate pathological cases of high frequency subscription oscillations, which could significantly decrease the throughput of CPS systems thereby affecting other subscribers. We analyze the benefits of our support through implementations of our algorithms in two CPS systems, and by evaluating our algorithms on two different application scenarios.

A content-based publish/subscribe framework for large-scale content delivery

The publish/subscribe communication paradigm has become an important architectural style for designing distributed systems and has recently been considered one of the most promising future network architectures that solves many challenges of content delivery in the current Internet. This work is concerned with scaling decentralized content-based publish/subscribe (CBPS) networks for large-scale content distribution. A fundamental step for CBPS networks to reach the large-scale is to move from the current exhaustive filtering service model, where a subscription selects every relevant publication, to a service model capturing the quantitative and qualitative heterogeneity of information consumers requirements. Moreover, the proposed work aims at leveraging caching for increasing the communication efficiency of CBPS operating at large-scale characterized by widely spread information consumers with heterogeneous requirements, large number of publications and scarcity of end-to-end bandwidth. We propose and design a service model for addressing the consumers’ requirements for content-based information retrieval and describe the relevant protocols necessary to implement such a service. We evaluate the proposed approach, by using realistic workload scenarios and comparing different content and interest forwarding strategies as well as caching policies in terms of resource efficiency and user perceived QoS metrics.

A Novel Approach to Adaptive Content-based Subscription Management in DHT-based Overlay Networks

Content-based Publish/Subscribe (CBPS) interaction paradigm is suitable for a variety of large scale dynamic applications: News delivery, stock quoting, air and metropolitan traffic control, on-line games, dissemination of multimedia contents, dissemination of auction bids, services and resources discovery, remote control of critical infrastructures and management of large scale systems. In contrast to their flexibility and expressiveness scalable CBPS systems are difficult to implement and the proposed solutions are not again mature. In this paper we propose a novel adaptive content-based subscription management system, relying on a Distributed Hash Table routing infrastructure. We define a model for the event space guaranteeing the expressiveness for any application domain. Also we provide mechanism to dynamically identify groups of users with similar preferences (multicast group). Two different approaches for multicast group identification are proposed and discussed: A dummy regular Grid partitioning of the event space and a more complex clustering of users preferences. When system evolves (e.g. joining and leaving of publisher and subscriber and/or nodes failure) multicast groups are reconfigured to maintain the desired level of system efficiency. System adaptation is supported by a scalable and dynamic application-level multicast infrastructure, and by a run time performance measurement mechanism. The paper gives a formal description of our solution and the main design guidelines. Also we discuss advantages and disadvantages of our proposal, compared with the existing solutions.

A security framework for Content-Based Publish–Subscribe system

Content-Based Publish–Subscribe system is an important emerging model in e-commerce applications. The distribution of contents to the Subscribers in a scalable and secured way is a very important requirement of these applications. Security schemes exercising centralized control cannot be applied directly to these applications due to the increased number of Subscribers subscribing to a particular event. This is because of the enormous growth of Internet. The centralized approach increases the burden on the central controller, which is the major entity in security management. To solve this problem, an efficient grouping is done based on the decentralized mechanism and a novel key management scheme is proposed for secured distribution of contents. The proposed method minimizes the overheads. The model is simulated and the results are found to be optimum compared to other methods.

Research on Routing Algorithm and Self-Configuration in Content-Based Publish-Subscribe System

Content-Based publish/subscribe systems have recently received an increasing attention. Efficient routing algorithms and self-configuration are two key issues in the area of large-scale content-based publish/subscribe systems. Although many routing algorithms have been proposed, none of them fully exploits multicast to enhance system performance and save network bandwidth. In addition, the vast majority of currently available publish-subscribe middleware has ignored this self-configuration problem. This paper first proposes a hierarchical system model with multicast clustering. Then a hybrid routing algorithm is presented, which can fully exploit multicast in order to reduce the used network bandwidth. Moreover, a multicast clustering replication protocol and a content-based multicast tree protocol are presented for coping with the node or link failures and rebuilding the event dispatcher trees. Experimental results reveal that the system has better routing efficiency and lower cost, and guarantees the self-configuration characteristic.