Apache Storm Research Articles

Stream-aware indexing for distributed inequality join processing

Inequality join is an operator to join data on inequality conditions and it is a fundamental building block for applications. While methods and optimizations exist for efficient inequality join in batch processing, little attention has been given to its streaming version, particularly to large-scale data-intensive applications that run on Distributed Stream Processing Systems (DSPSs). Designing an inequality join in streaming and distributed settings is not an easy task: (i) indexes have to be employed to efficiently support inequality-based comparisons, but the continuous stream of data imposes continuous insertions, updates, and deletions of elements in the indexes—hence a huge overhead for the DSPSs; (ii) oftentimes real data is skewed, which makes indexing even more challenging.To address these challenges, we propose the Stream-Aware inequality join (STA), an indexing method that can reduce redundancy and index update overhead. STA builds a separate in-memory index structure for hotkeys, i.e., the most frequently used keys, which are automatically identified with an efficient data sketch. On the other hand, the cold keys are treated using a linked set of index structures. In this way, STA avoids many superfluous index updates for frequent items. Finally, we implement four state-of-the-art inequality join solutions for a widely employed DSPS (Apache Storm) and compare their performance with STA on four real-world data sets and a synthetic one. The results of our experimental evaluation reveal that our stream-aware approach outperforms existing solutions.

Wearable Smart Cardiac Care: Take Vipasyana for Example

The most direct challenge to the healthcare system in Taiwan associated with population aging is the chronic and growing shortfall in medical and nursing practitioners. The integration of artificial intelligence, Internet of Things, and advanced 5G communication technologies is making healthcare more streamlined and intuitive to improve healthcare service efficiency, encourage cross-disciplinary cooperation to improve patient health management, and introduce and mainstream innovative service models such as telecare. To this end, the wearable smart cardiac care system 'Vipasyana', jointly developed by the interdisciplinary research team of the Departments of Electrical Engineering and Computer Science at National Yang Ming Chiao Tung University, not only has an innovative fully functional wireless wearable electrocardiograph but also uses machine learning-based detection/prediction methods to detect and monitor for emergencies such as myocardial infarction, ventricular tachycardia/ventricular fibrillation, and heart failure. To handle the real-time streaming sensing data generated during clinical trials, a big data real-time streaming framework based on Apache Storm and microservices has also been integrated. In addition, a mobile phone app and virtual reality 3D ward round system providing a user-friendly management and monitoring tool for medical staffs were developed and integrated into the system. The 'Vipasyana' system is capable of providing precision medical services to individual patients.

Big Data Analytic Tools Usage among Academic Libraries in Tanzania

This study examines the implementation of Big Data analytics in Tanzanian University libraries. Employing a qualitative approach within the interpretive paradigm, the study conducted interviews to collect data from 12 library staff across various University libraries in Tanzania. The qualitative data collected was analyzed using Atlasi.ti 7, focusing on emerging themes. The findings established that most University libraries in Tanzania utilize social networking sites for marketing their products and services. However, only a few academic libraries employ analytics tools such as Apache Hadoop, MongoDB, Apache Spark, Tableau, Zoho Analytics, Apache Storm, and others due to insufficient skills. The study concludes that library administrators could greatly benefit from integrating Big Data analytics into their daily operations. Recommendations include developing Big Data analytics policies to ensure proper handling of user data and offering training on the use of such analytic tools to librarians to equip them with necessary skills and techniques.

Big Data Analytic Tools Usage among Academic Libraries in Tanzania

This study examines the implementation of Big Data analytics in Tanzanian University libraries. Employing a qualitative approach within the interpretive paradigm, the study conducted interviews to collect data from 12 library staff across various University libraries in Tanzania. The qualitative data collected was analyzed using Atlasi.ti 7, focusing on emerging themes. The findings established that most University libraries in Tanzania utilize social networking sites for marketing their products and services. However, only a few academic libraries employ analytics tools such as Apache Hadoop, MongoDB, Apache Spark, Tableau, Zoho Analytics, Apache Storm, and others due to insufficient skills. The study concludes that library administrators could greatly benefit from integrating Big Data analytics into their daily operations. Recommendations include developing Big Data analytics policies to ensure proper handling of user data and offering training on the use of such analytic tools to librarians to equip them with necessary skills and techniques.

Adaptive key partitioning in distributed stream processing

In stream processing systems, Key Grouping is a commonly employed partitioning scheme for distributing input tuples among parallel instances of stateful operators. With key grouping, tuples shared public keys in the stream are designated to the specific instance responsible for that key. Typically, the implementation of key grouping involves the use of a hash function. While it is convenient and deterministic, it is also known to cause load imbalance between parallel instances, especially in the presence of skewed data streams. Key-Splitting is an effective technique that distributes tasks associated with keys to downstream operators, facilitating load balancing at a relatively low cost. However, overly increasing parallel instances can lead to excessive aggregation costs, becoming a system bottleneck. In this paper, we show the high aggregation cost brought by the Key-Splitting partitioner at different levels of key separation. To address this challenge, we introduce an adaptive Key-Splitting method which controlling the degree of key separation. We propose a partitioner named FlexD, which aims to achieve dynamic adaptation of key separation limits for streaming data. The partitioner employs key grouping to distribute rare keys and dynamic expansion of processing instances to distribute hot keys. We implemented our method on Apache Storm and evaluated it by using real-world and synthetic datasets. Experimental results show that our method achieves a good balance between load balancing and aggregation cost. Moreover, it outperforms existing methods, achieving higher throughput.

Bibliometric behavior of big data and digital marketing as real-time multimedia

Technological trends such as big data have generated interest in its application in digital marketing, due to the ease of precision in business and in daily decision-making, where there is a need to respond to the needs of the market in real time and achieve competitiveness. We aim to describe the bibliometric behavior of big data and digital marketing as real-time multimedia applications during the period from 2012 to 2023. We based our methodology on the bibliometric analysis of statistical relationships using VOSviewer software. We employed the normalization technique and applied the association strength method for keyword co-occurrence analysis and author co-citation analysis. Additionally, we used the hermeneutic technique to interpret the results. The findings indicate that research trends are associated with social networks; data processing; machine learning techniques; real-time system; online system; data analysis; data management. The contributing authors were Wang Y.; Chen Y.; Liu Y.; Zhang X.; Wang X.; Wang J.; Zhang Y.; Li J. We concluded that the common software in the study includes Hadoop, Reduced Map, Apache Spark, Twitter, Apache Storm, Spark Transmission, Transformer, and Weibo.

SPinDP: A High-Speed Distributed Processing Platform for Sampling and Filtering Data Streams

Recently, there has been an explosive generation of streaming data in various fields such as IoT and network attack detection, medical data monitoring, and financial trend analysis. These domains require precise and rapid analysis capabilities by minimizing noise from continuously generated raw data. In this paper, we propose SPinDP (Stream Purifier in Distributed Platform), an open source-based high-speed stream purification platform, to support real-time stream purification. SPinDP consists of four major components, Data Stream Processing Engine, Purification Library, Plan Manager, and Shared Storage, and operates based on open-source systems including Apache Storm and Apache Kafka. In these components, stream processing throughput and latency are critical performance metrics, and SPinDP significantly enhances distributed processing performance by utilizing the ultra-high-speed network RDMA (Remote Direct Memory Access). For the performance evaluation, we use a distributed cluster environment consisting of nine nodes, and we show that SPinDP’s stream processing throughput is more than 28 times higher than that of the existing Ethernet environment. SPinDP also significantly reduces the processing latency by more than 2473 times on average. These results indicate that the proposed SPinDP is an excellent integrated platform that can efficiently purify high-speed and large-scale streams through RDMA-based distributed processing.

An adaptive load balancing strategy for stateful join operator in skewed data stream environments

As one of the most computationally intensive operations in stream processing applications, join operation can cause severe load imbalance problem when dealing with skewed data. Most of the popular solutions focused on monitoring-based dynamic balancing strategies, making it difficult to quickly adapt to the changing frequency of data stream, and sometimes failing the balancing strategies that try to address the skewed load in the cluster. To address these issues, we propose to use the prediction results of a deep reinforcement learning model and adjust the grouping strategy in advance before the frequency change of data stream. It will enable the system to quickly adapt to data stream fluctuation, while managing the resources for effective resource utilization. The following contributions are made in this paper: 1) Explore the main factors that trigger the load skewness problem in distributed stream join systems and carefully model the load balancing problem at the application level. 2) Develop a Gated Recurrent Unit Sequence to Sequence model to predict key frequency distribution of streams, and propose a dynamic grouping algorithm and a feedback-based resource elasticity scaling algorithm to solve the load imbalance problem caused by hot keys in real time. 3) Design and implement an adaptive stream join system Aj-Stream based on the prediction model and the proposed algorithm on Apache Storm. 4) Evaluate the system performance through extensive experiments on a large scale real-world dataset and multiple synthetic datasets. The experimental results demonstrate that the Aj-Stream proposed in this paper exhibits stable throughput and latency performance with both static data streams of varying skewnesses and dynamic data streams. In comparison to existing stream-connected systems, Aj-Stream demonstrated a 22.1% increase in system throughput and a 45.5% decrease in system latency when dealing with frequently fluctuating data streams.

General-purpose data stream processing on heterogeneous architectures with WindFlow

Many emerging applications analyze data streams by running graphs of communicating tasks called operators. To develop and deploy such applications, Stream Processing Systems (SPSs) like Apache Storm and Flink have been made available to researchers and practitioners. They exhibit imperative or declarative programming interfaces to develop operators running arbitrary algorithms working on structured or unstructured data streams. In this context, the interest in leveraging hardware acceleration with GPUs has become more pronounced in high-throughput use cases. Unfortunately, GPU acceleration has been studied for relational operators working on structured streams only, while non-relational operators have often been overlooked. This paper presents WindFlow, a library supporting the seamless GPU offloading of general partitioned-stateful operators, extending the range of operators that benefit from hardware acceleration. Its design provides high throughput still exposing a high-level API to users compared with the raw utilization of GPUs in Apache Flink.

Adaptive Fragment-Based Parallel State Recovery for Stream Processing Systems

Today, large-scale cloud organizations are deploying datacenters and “edge” clusters globally to provide their users with low-latency access to their services. Running stream applications across these geo-distributed sites are emerging as a daily requirement, such as making business decisions from marketing streams, identifying spam campaigns from social network streams, and analyzing existing genomes in different labs and countries to track the sources of a potential epidemic. However, while the progress has been encouraging, the existing efforts have dominantly centered around <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">stateless stream processing</i> , leaving another urgent trend- <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">stateful stream processing</i> -much less explored. A driving need is that next-generation stream processing systems need to store and update states during processing, and most importantly, successfully recover large distributed states when faults and failures happen. Existing studies exhibit major limitations including: (1) they mostly inherit MapReduce's “single master/many workers” architecture, where the central master is responsible for all scheduling activities and easily becomes a scalability bottleneck; (2) they offer state recovery mainly through the use of three approaches: replication recovery, checkpointing recovery, and DStream-based lineage recovery, which are either slow, resource-expensive or failing to handle multiple failures; and (3) they are not adaptive to heterogeneous hardware settings in the cloud. In this paper, we present A-FP4S, a novel adaptive fragments-based parallel state recovery mechanism for stream processing systems to manage and recover large distributed states for a massive number of stream applications. The novelty of A-FP4S is that we organize stream operators into a distributed hash table (DHT) based peer-to-peer (P2P) overlay. Then we divide each node's local state into many fragments and periodically store them in each node's multiple neighbors (the leaf set nodes of DHT), ensuring that different sets of available fragments can reconstruct failed states in parallel. By doing that, this failure recovery mechanism is extremely scalable to the size of the lost state, significantly reduces the failure recovery time, and can tolerate multiple node failures. A-FP4S is adaptive to heterogeneous hardware settings (e.g., CPU speed, disk/file-system speed, network bandwidth) by automatic parameter tuning over phases. Compared to Apache Storm, A-FP4S achieves a significant 31.8% to 50.5% reduction in recovery latency. It can scale to many simultaneous failures and successfully recover the state, even more than half of the operators fail or get lost. Large-scale experiments using real-world datasets demonstrate A-FP4S's attractive scalability and adaptivity properties.

A COMPREHENSIVE STUDY ON BIG DATA FRAMEWORKS

With the advent of cloud computing technology, the generation of data from various sources has increased during the last few years. The current data processing technology must handle the enormous volumes of newly created data. Therefore, the studies in the literature have concentrated on big data, which has enormous volumes of almost unstructured data. Dealing with such data needs well-designed frameworks that fulfil developers’ needs and fit colourful purposes. Moreover, these frameworks can use for storing, processing, structuring, and analyzing data. The main problem facing cloud computing developers is selecting the most suitable framework for their applications. The literature includes many works on these frameworks. However, there is still a severe gap in providing comprehensive studies on this crucial area of research. Hence, this article presents a novel comprehensive comparison among the most popular frameworks for big data, such as Apache Hadoop, Apache Spark, Apache Flink, Apache Storm, and MongoDB. In addition, the main characteristics of each framework in terms of advantages and drawbacks are also deeply investigated in this article. Our research provides a comprehensive analysis of various metrics related to data processing, including data flow, computational model, overall performance, fault tolerance, scalability, interval processing, language support, latency, and processing speed. To our knowledge, no previous research has conducted a detailed study of all these characteristics simultaneously. Therefore, our study contributes significantly to the understanding of the factors that impact data processing and provides valuable insights for practitioners and researchers in the field

TWITTER SENTIMENTAL ANALYSIS

This paper presents the effectiveness of linguistic features to identify the sentiment of Twitter messages using the apache storm framework. We calculate the effectiveness of present lexical resources and features that capture information about the informal and creative language used in microblogging. In the past few years, there has been a huge growth in the use of microblogging platforms such as Twitter. Influenced by intensification, companies, and media organizations are increasingly seeking ways to excavate Twitter information about what people think and feel about their products and services. Here we download Twitter messages for a particular hashtag and carry out sentiment analysis i.e. to find a positive, negative or neutral sense of that tweet using apache storm framework. Each hashtag may have 1000 of comments and new comments are added every minute, in order to handle so many live tweets we are using apache storm framework.

SDN-enabled Resource Provisioning Framework for Geo-Distributed Streaming Analytics

Geographically distributed (geo-distributed) datacenters for stream data processing typically comprise multiple edges and core datacenters connected through Wide-Area Network (WAN) with a master node responsible for allocating tasks to worker nodes. Since WAN links significantly impact the performance of distributed task execution, the existing task assignment approach is unsuitable for distributed stream data processing with low latency and high throughput demand. In this paper, we propose SAFA, a resource provisioning framework using the Software-Defined Networking (SDN) concept with an SDN controller responsible for monitoring the WAN, selecting an appropriate subset of worker nodes, and assigning tasks to the designated worker nodes. We implemented the data plane of the framework in P4 and the control plane components in Python. We tested the performance of the proposed system on Apache Spark, Apache Storm, and Apache Flink using the Yahoo! streaming benchmark on a set of custom topologies. The results of the experiments validate that the proposed approach is viable for distributed stream processing and confirm that it can improve at least 1.64× the processing time of incoming events of the current stream processing systems.

IRONEDGE: Stream Processing Architecture for Edge Applications

This paper presents IRONEDGE, an architectural framework that can be used in different edge Stream Processing solutions for “Smart Infrastructure” scenarios, on a case-by-case basis. The architectural framework identifies the common components that any such solution should implement and a generic processing pipeline. In particular, the framework is considered in the context of a study case regarding Internet of Things (IoT) devices to be attached to rolling stock in a railway. A lack of computation and storage resources available in edge devices and infrequent network connectivity are not often seen in the existing literature, but were considered in this paper. Two distinct implementations of IRONEDGE were considered and tested. One, identified as Apache Kafka with Kafka Connect (K0-WC), uses Kafka Connect to pass messages from MQ Telemetry Transport (MQTT) to Apache Kafka. The second scenario, identified as Apache Kafka with No Kafka Connect (K1-NC), allows Apache Storm to consume messages directly. When the data rate increased, K0-WC showed low throughput resulting from high losses, whereas K1-NC displayed an increase in throughput, but did not match the input rate for the Data Reports. The results showed that the framework can be used for defining new solutions for edge Stream Processing scenarios and identified a reference implementation for the considered study case. In future work, the authors propose to extend the evaluation of the architectural variation of K1-NC.

STORMING DATA IN THE CIRCULAR ECONOMY

The traditional linear approach of “manufacture, consume, discard” cannot be sustainable in a world with ever-expanding consumption rates. With regulations and the public becoming more and more sensitive toward environmental issues, companies seem pressured to shift to new business models. This work is focused on reviewing current technological solutions being applied in the transition to the circular economy. Focus will be on the opportunities created by applying real-time communication frameworks in business processes. This will be done by identifying current problems and solutions to those problems, improvements that can be made by introducing real-time communication, how real-time communication is engineered, and ways these improvements will lead to a more circular economy. This article explores the application of real-time communication frameworks, specifically leveraging Apache Storm, to optimize resource utilization, enhance sustainability, and facilitate the transition from a linear economy to a circular model. Keywords: Real-Time Communication, Circular Economy, Sustainability, Apache Storm, Optimization DOI： https://doi.org/10.35741/issn.0258-2724.58.3.15

Development of infrastructure for anomalies detectionin big data

The work describes the conducted analysis of models, methods, and technologies for detecting anomalies in data. It concludes that, based on the analysis, the solution to the problem of anomaly detection in data should be considered as a complex technology, which consists of the formation and application of mathematical models in combination with the research of data processing approaches. The article analyses the current state of big data stream processing technologies and reflects the peculiarities of the most commonly used and advanced of them, e.g. Apache Hadoop, Apache Spark, Apache Cassandra, Apache Kafka, Apache Storm, and Apache Beam. On top of these, it pays attention to the infrastructure, in which the created software models can be deployed and used, taking into account the high-load real-time nature of the data. The article proposes to form an infrastructure for anomaly detection in data as an applied example of big data processing cloud infrastructure. The paper demonstrates the developed infrastructure model for anomaly detection in real-time stream data, which is based on an expert method of forming requirements for a software component, choosing an algorithm for detecting anomalies, selecting tools, and improving the algorithm. The highlighted anomaly detection tools allow us to create a secure real-time anomaly detection solution using Dataflow, BigQuery ML, and Cloud DLP. The paper presents the applied implementation of anomaly detection in real-time using GCP and Apache Beam -data stream analysis of software logs in the information system and detection of fraudulent ones among them, which will help improve the cyber security of the system. In the end, the work demonstrates possible improvements to the basic model that could help to speed it up

Delay-Resistant Geo-Distributed Analytics

Big data analytics platforms have played a critical role in the unprecedented success of data-driven applications. However, real-time and streaming data applications, and recent legislation, e.g., GDPR in Europe, have posed constraints on exchanging and analyzing data, especially personal data, across geographic regions. To address such constraints data has to be processed and analyzed in-situ and aggregated results have to be exchanged among the different sites for further processing. This introduces additional network delays due to the geographic distribution of the sites and potentially affecting the performance of analytics platforms that are designed to operate in datacenters with low network delays. In this paper, we show that the three most popular big data analytics systems (Apache Storm, Apache Spark, and Apache Flink) fail to tolerate round-trip times more than 30 milliseconds even when the input data rate is low. The execution time of distributed big data analytics tasks degrades substantially after this threshold, and some of the systems are more sensitive than others. A closer examination and understanding of the design of these systems show that there is no winner in all wide-area settings. However, we show that it is possible to improve the performance of all these popular big data analytics systems significantly amid even transcontinental delays (where inter-node delay is more than 30 milliseconds) and achieve performance comparable to this within a datacenter for the same load.

JORA: Blockchain-based efficient joint computing offloading and resource allocation for edge video streaming systems

Edge computing has been widely applied in video streaming systems, aiming to provide low-latency video streaming services. However, limited resources and a lack of incentivization affect the performance of edge video streaming systems. In this paper, we propose an efficient Joint computing Offloading and Resource Allocation (JORA) method for blockchain-supported edge video streaming systems (BEVS). Specifically, we develop a novel incentive mechanism based on the blockchain smart contract to enable automatic incentivization. Meanwhile, we customize a system prototype of BEVS on Amazon EC2 using Apache Storm. Preliminary experiments with the customized prototype reveal the coupling impacts of the offloading mode, the allocation scheme, and video redundancy on system performance and resource demand. Motivated by the results, we formulate joint computing offloading, resource allocation, and video adaptive compression as an optimization problem. We transform the original problem into a convex problem and a mixed-integer programming problem (MIP). We then develop an algorithm based on the generalized Benders decomposition (GBD) to solve the original problem in an iterative way. Extensive experiments demonstrate that the proposed scheme achieves full resource utilization and an effective trade-off between energy consumption and accuracy.

Network-aware worker placement for wide-area streaming analytics

Many organizations leverage Distributed Stream processing systems (DPSs) to get insights from the data generated by different users/devices, e.g., the Internet of Things (IoT) devices or user clicks on a website, on geographically distributed datacenters. The worker nodes in such environments are connected through Wide Area Network (WAN) links with various delays and bandwidth. Therefore, minimizing the execution latency of a task on the worker nodes while using the links with enough bandwidth and lower cost to steer the traffic of the applications is a challenging task. In this paper, we formulate the worker node placement for a geo-distributed DSPs network as a multi-criteria decision-making problem. Then, we propose an additive weighting-based approach to solve it. The users can prioritize the worker node placement according to the network-relevant parameters. We also propose a framework that can be integrated with the current DPSs to execute the tasks. We test our placement approach on three widely used stream processing systems, i.e., Apache Spark, Apache Storm, and Apache Flink, on three custom graphs adopted from the real cloud providers. We run the streaming query of the Yahoo! streaming benchmark on these three DPSs. The experimental results show that our approach improves the performance of Spark up to 2.2x–7.2x, Storm up to 1.2x–3.4x, and Flink up to 1.4x–3.3x compared with other placement approaches, which makes our framework useful for use in practical environments.

Balanced Schedule on Storm for Performance Enhancement

In recent years, real-time and big data aroused and received a lot of attention due to the spread of embedded systems in almost everything in life. This has led to many challenges that need to be solved to enhance and improve systems that work on big real-time data. Apache Storm is a system used for computing and analyzing big real-time data of distributed systems. This paper aims to develop a scheduler to improve the scheduling of the applications represented by topologies on the Storm cluster. The proposed scheduler is hybridization between the scheduling algorithms of A3 Storm and the Workload scheduler. Its objective is to minimize the communication between tasks while balancing the workload on all cluster machines. The proposed scheduler is compared with the A3 Storm and Fischer and Bernstein’s scheduling algorithm. The comparison has been made using four different topologies. The experimental results show that our proposed scheduler outperforms the two other schedulers in throughput and complete latency.