Event-driven Applications Research Articles

EVENT-DRIVEN PROGRAMMING FOR HIGH THROUGHPUT APPLICATIONS

EVENT-DRIVEN PROGRAMMING FOR HIGH THROUGHPUT APPLICATIONS

Tools and techniques for real-time data processing: A review

Real-time data processing is an essential component in the modern data landscape, where vast amounts of data are generated continuously from various sources such as Internet of Things devices, social media, financial transactions, and manufacturing systems. Unlike traditional batch processing methods that analyse data in intervals, real-time data processing enables the continuous intake, manipulation, and analysis of data within milliseconds of generation. This capability is critical for applications requiring instant insights and rapid decision-making, including fraud detection, predictive maintenance, real-time analytics, and autonomous operations. This paper reviews the tools and techniques that have revolutionized real-time data processing, with a focus on cutting-edge platforms such as Apache Kafka and Apache Flink, as well as cloud-native solutions. These technologies offer scalable and fault-tolerant systems capable of managing high-volume data streams while ensuring low latency and data consistency. Apache Kafka provides a highly scalable distributed messaging system, while Apache Flink combines stateful and stateless processing to support complex event-driven applications. This review highlights the. This paper reviews key techniques and tools used in real-time data processing, including stream processing, complex event processing, in-memory computing, micro-batching, and real-time dashboards. In addition, it highlights advancements in real-time data processing frameworks, their capabilities, and their impact on modern business applications. Additionally, the paper explores various tools used in real-time data processing, including Apache Kafka for data ingestion, Apache Flink and Spark Streaming for stream processing, Redis and Apache Druid for real-time storage, and Grafana and Kibana for data visualization. By examining these techniques and tools, this paper highlights the importance of real-time data processing in enabling businesses to make data-driven decisions with minimal latency, ultimately gaining a competitive edge in the rapidly evolving digital world.

Building Scalable Data Processing Systems with Kafka on Kubernetes

Real-time data processing has become a critical requirement for modern enterprises, presenting unique challenges in scalability, reliability, and performance. This technical article explores the synergy between Apache Kafka and Kubernetes in building robust, production-grade streaming architectures. This article presents a detailed examination of how Kafka's distributed streaming capabilities, combined with Kubernetes' container orchestration, create a resilient foundation for handling high-velocity data streams. Through practical examples from insurance claims processing and manufacturing IoT systems, we demonstrate proven patterns for deployment, scaling, and monitoring. The article provides deep technical insights into topic partitioning strategies, StatefulSet configurations, and operational best practices while addressing critical concerns around security, cost optimization, and disaster recovery. This article shows that this architectural approach can significantly reduce processing latency, handle unpredictable workload spikes, and maintain system reliability at scale, making it particularly valuable for enterprises dealing with real-time analytics and event-driven applications.

A Comprehensive Study of Azure Functions: Features, Architectures, Use Case and Functionality

Azure Function is a service available on cloud, offered by Microsoft Azure. It allows you to build and run event-driven applications without having to manage infrastructure. The code can be developed in any programming language. Functions in Azure are triggered by events. These events can come from various sources such as HTTP requests, message queues (e.g., Azure Queue Storage, Azure Service Bus), timers, blob storage changes, and more. This paper provides an introduction of how Azure Function works in Azure cloud.

Reliable and efficient webserver management for task scheduling in edge-cloud platform

<span lang="EN-US">The development in the field of cloud webserver management for the execution of the workflow and meeting the quality-of-service (QoS) prerequisites in a distributed cloud environment has been a challenging task. Though, internet of things (IoT) of work presented for the scheduling of the workflow in a heterogeneous cloud environment. Moreover, the rapid development in the field of cloud computing like edge-cloud computing creates new methods to schedule the workflow in a heterogenous cloud environment to process different tasks like IoT, event-driven applications, and different network applications. The current methods used for workflow scheduling have failed to provide better trade-offs to meet reliable performance with minimal delay. In this paper, a novel web server resource management framework is presented namely the reliable and efficient webserver management (REWM) framework for the edge-cloud environment. The experiment is conducted on complex bioinformatic workflows; the result shows the significant reduction of cost and energy by the proposed REWM in comparison with standard webserver management methodology.</span>

End-to-End Security for Distributed Event-driven Enclave Applications on Heterogeneous TEEs

This article presents an approach to provide strong assurance of the secure execution of distributed event-driven applications on shared infrastructures, while relying on a small Trusted Computing Base. We build upon and extend security primitives provided by Trusted Execution Environments (TEEs) to guarantee authenticity and integrity properties of applications, and to secure control of input and output devices. More specifically, we guarantee that if an output is produced by the application, it was allowed to be produced by the application’s source code based on an authentic trace of inputs. We present an integrated open-source framework to develop, deploy, and use such applications across heterogeneous TEEs. Beyond authenticity and integrity, our framework optionally provides confidentiality and a notion of availability, and facilitates software development at a high level of abstraction over the platform-specific TEE layer. We support event-driven programming to develop distributed enclave applications in Rust and C for heterogeneous TEE, including Intel SGX, ARM TrustZone, and Sancus. In this article we discuss the workings of our approach, the extensions we made to the Sancus processor, and the integration of our development model with commercial TEEs. Our evaluation of security and performance aspects show that TEEs, together with our programming model, form a basis for powerful security architectures for dependable systems in domains such as Industrial Control Systems and the Internet of Things, illustrating our framework’s unique suitability for a broad range of use cases which combine cloud processing, mobile and edge devices, and lightweight sensing and actuation.

CEPEDALoCo: An event-driven architecture for integrating complex event processing and blockchain through low-code

Internet of Things (IoT) is made up of millions of devices generating large amounts of heterogeneous data from multiple sources. These devices can be from multiple manufacturers which makes their use in terms of data acquisition, processing, analysis and actions on these data challenging. Additionally, all these data must be analyzed and correlated in real time for the early detection of situations of interest (complex events) and subsequent decision making. These complex events must be able to automatically trigger decisions and be stored in a secure, immutable and accessible way. In this context, Event-Driven Applications (EDAs) are a solution to meet these needs, however, developing such applications requires vast knowledge in certain technologies. To address these challenges, an EDA is proposed in this paper to integrate Complex Event Processing (CEP) and blockchain through the low-code paradigm. This proposal allows for the development of EDAs in a user-friendly way. These applications make it possible to integrate IoT devices from multiple manufacturers and with different data formats together with CEP technology for complex event detection and blockchain for secure, immutable and accessible event storage. To demonstrate the feasibility, the architecture was applied and evaluated in a case study related to measuring and acting on air quality using IoT devices that measure different pollutants and factors such as temperature, humidity and wind. The results show that the graphically designed EDAs facilitate the real-time analysis of the collected IoT data via a CEP engine, whose outcome is transparently and automatically registered in a blockchain network.

Mitigating Cold Start Problem in Serverless Computing: A Reinforcement Learning Approach

Serverless computing has revolutionized the world of cloud-based and event-driven applications with the introduction of Function as a Service (FaaS) as the latest cloud computing model. This computational model increases the level of abstraction from the infrastructure and breaks the program into small units called functions. Thus, it brings benefits, such as ease of development, saving resources, and reducing product launch time for enterprises and developers. Thanks to the scale-to-zero feature of this computational model, idle functions with no traffic will be depreciated from memory. However, this cost-saving approach adversely impacts delay leading to the cold start problem. Unfortunately, the existing solutions to alleviate the cold start delay are not resource efficient as they follow a fixed policy over time. Thereby, this article proposes a novel two-layer adaptive approach to tackle this issue. The first layer utilizes a holistic reinforcement learning algorithm to discover the function invocation patterns over time for determining the best time to keep the containers warm. The second layer is designed based on a long short-term memory (LSTM) to predict the function invocation times in the future to determine the required prewarmed containers. The experimental results on the Openwhisk platform show that the proposed approach reduces the memory consumption by 12.73% and improves the execution invocations on prewarmed containers by 22.65% compared to the Openwhisk platform.

NemoTrainer: Automated Conditioning for Stimulus-Directed Navigation and Decision Making in Free-Swimming Zebrafish.

Current methods for associative conditioning in animals involve human intervention that is labor intensive, stressful to animals, and introduces experimenter bias in the data. Here, we describe a simple apparatus and a flexible, microcontroller-based conditioning paradigm that minimizes human intervention. Our methodology exploits directed movement towards a target that depends on spatial working memory, including processing of sensory inputs, motivational drive, and attentional mechanisms. Within a stimulus-driven conditioning paradigm designed to train zebrafish, we present a localized pulse of light via LEDs and/or sounds via an underwater transducer. A webcam placed below a glass tank records fish-swimming behavior. For classical conditioning, animals simply associate a sound or light with an unconditioned stimulus, such as a small food reward presented at a fixed location, and swim towards that location to obtain a few grains of food dispensed automatically via a sensor-triggered, stepper motor. During operant conditioning, a fish must first approach a proximity sensor at a remote location and then swim to the reward location. For both types of conditioning, a timing-gated interrupt activates stepper motors via custom software embedded within a microcontroller (Arduino). "Ardulink", a Java facility, implements Arduino-computer communication protocols. In this way, a Java-based user interface running on a host computer can provide full experimental control. Alternatively, a similar level of control is achieved via an Arduino script communicating with an event-driven application controller running on the host computer. Either approach can enable precise, multi-day scheduling of training, including timing, location, and intensity of stimulus parameters; and the feeder. Learning can be tracked by monitoring turning, location, response times, and directional swimming of individual fish. This facilitates the comparison of performance within and across a cohort of animals. Our scheduling and control software and apparatus ("NemoTrainer") can be used to study multiple aspects of species-specific behaviors as well as the effects on them of various interventions.

Towards an Accessible Platform for Multimodal Extended Reality Smart Environments

This article presents the DEMOS prototype platform for creating and exploring multimodal extended-reality smart environments. Modular distributed event-driven applications are created with the help of visual codeless design tools for configuring and linking processing nodes in an oriented dataflow graph. We tested the conceptual logical templates by building two applications that tackle driver arousal state for safety and enhanced museum experiences for cultural purposes, and later by evaluating programmer and nonprogrammer students’ ability to use the design logic. The applications involve formula-based and decision-based processing of data coming from smart sensors, web services, and libraries. Interaction patterns within the distributed event-driven applications use elements of mixed reality and the Internet of Things, creating an intelligent environment based on near-field communication-triggering points. We discuss the platform as a solution to bridging the digital divide, analyzing novel technologies that support the development of a sustainable digital ecosystem.

EDALoCo: Enhancing the accessibility of blockchains through a low-code approach to the development of event-driven applications for smart contract management

Blockchain is a cutting-edge technology based on a distributed, secure and immutable ledger that facilitates the registration of transactions and the traceability of tangible and intangible assets without requiring central governance. The agreements between the nodes participating in a blockchain network are defined through smart contracts. However, the compilation, deployment, interaction and monitoring of these smart contracts is a barrier compromising the accessibility of blockchains by non-expert developers. To address this challenge, in this paper, we propose a low-code approach, called EDALoCo, that facilitates the development of event-driven applications for smart contract management. These applications make blockchain more accessible for software developers who are non-experts in this technology as these can be modeled through graphical flows, which specify the communications between data producers, data processors and data consumers. Specifically, we have enhanced the open-source Node-RED low-code platform with blockchain technology, giving support for the creation of user-friendly and lightweight event-driven applications that can compile and deploy smart contracts in a particular blockchain. Additionally, this platform extension allows users to interact with and monitor the smart contracts already deployed in a blockchain network, hiding the implementation details from non-experts in blockchain. This approach was successfully applied to a case study of COVID-19 vaccines to monitor and obtain the temperatures to which these vaccines are continuously exposed, to process them and then to store them in a blockchain network with the aim of making them immutable and traceable to any user. As a conclusion, our approach enables the integration of blockchain with the low-code paradigm, simplifying the development of lightweight event-driven applications for smart contract management. The approach comprises a novel open-source solution that makes data security, immutability and traceability more accessible to software developers who are non-blockchain experts.

Topology-Level Reactivity in Distributed Reactive Programs

Reactive programming is a popular paradigm to program event-driven applications, and it is often proposed as a paradigm to write distributed applications. One such type of application is *prosumer* applications, which are distributed applications that both produce and consume many events. We analyse the problems that occur when using a reactive programming language or framework to implement prosumer applications. We find that the assumption of an open network, which means prosumers of various types spontaneously join and leave the network, can cause a lot of code complexity or run-time inefficiency. At the basis of these issues lies *acquaintance management*: the ability to discover prosumers as they join and leave the network, and correctly maintaining this state throughout the reactive program. Most existing reactive programming languages and frameworks have limited support for managing acquaintances, resulting in accidental complexity of the code or inefficient computations. In this paper we present acquaintance management for reactive programs. First, we design an *acquaintance discovery* mechanism to create a *flock* that automatically discovers prosumers on the network. An important aspect of flocks is their integration with reactive programs, such that a reactive program can correctly and efficiently maintain its state. To this end we design an *acquaintance maintenance* mechanism: a new type of operator for functional reactive programming languages that we call `deploy-*`. The `deploy-*` operator enables correct and efficient reactions to time-varying collections of discovered prosumers. The proposed mechanisms are implemented in a reactive programming language called Stella, which serves as a linguistic vehicle to demonstrate the ideas of our approach. Our implementation of acquaintance management results in computationally efficient and idiomatic reactive code. We evaluate our approach quantitatively via benchmarks that show that our implementation is efficient: computations will efficiently update whenever a new prosumer is discovered, or a connected prosumer is dropped. To evaluate the distributed capabilities of our prototype implementation, we implement a use-case that simulates the bike-sharing infrastructure of Brussels, and we run it on a Raspberry Pi cluster computer. We consider our work to be an important step to use functional reactive programming to build distributed systems for open networks, in other words, distributed reactive programs that involve many prosumer devices and sensors that spontaneously join and leave the network.

Investigating the Suitability of Different Routing Strategies In Specific VANET Application Class

Vehicular Adhoc Network (VANET), is an emerging technology that holds the opportunity to create potential applications that directly impact peoples' lives, traffic management, and infotainment services. Understanding VANET applications and the available routing protocols can help to infer the most suitable protocols that satisfy VANET application requirements. This paper develops a systematic classification methodology to classify VANET applications from a routing perspective, each application class has different network requirements which are laid down by VANET Projects conducted in different countries. Some of these requirements are related to the routing aspects and need to be satisfied by the selected routing strategies (proactive and reactive). The paper identifies routing strategies performance metrics related to each application class requirement, to efficiently guide the development of these routing strategies towards guaranteeing satisfactory performance for the applications under a wide variety of realistic VANET scenarios. It is also worth mentioning that minimum delay is a requirement needed by time and event-driven application classes. However, high reliability is a requirement needed by on-demand applications. The paper aims to provide a comparative study on the performance of routing strategies in different VANET application classes, to identify which routing strategies have better performance in specific VANET applications class. End-to-end delay is employed as a performance metric to evaluate the short delay requirement, while, the Routing Overhead (RO) is used to assess the reliability requirement. Simulation results showed that proactive routing protocol has a lower delay, which means that it is suitable for delay-sensitive applications such as time-driven and event-driven applications. The result also showed that the reactive routing protocol outperforms the proactive routing protocol in terms of RO, which means that reactive routing protocols can be nominated as proper routing strategies to satisfy the reliability requirement of the On-demand driven applications.

THE PLATFORM FOR CREATION OF EVENT-DRIVEN APPLICATIONS BASED ON WOLFRAM MATHEMATICA AND APACHE KAFKA

The article is devoted to the study and development of the mechanism of interaction between Wolfram Mathematica programs and Apache Kafka queue to provide the ability to build event-driven applications based on it. The subject of the research is the practical principles of building a mechanism for interaction between Wolfram Mathematica and Apache Kafka through a proxy-server. The purpose of the article is to develop and substantiate practical recommendations regarding the formation of proxy-server and a mechanism for its work to publishing messages to the Apache Kafka queue and reading messages from it for programs of the mathematical processor Wolfram Mathematica, which will make it possible to build event-driven applications. The tasks are: to determine the mechanism of such interaction, prove the choice of tools for its implementation, create and test the obtained results. The research used the following tools: Apache Kafka, Kafkacat, servers Ubuntu 20 LTS, the method of developing the Wolfram Mathematica package. The results of the research: the mechanism of interaction between Wolfram Mathematica and Apache Kafka through a proxy-server was determined and the corresponding toolkit was created on its basis in the form of two Mathematica packages, which are built on using bash-scripts, Apache Kafka and third-party Kafkacat software. The first - for use on the end user's computer, the second – on a compute server with a remote Mathematica kernel. It is confirmed that the Mathematica processor is currently not suitable in its pure form for real-time data analysis. Conclusions. Practical recommendations have been developed and substantiated regarding the formation of the mechanism of interaction between the Wolfram Mathematica mathematical processor and the Apache Kafka queue manager through a proxy-server for the possibility of working in two directions with the queue: publishing messages and reading them. A toolkit for such interaction in the form of Mathematica packages has been created, their capabilities have been demonstrated. The economic benefit of using the described tools is shown. Future ways of its improvement are given.

THE MECHANISM FOR CREATION OF EVENT-DRIVEN APPLICATIONS BASED ON WOLFRAM MATHEMATICA AND APACHE KAFKA

The article is devoted to the research and development of the mechanism of interaction between Wolfram Mathematica programs and Apache Kafka queue to provide the ability to build event-driven applications based on it. The subject of the research is the practical principles of building a mechanism for interaction between Wolfram Mathematica and Apache Kafka. The purpose of the article is to develop and substantiate practical recommendations regarding the formation of a mechanism for publishing messages to the Apache Kafka queue and reading messages from it for programs of the mathematical processor Wolfram Mathematica, which will make it possible to build event-driven applications. Tasks: to determine the mechanism of such interaction, prove the choice of tools for its implementation, create and test the obtained results. The research used the following tools: Apache Kafka, Kafkacat, the method of developing the Wolfram Mathematica package. The results of the research: the mechanism of interaction between Wolfram Mathematica and Apache Kafka was determined and the corresponding toolkit was created on its basis in the form of two Mathematica packages, which are built on using Apache Kafka as a queue client and third-party Kafkacat software, respectively. It is shown that the first option is less reliable and consumes much more computer resources during operation. It has been demonstrated that the Mathematica processor is currently not suitable in its pure form for real-time data analysis. Recommendations are given regarding the use of built-in compilation functions to increase the speed of such processing. Conclusions. Practical recommendations have been developed and substantiated regarding the formation of the mechanism of interaction between the Wolfram Mathematica mathematical processor and the Apache Kafka queue manager for the possibility of working in two directions with the queue: publishing messages and reading them. A toolkit for such interaction in the form of Mathematica packages has been created, their capabilities have been demonstrated, as well as comparison with each other. The economic benefit of using the described tools is shown. Future ways of its improvement are given.

Serverless Event-driven Applications Development Tools and Techniques

Serverless, a new cloud-based architecture, brings development and deployment flexibility to a new level by significantly decreasing the size of the deployment units. Nevertheless, it still hasn’t been clearly defined for which applications it should be employed and how to use it most effectively, and this is the focus of this research.The study uses Microsoft Azure Functions – one of the popular mature tools – because of its stateful orchestrators – Durable Functions. The tool is used to present and describe four flexible serverless patterns with code examples.The first pattern is HTTP nanoservices. The example demonstrates how flexible can be the Function-as-a-Service model, which uses relatively small functions as deployment units.The second usage scenario described is a small logic layer between a few other cloud services. Thanks to its event-driver nature, serverless is well-suited for such tasks as making an action in one service after a specific event from another one. New functions easily integrate with the API from the first example.The third scenario – distributed computing – relies on the ability of Durable Functions to launch a myriad of functions in parallel and then aggregate their results. and distributed computing. A custom MapReduce implementation is presented in this section.The last pattern described in this research significantly simplifies concurrent working with mutable data by implementing the actor model. Durable Entities guarantee that messages are delivered reliably and in order, and also the absence of deadlocks.The results of this work can be used as a practical guide to serverless main concepts and usage scenarios. Main topic of future research was chosen to be the development of a full-fledged serverless application using typical patterns to study the architecture in more depth.Manuscript received 08.06.2020

An extended ACO-based mobile sink path determination in wireless sensor networks

In wireless sensor networks (WSNs), a mobile sink accumulate the data instead of routing directly to the sink to avoid the hotspot problem. In this process, it traverses a predetermined path by visiting a set of nodes called the rendezvous point (RP), and all the non-rendezvous points can transmit their data to the closest RP. Identifying the best collection of RPs and determining the mobile sink traveling path will decrease data loss and improve network performance. However, choosing a set of RPs and the route between them is a challenging task. It is more complicated in the event-driven applications due to the uneven data rate of SNs. In this context, we propose an extended ant colony optimization (ACO)-based mobile sink path construction for event-driven WSNs. In this, the best set of the RPs and the efficient mobile sink traveling path between them is determined. In addition to this, the RPs re-selection mechanism also adopted for balancing the energy between the nodes. After that, the virtual RPs are introduced to minimize the data transmissions between the sensor nodes and RPs. This process will improve WSNs’ performance in terms of reducing data losses while increasing network lifetime. The improved performance of the extended ACO-MSPD over existing is confirmed through simulation tests.

Energy-efficient data retention in D flip-flops using STT-MTJ

PurposeEmerging event-driven applications such as the internet-of-things requires an ultra-low power operation to prolong battery life. Shutting down non-functional block during standby mode is an efficient way to save power. However, it results in a loss of system state, and a considerable amount of energy is required to restore the system state. Conventional state retentive flip-flops have an “Always ON” circuitry, which results in large leakage power consumption, especially during long standby periods. Therefore, this paper aims to explore the emerging non-volatile memory element spin transfer torque-magnetic tunnel junction (STT-MTJ) as one the prospective candidate to obtain a low-power solution to state retention.Design/methodology/approachThe conventional D flip-flop is modified by using STT-MTJ to incorporate non-volatility in slave latch. Two novel designs are proposed in this paper, which can store the data of a flip-flip into the MTJs before power off and restores after power on to resume the operation from pre-standby state.FindingsA comparison of the proposed design with the conventional state retentive flip-flop shows 100 per cent reduction in leakage power during standby mode with 66-69 per cent active power and 55-64 per cent delay overhead. Also, a comparison with existing MTJ-based non-volatile flip-flop shows a reduction in energy consumption and area overhead. Furthermore, use of a fully depleted-silicon on insulator and fin field-effect transistor substituting a complementary metal oxide semiconductor results in 70-80 per cent reduction in the total power consumption.Originality/valueTwo novel state-retentive D flip-flops using STT-MTJ are proposed in this paper, which aims to obtain zero leakage power during standby mode.

Broadcast and Reliable Coverage based Efficient Recursive Routing in Large-Scale WSNs

This paper presents a Broadcast and Reliable Coverage based Efficient Recursive Routing (BRCER) in Large-Scale Wireless Sensor Networks for Event-Driven Applications with Mobile Sinks (MSs). For data routing in the network, the proposed protocol updates the connectivity list by using local broadcast and multicast announcement. Instead of broadcasting in the whole network, the local broadcasting is used to save sensor node energy which maximizes the battery lifetime of sensors. However, the multicast announcement provides the appropriate path whenever the event activities are performed by MSs. In addition, reliable communication connectivity among sensor nodes, which depends on the surrounding and the deployment of sensor nodes, is also our prime concern for better quality of service (QoS). The main focus of BRCER protocol is to timely route the data with proper load distribution in the network with reliable connectivity. Theoretical concept and simulation results further validate that BRCER protocol can withstand in worse environmental conditions with effective network performance.

Simplifying CPS Application Development through Fine-grained, Automatic Timeout Predictions

Application development for Cyber Physical Systems (CPS) is challenging, because the wireless network and the devices introduce latencies that vary continuously along with the load, status, or environmental conditions of the infrastructure. Reactive programming is well suited for the development of event-driven applications, yet current reactive programming frameworks require developers to predict event arrival time-boundaries at compile time, which is impractical, if not impossible, for CPS. Thus, there is a tradeoff between timeliness and completeness of complex event computations, e.g., operational efficiency in a manufacturing plant: Waiting too long until all individual events arrive can fail to produce a useful result, while not waiting long enough may lead to faults due to incomplete status information. In this article, we propose (a) a set of extensions to state-of-the-art reactive programming frameworks, which remove the burden of specifying timeouts at compile time by utilizing (b) Khronos, a middleware that automatically determines timeouts by taking into account variations in event arrival times due to the underlying infrastructure. Evaluation on a physical testbed shows that the extensions significantly decrease developer effort and that Khronos considerably improves timeliness under varying network configurations and conditions, while still satisfying the application’s tolerance to missed events.