Stream Analytics Research Articles

The Development of Simulation and Optimisation Tools with an Intuitive User Interface to Improve the Operation of Electric Arc Furnaces

The paper presents a novel decision support system designed to improve the efficiency and effectiveness of decision-making for electric arc furnace (EAF) operators. The system integrates two primary tools: the EAF Simulator, which is based on advanced mechanistic models, and the EAF Optimiser, which uses data-driven models trained on historical data. These tools enable the simulation and optimisation of furnace settings in real time and provide operators with important insights. A key objective was to develop a user-friendly interface with the Siemens Insights Hub Cloud Service and Node-RED that enables interactive management and support. The interface allows operators to analyse and compare past and simulated batches by adjusting the input data and parameters, resulting in improved optimisation and reduced costs. In addition, the system focuses on the collection and pre-processing of input data for the simulator and optimiser and uses Message Queuing Telemetry Transport (MQTT)communication between the user interfaces and models to ensure seamless data exchange. The EAF Simulator uses a comprehensive mathematical model to simulate the complex dynamics of heat and mass transfer, while the EAF Optimiser uses a fuzzy logic-based approach to predict optimal energy consumption. The integration with Siemens Edge Streaming Analytics ensures robust data collection and real-time responsiveness. The dual-interface design improves user accessibility and operational flexibility. This system has significant potential to reduce energy consumption by up to 10% and melting times by up to 15%, improving the efficiency and sustainability of the entire process.

IoT-based Machine Learning Comparative Models of Stream Water Parameters Forecasting for Freshwater Lobster

Water quality parameters such as dissolved oxygen, pH, and mineral content are important factors for aquaculture. Predictive analytics can predict water conditions in aquaculture and significantly reduce the mortality probability of aquaculture products. This paper applied stream predictive analytics to the freshwater lobster farming dataset where its real-time data supplied by End Node Unit (ENU) which integrated with dissolved oxygen (DO), potential hydrogen (pH), electrical conductivity (EC), and total dissolved solids (TDS). The real-time data of ENU in Structured Query Language (SQL) is normally displayed for remote monitoring and the analytics will only be done after in different processing platform called batch analytics. Instead of batch, this paper demonstrates capability of stream analytics where the real-time data query from ENU streaming through Structured Query Language (SQL) right into R Studio and Autoregressive Integrated Moving Average (ARIMA) predictions executed on the query table simultaneously on the same processing platform. Previously, ARIMA, Neural Network Autoregressive (NNETAR), and Naïve Bayes, were run and evaluated in R Studio to identify the best algorithm for stream analytics. Prediction procedure in R studio start with importing real-time data stored in SQL database and stream into R Studio using command of “dbGetQuery(con,sql)”. These three models evaluated the performance of freshwater lobster water conditions, dissolved oxygen (DO), potential hydrogen (pH), electrical conductivity (EC), and total dissolved solids (TDS). The data was collected for six months, and 70% was used as training data and 30% as test data. Compared to NNETAR and Naïve Bayes, ARIMA fits the entire data set well for 7 days; the ARIMA model exhibited lower absolute errors for pH and electrical conductivity, with errors ranging from 0.04 to 1.7 across days, while the NNETAR model had generally lower errors for TDS, with errors ranging from 0.3 to 0.7; however, the Naïve Bayes model's performance varied, with the lowest error for DO on day (5) 0.15 but higher errors for other parameters and days, including the highest error for electrical conductivity on day (6) 6.2. In conclusion, the average absolute errors for DO, pH, EC, and TDS are 0.163, 0.064, 0.705, and 0.498, respectively. Our findings underscore the efficacy of ARIMA for comprehensive water quality via stream prediction while highlighting the nuanced strengths and weaknesses of each model in forecasting specific parameters. This study contributes to the aquaculture literature by providing a nuanced comparative analysis of predictive models tailored to freshwater lobster farming, emphasizing the imperative role of stream predictive modelling. It enables real-time monitoring of water quality parameters, ensuring prompt interventions to maintain optimal conditions, thereby minimizing risks, enhancing aquaculture productivity, and ultimately contributing to sustainable and efficient freshwater lobster farming practices.

Integrating Azure Services for Real Time Data Analytics and Big Data Processing

Integrating Azure services for real-time data analytics and big data processing is a transformative approach that leverages the power of cloud computing to handle vast amounts of data at scale. Azure offers a wide array of services such as Azure Synapse Analytics, Azure Data Lake, Azure Stream Analytics, and Azure Databricks, which together enable businesses to extract valuable insights from both structured and unstructured data in real time. These services provide seamless integration, allowing for real-time data ingestion, transformation, and analysis, facilitating quicker decision-making processes. Azure’s capabilities in big data processing allow for scalable and efficient storage of data, ensuring optimal performance even as data volumes grow. By utilizing tools like Azure Data Factory and Azure HDInsight, organizations can automate complex workflows and apply machine learning algorithms for predictive analytics, driving innovation and competitive advantage. The integration of Azure's real-time analytics services also supports the use of advanced technologies like artificial intelligence (AI) and Internet of Things (IoT), enabling companies to monitor live streams of data from various devices and platforms. This infrastructure not only enhances operational efficiency but also ensures a more proactive approach to problem-solving by identifying trends and anomalies as they occur. In conclusion, Azure’s comprehensive suite of services offers a robust platform for managing and analysing big data in real time, empowering organizations to leverage data for better decision-making and improved business outcomes.

PECJ: Stream Window Join on Disorder Data Streams with Proactive Error Compensation

Stream Window Join (SWJ), a vital operation in stream analytics, struggles with achieving a balance between accuracy and latency due to out-of-order data arrivals. Existing methods predominantly rely on adaptive buffering, but often fall short in performance, thereby constraining practical applications. We introduce PECJ, a solution that proactively incorporates unobserved data to enhance accuracy while reducing latency, thus requiring robust predictive modeling of stream oscillation. At the heart of PECJ lies a mathematical formulation of the posterior distribution approximation (PDA) problem using variational inference (VI). This approach circumvents error propagation while meeting the low-latency demands of SWJ. We detail the implementation of PECJ, striking a balance between complexity and generality, and discuss both analytical and learning-based approaches. Experimental evaluations reveal PECJ's superior performance. The successful integration of PECJ into a multi-threaded SWJ benchmark testbed further establishes its practical value, demonstrating promising advancements in enhancing data stream processing capabilities amidst out-of-order data.

Enhancing Online Food Delivery Systems through Comprehensive Text Analytics and Strategic Data Integration

Addressing challenges in the online food delivery system involves employing various data analytics techniques. Text Analytics, encompassing web analytics, social media analytics, stream analytics, and geospatial analytics, plays a pivotal role in managing and extracting valuable insights. The use of third-party systems by many companies to meet the demand for online food delivery presents issues related to control. Furthermore, information overload and poorly organized data contribute to observed problems. This research proposes effective data integration as a solution, facilitating strategic analytics for optimal system performance. Proper data sorting enables adaptive planning and priority shifts tailored to customer satisfaction. The framework of data integration is crucial in illustrating the comprehensive analysis of online food delivery systems. The report also delves into the challenges associated with implementing text analytics.

Enabling Streaming Analytics in Satellite Edge Computing via Timely Evaluation of Big Data Queries

Enabling Streaming Analytics in Satellite Edge Computing via Timely Evaluation of Big Data Queries

OmniSketch: Efficient Multi-Dimensional High-Velocity Stream Analytics with Arbitrary Predicates

A key need in different disciplines is to perform analytics over fast-paced data streams, similar in nature to the traditional OLAP analytics in relational databases - i.e., with filters and aggregates. Storing unbounded streams, however, is not a realistic, or desired approach due to the high storage requirements, and the delays introduced when storing massive data. Accordingly, many synopses/sketches have been proposed that can summarize the stream in small memory (usually sufficiently small to be stored in RAM), such that aggregate queries can be efficiently approximated, without storing the full stream. However, past synopses predominantly focus on summarizing single-attribute streams, and cannot handle filters and constraints on arbitrary subsets of multiple attributes efficiently. In this work, we propose OmniSketch, the first sketch that scales to fast-paced and complex data streams (with many attributes), and supports count aggregates with filters on multiple attributes, dynamically chosen at query time. The sketch offers probabilistic guarantees, a favorable space-accuracy tradeoff, and a worst-case logarithmic complexity for updating and for query execution. We demonstrate experimentally with both real and synthetic data that the sketch outperforms the state-of-the-art, and that it can approximate complex ad-hoc queries within the configured accuracy guarantees, with small memory requirements.

YOLO and LSH-based video stream analytics landscape for short-term traffic density surveillance at road networks

YOLO and LSH-based video stream analytics landscape for short-term traffic density surveillance at road networks

AMNES: Accelerating the Computation of Data Correlation Using FPGAs

A widely used approach to characterize input data in both databases and ML is computing the correlation between attributes. The operation is supported by all major database engines and ML platforms. However, it is an expensive operation as the number of attributes involved grows. To address the issue, in this paper we introduce AMNES, a stream analytics system offloading the correlation operator into an FPGA-based network interface card. AMNES processes data at network line rate and the design can be used in combination with smart storage or SmartNICs to implement near data or in-network data processing. AMNES design goes beyond matrix multiplication and offers a customized solution for correlation computation bypassing the CPU. Our experiments show that AMNES can sustain streams arriving at 100 Gbps over an RDMA network, while requiring only ten milliseconds to compute the correlation coefficients among 64 streams, an order of magnitude better than competing CPU or GPU designs.

Performance analysis of artificial lift systems deployed in natural gas wells: A time-series analytics approach

In Coal Seam Gas (CSG) production, Artificial Lift (AL) systems comprising various downhole pumps produce coal-fine-laden water. Over time, the accumulation of coal fines results in mechanical failures of such pumps, which leads to a loss in natural gas production. This problem is exacerbated by CSG operators having to manage thousands of wells in their day-to-day operations. The reason for having such an exuberant number of natural gas wells is due to the fact that CSG reservoirs have a short production cycle when compared to conventional reservoirs. Therefore, thousands of CSG wells must operate simultaneously to sustain natural gas volumes that satisfy domestic energy demand, and help meet contractual export obligations.To manage a large fleet of CSG wells, operators rely on Petroleum and Surveillance Engineers to mitigate production issues and advise timely corrective actions. Current methods of monitoring CSG wells involve observing real-time trends and alarms from Supervisory Control and Data Acquisition (SCADA) systems. However, managing a large fleet of wells with a monitoring-only approach can be detrimental to AL operations. Methods such as Exception Based Surveillance have been used to improve engineers' workload, but such methods are informative at best. In addition, they do not provide early indications of changed downhole pump performance. To improve how AL systems are monitored in CSG operation, we propose a novel well surveillance method that can help mitigate pump failure and provide engineers with insightful information to take corrective actions.In this work, we will present an innovative time-series analytics approach that examines the performance of AL systems in near real-time and helps Well Surveillance and Production Engineers make timely decisions that aid in mitigating pump failures. We used time-series data from 448 CSG wells to build the streaming analytics methodology that autonomously detects various performance states of downhole pumps during CSG production. Furthermore, we successfully detected the onset of AL systems failures, such as solids build-up, gas intake, high torque and pump degradation. We validated our solution on a separate set of 428 wells and were able to show reliable detection of detrimental events on live data from producing natural gas wells. This solution is currently deployed by 2 CSG operators where real-time notifications aid Petroleum and Well Surveillance Engineers with proactively managing AL systems across multiple CSG assets.

Real-Time Streaming Analytics using Big Data Paradigm and Predictive Modelling based on Deep Learning

With the evolution of distributed streaming platforms analysing humongous time series data, which is streamed continuously from IoT devices become lot easier. In most of the IoT networks the data are in motion or in data centre/cloud. It is possible to process this data in real time similar to edge devices using the big data framework. In data intensive applications predictive analytics require more resources to perform complex computations. Apache Flink framework is capable of performing real time streaming of schema less data and scales very high in distributed environment with low latency, it is used to collect and store the data in the cloud. This work suggests a suitable environment to collect, transport, preprocess and aggregate the data stream to perform predictive analytics using deep learning models. Deep learning automatically extracts features and builds models after training, it has the potential to solve problems that can't be solved by conventional machine learning models. Therefore, the use of algorithms based on deep learning is recommended for forecasting temporal data. Also, we discuss a number of different deep learning forecasting models and analyse the performance of different deep learning forecasting models in order to determine which one is the effective model for single step, multi step and multi variant methods based on error functions with respect to streamed sensor data.

RTCoInfer: Real-Time Collaborative CNN Inference for Stream Analytics on Ubiquitous Images

Emerging intelligent applications based on accurate and timely stream analytics require real-time CNN inference of massive data continuously generated at the pervasive end devices. Due to the resource constraints, neither computing locally at end devices nor transmitting to remote servers is competent for computation-intensive CNN inference on large-volume images in real-time. Therefore, Collaborative Inference (CI), which conducts inference sequentially from the local device to the remote server with compressed intermediate inference data, is rapidly promoted. Due to the essential communication in collaboration, the CI efficiency is sensitive to network conditions, and will degrade under the unpredictable network fluctuations in practice, which may cause a severe delay in CI and degrade the responsiveness of stream analytics. For accurate and timely stream analytics in practical fluctuating networks, we present RTCoInfer, the real-time CI framework with run-time transmission adaption considering the network conditions. Specifically, we propose a novel Switchable CNN integrating CNNs with different compression rates on the partition layer for the run-time transmission adjustment, and construct a real-time controller determining the compression rate to maintain the real-time CI for stream analytics. Extensive experiments show that, compared with state-of-the-art methods, RTCoInfer achieves better efficiency and unprecedented resilience in real-time stream analytics.

OpenStackDP: a scalable network security framework for SDN-based OpenStack cloud infrastructure

Network Intrusion Detection Systems (NIDS) and firewalls are the de facto solutions in the modern cloud to detect cyberattacks and minimize potential hazards for tenant networks. Most of the existing firewalls, perimeter security, and middlebox solutions are built on static rules/signatures or simple rule matching, making them inflexible, susceptible to bugs, and difficult to introduce new services. This paper aims to improve network management in OpenStack Clouds by taking advantage of the combination of software-defined networking (SDN), Network Function Virtualization (NFV), and machine learning/artificial intelligence (ML/AI) and for making networks more predictable, reliable, and secure. Artificial intelligence is being used to monitor the behavior of the virtual machines and applications running in the OpenStack SDN cloud so that when any issues or degradations are noticed, the decision can be quickly made on how to handle that issue, being able to analyze data in motion, starting at the edge. The OpenStackDP framework comprises lightweight monitoring, anomaly-detecting intelligent sensors embedded in the data plane, a threat analytics engine based on ML/AI algorithms running inside switch hardware/network co-processor, and defensive actions deployed as virtual network functions (VNFs). This network data plane-based architecture makes high-speed threat detection and rapid response possible and enables a much higher degree of security. We have built the framework with advanced streaming analytics technologies, algorithms, and machine learning to draw knowledge from this data that is in motion before the malicious traffic goes to the tenant compute nodes or long-term data store. Cloud providers and users will benefit from improved Quality-of-Services (QoS) and faster recovery from cyber-attacks and compromised switches. The multi-phase collaborative anomaly detection scheme demonstrates an accuracy of 99.81%, average latencies of 0.27 ms, and response speed within 9 s. The simulations and analysis show that the OpenStackDP network analytics framework substantially secures and outperforms prior SDN-based OpenStack solutions for Cloud architectures.

AEDS-IoT: Adaptive clustering-based Event Detection Scheme for IoT data streams

The emergence of the Internet of Things is expected to produce a huge amount of data streams for sensing the phenomenon from the real world. These data patterns i.e. data distributions keep on evolving and are subject to change based on the changes in the external environment. Valuable and useful information should be extracted from this data stream in real-time- to find out the events of interest. Pattern extraction from stored data and its analytics is convenient, though stream analytics has major challenges associated with it like identifying the event patterns from limited data in the stream, concept drift detection, outlier detection, and handling, limited storage, and processing capacity etc. Traditional methods, in general, work well with historical data yet have limitations while dealing with real-time data streams. Hence, the present work proposes adaptive clustering-based Event Detection Scheme for IoT (AEDS-IoT) stream data, which can infer events of interest from data distribution patterns present in streaming data. Initially, the scheme uses an adaptive window to collect the streaming data and processes it to detect the events. Further, the event patterns are detected by adaptive clustering. After identifying event patterns, the user assigns labels based on spatial–temporal conditions. The proposed AEDS methodology updates the patterns generated from upcoming stream data received. The performance of the proposed scheme is compared against the state of the scheme using the Arhus dataset. To corroborate the scheme with experimental evidence, the scheme was applied to the data collected from the Ganga river(the life line of North India) using smart water sensors. The performance and efficacy of AEDS-IoT were also tested on these data to detect upcoming event patterns from the data stream.

Sentiment Analysis Using Streaming Analytics for Customer 360 Decision Support Approach: A Case Study in Insurance Industry

Sentiment Analysis Using Streaming Analytics for Customer 360 Decision Support Approach: A Case Study in Insurance Industry

Performance Analysis of Hybrid RR Algorithm for Anomaly Detection in Streaming Data

Automated live video stream analytics has been extensively researched in recent times. Most of the traditional methods for video anomaly detection is supervised and use a single classifier to identify an anomaly in a frame. We propose a 3-stage ensemble-based unsupervised deep reinforcement algorithm with an underlying Long Short Term Memory (LSTM) based Recurrent Neural Network (RNN). In the first stage, an ensemble of LSTM-RNNs are deployed to generate the anomaly score. The second stage uses the least square method for optimal anomaly score generation. The third stage adopts award-based reinforcement learning to update the model. The proposed Hybrid Ensemble RR Model was tested on standard pedestrian datasets UCSDPed1, USDPed2. The data set has 70 videos in UCSD Ped1 and 28 videos in UCSD Ped2 with a total of 18560 frames. Since a real-time stream has strict memory constraints and storage issues, a simple computing machine does not suffice in performing analytics with stream data. Hence the proposed research is designed to work on a GPU (Graphics Processing Unit), TPU (Tensor Processing Unit) supported framework. As shown in the experimental results section, recorded observations on frame-level EER (Equal Error Rate) and AUC (Area Under Curve) showed a 9% reduction in EER in UCSD Ped1, a 13% reduction in ERR in UCSD Ped2 and a 4% improvement in accuracy in both datasets.

An edge–cloud integrated framework for flexible and dynamic stream analytics

With the popularity of Internet of Things (IoT), edge computing and cloud computing, more and more stream analytics applications are being developed including real-time trend prediction and object detection on top of IoT sensing data. One popular type of stream analytics is the recurrent neural network (RNN) deep learning model based time series or sequence data prediction and forecasting. Different from traditional analytics that assumes data are available ahead of time and will not change, stream analytics deals with data that are being generated continuously and data trend/distribution could change (a.k.a. concept drift), which will cause prediction/forecasting accuracy to drop over time. One other challenge is to find the best resource provisioning for stream analytics to achieve good overall latency. In this paper, we study how to best leverage edge and cloud resources to achieve better accuracy and latency for stream analytics using a type of RNN model called long short-term memory (LSTM). We propose a novel edge–cloud integrated framework for hybrid stream analytics that supports low latency inference on the edge and high capacity training on the cloud. To achieve flexible deployment, we study different approaches of deploying our hybrid learning framework including edge-centric, cloud-centric and edge–cloud integrated. Further, our hybrid learning framework can dynamically combine inference results from an LSTM model pre-trained based on historical data and another LSTM model re-trained periodically based on the most recent data. Using real-world and simulated stream datasets, our experiments show the proposed edge–cloud deployment is the best among all three deployment types in terms of latency. For accuracy, the experiments show our dynamic learning approach performs the best among all learning approaches for all three concept drift scenarios.

A comparative study on online machine learning techniques for network traffic streams analysis

Modern networks generate a massive amount of traffic data streams. Analyzing this data is essential for various purposes, such as network resources management and cyber-security analysis. There is an urgent need for data analytic methods that can perform network data processing in an online manner based on the arrival of new data. Online machine learning (OL) techniques promise to support such type of data analytics. In this paper, we investigate and compare the OL techniques that facilitate data stream analytics in the networking domain. We also investigate the importance of traffic data analytics and highlight the advantages of online learning in this regard, as well as the challenges associated with OL-based network traffic stream analysis, e.g., concept drift and the imbalanced classes. We review the data stream processing tools and frameworks that can be used to process such data online or on-the-fly along with their pros and cons, and their integrability with de facto data processing frameworks. To explore the performance of OL techniques, we conduct an empirical evaluation on the performance of different ensemble- and tree-based algorithms for network traffic classification. Finally, the open issues and the future directions in analyzing traffic data streams are presented. This technical study presents valuable insights and outlook for the network research community when dealing with the requirements and purposes of online data streams analytics and learning in the networking domain.

Interactive Graph Stream Analytics in Arkouda

Data from emerging applications, such as cybersecurity and social networking, can be abstracted as graphs whose edges are updated sequentially in the form of a stream. The challenging problem of interactive graph stream analytics is the quick response of the queries on terabyte and beyond graph stream data from end users. In this paper, a succinct and efficient double index data structure is designed to build the sketch of a graph stream to meet general queries. A single pass stream model, which includes general sketch building, distributed sketch based analysis algorithms and regression based approximation solution generation, is developed, and a typical graph algorithm—triangle counting—is implemented to evaluate the proposed method. Experimental results on power law and normal distribution graph streams show that our method can generate accurate results (mean relative error less than 4%) with a high performance. All our methods and code have been implemented in an open source framework, Arkouda, and are available from our GitHub repository, Bader-Research. This work provides the large and rapidly growing Python community with a powerful way to handle terabyte and beyond graph stream data using their laptops.

Ultra-Reliable and Low-Latency Computing in the Edge with Kubernetes

Novel applications will require extending traditional cloud computing infrastructure with compute resources deployed close to the end user. Edge and fog computing tightly integrated with carrier networks can fulfill this demand. The emphasis is on integration: the rigorous delay constraints, ensuring reliability on the distributed, remote compute nodes, and the sheer scale of the system altogether call for a powerful resource provisioning platform that offers the applications the best of the underlying infrastructure. We therefore propose Kubernetes-edge-scheduler that provides high reliability for applications in the edge, while provisioning less than 10% of resources for this purpose, and at the same time, it guarantees compliance with the latency requirements that end users expect. We present a novel topology clustering method that considers application latency requirements, and enables scheduling applications even on a worldwide scale of edge clusters. We demonstrate that in a potential use case, a distributed stream analytics application, our orchestration system can reduce the job completion time to 40% of the baseline provided by the default Kubernetes scheduler.