Real-time Data Distribution Research Articles

WATERHYPERNET: a prototype network of automated in situ measurements of hyperspectral water reflectance for satellite validation and water quality monitoring

This paper describes a prototype network of automated in situ measurements of hyperspectral water reflectance suitable for satellite validation and water quality monitoring. Radiometric validation of satellite-derived water reflectance is essential to ensure that only reliable data, e.g., for estimating water quality parameters such as chlorophyll a concentration, reach end-users. Analysis of the differences between satellite and in situ water reflectance measurements, particularly unmasked outliers, can provide recommendations on where satellite data processing algorithms need to be improved. In a massively multi-mission context, including Newspace constellations, hyperspectral missions and missions with broad spectral bands not designed for “water colour”, the advantage of hyperspectral over multispectral in situ measurements is clear. Two hyperspectral measurement systems, PANTHYR (based on the mature TRIOS/RAMSES radiometer) and HYPSTAR® (a newly designed radiometer), have been integrated here in the WATERHYPERNET network with SI-traceable calibration and characterisation. The systems have common data acquisition protocol, data processing and quality control. The choice of validation site and viewing geometry and installation considerations are described in detail. Three demonstration cases are described: 1. PANTHYR data from two sites are used to validate Sentinel-2/MSI (A&B); 2. HYPSTAR® data at six sites are used to validate Sentinel-3/OLCI (A&B); 3. PANTHYR and HYPSTAR® data in Belgian North Sea waters are used to monitor phytoplankton parameters, including Phaeocystis globosa, over two 5 month periods. Conclusion are drawn regarding the quality of Sentinel-2/MSI and Sentinel-3/OLCI data, including indications where improvements could be made. For example, a positive bias (mean difference) is found for ACOLITE_DSF processing of Sentinel-2 in clear waters (Acqua Alta) and clues are provided on how to improve this processing. The utility of these in situ measurements, even without accompanying hyperspectral satellite data, is demonstrated for phytoplankton monitoring. The future evolution of the WATERHYPERNET network is outlined, including geographical expansion, improvements to hardware reliability and to the measurement method (including uncertainty estimation) and plans for daily distribution of near real-time data.

OEM: An operation-aware event matching algorithm for content-based Pub/Sub systems

Due to the continuous expansion of the user base and the heightened demands for quality of service (QoS), real-time data distribution has encountered significant challenges in event matching within large-scale content-based publish/subscribe (CPS) systems. Many efficient algorithms have been proposed to enhance the matching performance. However, most existing algorithms scarcely consider operation characteristics, which often leads to performance degradation due to a multitude of repetitive operations, such as comparisons, additions, and assignments. In this paper, we propose an Operation-aware Event Matching algorithm called OEM, which takes into account the efficiency of CPU operations, SIMD instructions, and multi-thread architecture. First, we introduce a bitset-based subscription cache mechanism (SCM), which enables the efficient execution of bitwise OR operations during the matching process. Furthermore, we propose two optimizations, namely an optimal skewness-aware cell grouping (OSCG) strategy and an attribute-based clustering mechanism (ACM), to enhance the efficiency of OEM. These optimizations effectively address the challenges posed by data skewness, low cache efficiency, and high dimensionality. In addition, we establish a performance analysis model that characterizes the trade-off between memory consumption and performance improvement. We conducted extensive experiments to evaluate the performance of OEM. Compared to six state-of-the-art algorithms, namely REIN, Ada-REIN, TAMA, OpIndex, PEM and fgPEM, OEM achieves an average reduction in matching time by 40.4×, 38.5×, 12.0×, 33.1×, 27.8×, and 25.5×, respectively.

HydroRTC: A web-based data transfer and communication library for collaborative data processing and sharing in the hydrological domain

The exponential growth in data generated by satellites, radars, sensors, and analysis and reanalysis from model outputs for the hydrological domain requires efficient real-time data management and distribution mechanisms. This paper introduces HydroRTC, a web-based data transfer and communication library designed to accelerate large-scale data sharing and analysis. Leveraging next-generation web technologies like WebSockets, WebRTC and Node.js, the library enables seamless peer-to-peer sharing, smart data transmission, and large dataset streaming. Three primary scenarios are presented as use cases, demonstrating the potential of HydroRTC as server-to-peer with intelligent data scheduling and large data streaming, peer-to-peer data sharing, and peer-to-server for data exchange. HydroRTC offers a promising solution for collaborative infrastructures in the hydrological and environmental domain, allowing real-time and high-throughput data sharing and transfer for enhancing research efficiency and collaboration capabilities.

Performance Evaluation and Validation of Intelligent Security Mechanism in Software Defined Network

Network attacks are discovered using intrusion detection systems (IDS), one of the most crucial security solutions. Machine learning techniques-based intrusion detection approaches have been rapidly created as a result of the widespread use of standard machine learning algorithms in the security field. Unfortunately, as technology has advanced and there have been faults in the machine learning-based intrusion detection system, the system has consistently failed to fulfill the standards for cyber security. Generative adversarial networks (GANs) have drawn a lot of interest recently and have been utilized widely in anomaly detection due to their enormous capacity for learning difficult high-dimensional real time data distribution. Traditional machine learning algorithms for intrusion detection have a number of drawbacks that deep learning techniques can significantly mitigate. With the help of a real time dataset, this work suggests employing GANs and its variants to detect network intrusions in SDN. The feasibility and comparison results are also presented. For different kinds of datasets, the BiGAN outcomes outperform the GAN.

An enhanced self-learning-based clustering scheme for real-time traffic data distribution in wireless networks

The process of examining the data flow over the internet to identify abnormalities in wireless network performance is known as network traffic analysis. When analyzing network traffic data, traffic classification becomes an important task. The traffic data classification is used to determine whether data in network traffic is in real-time or not. This analysis controls network traffic data in a network and allows for efficient network performance improvement. Real-time and non-real-time data are effectively classified from the given input data set using data mining clustering and classification algorithms. The proposed work focuses on the performance of traffic data classification with high clustering accuracy and low Classification Time (CT). This research work is carried out to fill the gap in the existing network traffic classification algorithms. However, the traffic data classification remained unaddressed for performing the network traffic analysis effectively. Then, we proposed an Enhanced Self-Learning-based Clustering Scheme (ESLCS) using an enhanced unsupervised algorithm and adaptive seeding approach to improve the classification accuracy while performing the real-time traffic data distribution in wireless networks. Test-bed results demonstrate that the proposed model enhances the clustering accuracy and True Positive Rate (TPR) effectively as well as reduces the CT time and Communication Overhead (CO) substantially to compare with the peer-existing routing techniques.

A dynamic adaptive iterative clustered federated learning scheme

Clustered federated learning (CFL), as an important research branch of personalized federated learning (FL), can better cope with the highly statistically heterogeneous federated learning environment and provide higher quality services to clients. However, existing CFL schemes have difficulties in adapting to real-time data distribution changes due to disadvantages such as relatively fixed cluster structure. This poses a great challenge to the practical deployment of CFL schemes. To address the common problems of existing CFL schemes, we propose a more flexible dynamic adaptive cluster federated learning scheme (AICFL). AICFL uses the mutual sensitivity between models and data as intuition to perform cluster identity estimation, cluster addition, cluster model updating, and cluster deletion in the early iterations of FL to find the optimal client cluster partitioning. Firstly, this process does not require a priori estimation of the number of clusters and does not require the online participation of all clients. Secondly, during cluster partitioning, AICFL is able to adjust the cluster structure in real time based on the overall data distribution. Moreover, AICFL has the same ability to adapt to changes in the system environment in the middle and late stages of FL. The experimental results show that our scheme gives the most reasonable cluster partitioning results in all cases which indicates that AICFL is able to cope with the above-mentioned distribution changes well, and has better adaptability and better flexibility than other schemes.

The Evryscope Fast Transient Engine: Real-time Detection for Rapidly Evolving Transients

Astrophysical transients with rapid developments on subhour timescales are intrinsically rare. Due to their short durations, events like stellar superflares, optical flashes from gamma-ray bursts, and shock breakouts from young supernovae are difficult to identify on timescales that enable spectroscopic follow-up. This paper presents the Evryscope Fast Transient Engine (EFTE), a new data reduction pipeline that is designed to provide low-latency transient alerts from the Evryscopes—a north–south pair of ultra-wide-field telescopes with an instantaneous footprint covering 38% of the entire sky—and tools for building long-term light curves from Evryscope data. EFTE leverages the optical stability of the Evryscopes by using a simple direct image subtraction routine that is suited to continuously monitoring the transient sky at a cadence of a minute. Candidates are produced within the base Evryscope 2 minute cadence for 98.5% of images, and internally filtered using vetnet, a convolutional neural network real–bogus classifier. EFTE provides an extensible and robust architecture for transient surveys probing similar timescales, and serves as the software test bed for the real-time analysis pipelines and public data distribution systems for the Argus Array, a next-generation all-sky observatory with a data rate 62 times higher than that of Evryscope.

Toward a Novel RESTFUL Big Data-Based Urban Traffic Incident Data Web Service for Connected Vehicles

Abstract Connected vehicles (CVs) are an emerging technology in intelligent transportation systems. Currently, many data-driven intelligent transportation systems (D2ITS) use CV data. Unfortunately, these D2ITS still need serious improvement before they meet higher-level visualization needs. Thus, we aim to develop a new, intelligent data-driven transportation system framework. We focus on visualizing real-time CV data using a big data analytic system in urban areas. In response, we first propose an effective real-time data distribution approach within the Vehicular Ad-hoc NETwork. Second, we develop novel strategies for aggregating, extracting and ingesting data. We provide scalable and fault-tolerant delivery methods without interruption or delay. Finally, we proposed a novel visualization REpresentational State Transfer (REST) web service. We used Simulation of Urban MObility, OMNET++ and Veins to simulate a traffic incident dataset. Then, we tested the Basic Safety Messages in an experimental big data cluster. We used NIFI, Kafka and Cassandra for ingestion, distribution, delivery and storage. The results show accurate performance for packet loss, packet delivery and communication delay. Also, it indicates high throughput and low latency for distributed data delivery systems. Additionally, we obtained the smallest response time for the RESTFUL visualization web service.

Artificial Intelligence Based Clustering with Routing Protocol for Internet of Vehicles

With recent advances made in Internet of Vehicles (IoV) and Cloud Computing (CC), the Intelligent Transportation Systems (ITS) find it advantageous in terms of improvement in quality and interactivity of urban transportation service, mitigation of costs incurred, reduction in resource utilization, and improvement in traffic management capabilities. Many traffic-related problems in future smart cities can be sorted out with the incorporation of IoV in transportation. IoV communication enables the collection and distribution of real-time essential data regarding road network condition. In this scenario, energy-efficient and reliable intercommunication routes are essential among vehicular nodes in sustainable urban computing. With this motivation, the current research article presents a new Artificial Intelligence-based Energy Efficient Clustering with Routing (AI-EECR) Protocol for IoV in urban computing. The proposed AI-EECR protocol operates under three stages namely, network initialization, Cluster Head (CH) selection, and routing protocol. The presented AI-EECR protocol determines the CHs from vehicles with the help of Quantum Chemical Reaction Optimization (QCRO) algorithm. QCRO algorithm derives a fitness function with the help of vehicle speed, trust level, and energy level of the vehicle. In order to make appropriate routing decisions, a set of relay nodes was selected using Group Teaching Optimization Algorithm (GTOA). The performance of the presented AI-EECR model, in terms of energy efficiency, was validated against different aspects and a brief comparative analysis was conducted. The experimental outcomes established that AI-EECR model outperformed the existing methods under different measures.

Multi-source unsupervised soft sensor based on joint distribution alignment and mapping structure preservation

Aiming at the problem of mismatch between real-time data distribution and modeling data distribution caused by the change of working conditions in industrial process, which leads to the performance deterioration of the soft sensor model, a multi-source unsupervised soft sensor method based on joint distribution alignment and mapping structure preservation is proposed. Firstly, the method uses the hypergraph to establish the complex structure of feature and label, and clusters the hypergraph matrix in multiple views to completely construct the class pseudo label; then dynamic distribution alignment is used to adapt marginal distribution and conditional distribution between the data of historical working conditions and the current working conditions, and the hypergraph Laplacian operator is introduced for manifold regularization to prevent the mapping relationship between feature and label from being destroyed; finally, similar working conditions are introduced to further enhance the robustness of the model. The experimental results show that compared with the traditional unsupervised soft sensor methods, the method used in this paper can effectively improve the prediction accuracy of the model.

GHTnet: Tri-Branch deep learning network for real-time electricity price forecasting

A highly accurate electricity price prediction model is of the utmost importance for multiple power systems tasks, such as generation dispatch and bidding. Due to the liberalization of the electricity market, as well as high renewable penetration, the properties of electricity price time series are becoming more stochastic and complex. Traditional statistical methods and machine learning algorithms cannot model such volatile market conditions with high fidelity. In this paper, we propose a data-driven deep learning network (GHTnet) to capture the temporal distribution of real-time price data. A new CNN module, based on GoogLeNet, is developed to capture the high-frequency features of this data, while inclusion of time series summary statistics is shown to improve the forecasting of volatile price spikes. The deep learning model is developed and validated on real-time price time series from 49 generators in the New York Independent System Operator (NYISO), achieving significant performance improvements over that of state-of-the-art benchmark methods, with an average 17.34% improvement in MAPE.

Fishery Data Distribution System Based on Distance Prior Network Coding Strategy with Buffer Mapping Mechanism

Marine fishery production safety has always been a livelihood issue of high concern to government departments at all levels in China. With the construction of marine informatization, a large number of fishing vessels have entered the Vessel Monitoring System (VMS). The massive, highly concurrent, and continuous track data generated by these vessels has posed great challenges to real-time data distribution. To cope with low distribution efficiency in the original data distribution system, this paper proposed a data distribution model based on the combination of network coding and UDP, which improves both reliability and efficiency of data distribution. In order to further enhance the efficiency of data distribution, a Codeword Distance Priority Protocol based on Buffer Map (CDPPBM) was added to the proposed model for the concentration of innovative codewords in the network, thereby increasing the effectiveness of data received by nodes. Experimental results show that the protocol proposed in this paper improves the data distribution efficiency by about 75% on average, compared with the LT code. The influence of data block size on network coded data distribution system is not involved in the previous work. Therefore, this paper discusses the Block to Piece Protocol (BPP) for large files, divides large files into fixed sizes equally, and distributesndata blocks multiple times to find the optimal piece size. The experimental results show that there is an optimalnfor large files of different sizes, which can maximize the efficiency of data distribution.

Simulink Implementation of the Data Distribution Service for Vehicular Controllers on Top of GBE and AFDX

AbstractVehicular networks have seen major changes in the past few years in order to offer reliable and real-time capable high-speed data transmission between electrical and mechatronic components to map current and future innovative functions into distributed systems within automotive applications. In the same context, the real-time middleware data distribution service (DDS) is an appropriate alternative for the standard vehicular middleware considering that it handles quality of service (QoS) parameters including real-time ones. In this paper, we are proposing a new approach for DDS implementation and integration into the vehicular system by creating a model-based design blocks. To validate this implementation, we have used the case of the Society of Automotive Engineers (SAE) vehicle benchmark as a simulation and test model. Therefore, we designed a Simulink vehicle as specified by SAE benchmark. Then, we have introduced a new methodology to link each module to a Simulink DDS blockset. The goal of this approach is to facilitate the use of DDS with vehicular controllers and to reduce the deployment and configuration complexities associated with DDS. It will also enable distributed real-time embedded systems developers to concentrate more on the business logic of the application instead of the low-level implementation details. The final developed architecture has been tested using three different types of real-time networks: FlexRay, Gigabit Ethernet and AFDX, to demonstrate that real-time application’s QoS are always met using this model.

Ethernet evaluation in data distribution traffic for the LHCb filtering farm at CERN

This paper evaluates the real-time distribution of data over Ethernet for the upgraded LHCb data acquisition cluster at CERN. The system commissioning ends in 2021 and its total estimated input throughput is 32 Terabits per second. After the events are assembled, they must be distributed for further data selection to the filtering farm of the online trigger. High-throughput and very low overhead transmissions will be an essential feature of such a system. In this work RoCE (Remote Direct Memory Access over Converged Ethernet) high-throughput Ethernet protocol and Ethernet flow control algorithms have been used to implement lossless event distribution. To generate LHCb-like traffic, a custom benchmark has been implemented. It was used to stress-test the selected Ethernet networks and to check resilience to uneven workload distribution. Performance tests were made with selected evaluation clusters. 100 Gb/s and 25 Gb/s links were used. Performance results and overall evaluation of this Ethernet-based approach are discussed.

Efficient pattern matching for uncertain time series data with optimal sampling and dimensionality reduction

Time series data mining becomes an active research area due to the rapid proliferation of temporal-dependent applications. Dimensionality reduction and uncertainty handling play a pivotal role in extracting the time series pattern. Most of the dimensionality reduction schemes are designed based on the assumption that every class of samples follows the Gaussian distribution. Lack of this property in real time data distribution does not allow dimensionality reduction techniques to characterize the different classes well and measure the data uncertainty accurately. In addition to, applying an uncertainty measurement evenly on inconsistent time series data samples may underestimate the source of uncertainty among various sub-samples. This paper presents the Handling UNcertainty and missing value prediction in Time series (HUNT). The proposed approach employs Adaptive Reservoir Filling for sampling the time series and Discrepant Sample dependent Chebyshev inequality for handling the uncertainty. The HUNT implements the adaptive reservoir filling using discrepancy estimation over a statistical population and decides the reservoir size according to the variations in the data stream. The state of the statistical population ensures the uncertainty handling over discrepant samples. The proposed approach precisely replaces the missing values with the support of the Mean-Mode imputation method. To effectively select the key features, it applies both the indirect and direct performance measures on the statistical samples. Finally, the proposed model generates the fine-tuned statistical samples through segmentation to facilitate the time series pattern matching. The experimental results demonstrate that the HUNT approach significantly outperforms the existing time series pattern matching approaches such as KSample approach by 18% higher recall and UG-Miner approach by 20% minimum Mean Absolute Error (MAE) while testing on the Weather forecasting dataset.

Virtual technology of cache and real-time big data distribution in cloud computing big data center

By comparing several cloud computing of big data network center during COVID-19, this paper proposes a new topology model, which realizes two functions of cloud computing big data center caching and big data real-time distribution. In addition, cloud computing network requires higher performance than traditional application big data center, which makes the consideration of network platform construction performance different from the traditional understanding. During COVID-19, we deeply understood the underlying attributes of cloud, combined with the topology model, we can realize the decoupling of cloud computing big data system, change the situation of direct connection between upstream and downstream, and have more reliable and efficient transmission of message and command big data.

Real-Time Characteristics of ROS 2.0 in Multiagent Robot Systems: An Empirical Study

Due to its ease of use and flexibility, the robot operating system (ROS) is increasingly becoming the most popular middleware for robot applications, even in multiagent systems. Since ROS 1.0 does not satisfy real-time requirements, ROS 2.0 was developed, and it improved the communication stack with the real-time data distribution service (DDS) protocol. However, the actual performance level to be expected is still unknown and can largely depend on the operating system and the kernel being used, the DDS distribution, and the overall software load of the system. In this article, we present an empirical study that evaluates the real-time performance of ROS 2.0 in both the system and communication software layers. In the system layer, the deterministic behavior of the ROS 2.0 nodes is thoroughly observed with regard to whether the tasks are schedulable and can function within the specified temporal deadline. In the communication layer, special attention is devoted to the rate of data loss and the overall latency of messages between nodes. Experiments are performed in various working conditions; for example, the system load is increased to define the real-time performance of the tasks. For reference, the results are compared with the those from the traditional ROS variation. Moreover, we implement a multiagent service robot system to verify the suitability of ROS 2.0 for real-world applications. Our results show that the application of ROS 2.0 is more suitable than that of ROS 1.0 in terms of real-time performance.

DDS middleware on top of FlexRay networks: Simulink blockset implementation of electrical vehicle using FlexRay protocol and its adaptation to DDS concept

We designed a Simulink vehicle blockset in order to integrate safety, reliability, best condition of luxury and comfort in modern models by adding or modifying some old algorithms, implemented as application layer components. The proposed blockset corresponds to the Society of Automotive Engineers benchmark which is normally connected by the CAN bus that we replaced by the FlexRay bus. We used the Embedded MATLAB tool for implementing this blockset in order to generate a portable C standard code. To connect the designed sub-blocks together, we applied the real-time middleware data distribution service (DDS) and the FlexRay protocol. In this paper, we focused our interest on the implementation of the Simulink blockset with its different sub-blocks and we then identified their DDS entities and their QoS profiles. We also calculated the worst-case response time, based on the full scheduling model, which served to configure the DDS deadline QoS and the FlexRay protocol parameters.

Fuzzy Rough Set Inspired Rate Adaptation and resource allocation using Hidden Markov Model (FRSIRA-HMM) in mobile ad hoc networks

Data transmission in ad hoc networks necessitates different quality of services (QoS) for ensuring reliable throughput of the network. This reliability in network performance is achievable through a rate adaptation scheme that aids in effective allocation of resource for ensuring QoS, based on network demand. In this approach, the available bandwidth is forecasted based on Hidden Markov Model for improving the efficacy in resource allocation and the new users are facilitated to utilize the resources based on the estimation of currently available resources determined through rough set theory. The rate of adaptation is forecasted based on the present load condition of the network by Rough set theory, which forms the input of the Hidden Markov Model. The fuzzy Rough set concept is mainly used in this technique as they are reliable in analyzing data that not require any basic or auxiliary information pertaining to data than its fuzzy set theory, Bayesian theory and Shafer–Dempster theory counter parts that necessitate the assignment of probability. Fuzzy Rough set concept determines the rate of adaptation through the estimation of queuing state, queuing delay, channel utilization rate, and bandwidth availability. This fuzzy Rough set concept also considers the resource allocation as the significant entity as the channel allocation in dynamic networks purely relies on the dynamic channel variation that alternates in real time distribution of data.

A tree-based data collection protocol for optical unmanned aerial vehicle networks

In Wireless Sensor Networks (WSNs), sensors are spread over a remote and hardly accessed sensing zone. In this situation, Unmanned Aerial Vehicles (UAVs) allow to facilitate the data gathering from sensors in real time. In the present paper, we consider an optical UAV network that allows the real time data distribution to a base station, based on multiple collection trees and multiple base stations. Each UAV is identified by an optical codeword with adequate structure to reduce the codeword management process implied by the UAV mobility. Using the attributed codewords, each UAV can optically switch the gathered data to reach a base station for processing. Furthermore, we present quality-based attachment and re-attachment procedures allowing to an UAV to attach and re-attach to a tree whether it notices the quality degradation of its link to the collection tree.