Continuous Data Flow Research Articles

Evaluating the impact of a coordinated checkpointing in distributed data streams processing systems using discrete event simulation

Data Streams Processing systems process continuous flows of data under Quality of Service requirements. Data streams often contain critical information which requires real-time processing. To guarantee systems' dependability and avoid information loss, one must use a fault-tolerance strategy. However, there are several strategies available, and the proper evaluation of which mechanism is better for each system architecture is challenging, especially in large-scale distributed systems. In this paper, we propose a discrete simulation model for investigating the impacts of the Coordinated Checkpoint fault tolerance strategy imposes on Data Stream Processing Systems. Results show that this strategy critically affects stream processing in failure-prone situations due to an increase in latency up to 120% and information loss, reaching 95% of the processing window in the worst case.

Novel Bit-Reordering Circuit for Continuous-Flow Parallel FFT Architectures

This brief presents a parallel bit-reordering circuit for calculating bit reordering on a stream of parallel data. The circuit consists of buffers, multiplexers, a bit-reversal circuit for single path serial data, and control signal generators. A novel data reordering mechanism has been devised that has twofold benefits: firstly, it helps in attaining continuous flow of data with minimum memory and minimum latency; secondly, the area and power requirement of the circuit reduces by 15-20%. The proposed circuit provides a solution for reordering the data of different common radices such as radix-2 k , radix-4, radix-8, and other radices. The circuit generates natural order output for variable power-of-2 (2 n ) fast Fourier transform (FFT) lengths.

Generalizable Resource Allocation in Stream Processing via Deep Reinforcement Learning

This paper considers the problem of resource allocation in stream processing, where continuous data flows must be processed in real time in a large distributed system. To maximize system throughput, the resource allocation strategy that partitions the computation tasks of a stream processing graph onto computing devices must simultaneously balance workload distribution and minimize communication. Since this problem of graph partitioning is known to be NP-complete yet crucial to practical streaming systems, many heuristic-based algorithms have been developed to find reasonably good solutions. In this paper, we present a graph-aware encoder-decoder framework to learn a generalizable resource allocation strategy that can properly distribute computation tasks of stream processing graphs unobserved from training data. We, for the first time, propose to leverage graph embedding to learn the structural information of the stream processing graphs. Jointly trained with the graph-aware decoder using deep reinforcement learning, our approach can effectively find optimized solutions for unseen graphs. Our experiments show that the proposed model outperforms both METIS, a state-of-the-art graph partitioning algorithm, and an LSTM-based encoder-decoder model, in about 70% of the test cases.

Experimental Data of Muon Hodoscope URAGAN for Investigations of Geoffective Processes in the Heliosphere

Muon hodoscope URAGAN continuously detects the angular distribution of muons in a wide range of zenith angles and allows one to obtain information about variations, both in the intensity and in angular characteristics of the muon flux, caused by active processes in the heliosphere, the magnetosphere and atmosphere of the Earth. Various parameters of the muon flux anisotropy and methods of calculation of these parameters are discussed. Real-time processing of a continuous flow of multidimensional data from the muon hodoscope URAGAN is quite a challenge. In the article, methods of formation and primary analysis of the data, their processing in real time and obtaining time series of various parameters of integral counting rate and angular anisotropy of the muon flux, which are important for the physical analysis of modulation processes of cosmic rays in the heliosphere, are presented.

BioMeT and Algorithm Challenges: A Proposed Digital Standardized Evaluation Framework.

Technology is advancing at an extraordinary rate. Continuous flows of novel data are being generated with the potential to revolutionize how we better identify, treat, manage, and prevent disease across therapeutic areas. However, lack of security of confidence in digital health technologies is hampering adoption, particularly for biometric monitoring technologies (BioMeTs) where frontline healthcare professionals are struggling to determine which BioMeTs are fit-for-purpose and in which context. Here, we discuss the challenges to adoption and offer pragmatic guidance regarding BioMeTs, cumulating in a proposed framework to advance their development and deployment in healthcare, health research, and health promotion. Furthermore, the framework proposes a process to establish an audit trail of BioMeTs (hardware and algorithms), to instill trust amongst multidisciplinary users.

Anomaly Detection Based Latency-Aware Energy Consumption Optimization For IoT Data-Flow Services.

The continuous data-flow application in the IoT integrates the functions of fog, edge, and cloud computing. Its typical paradigm is the E-Health system. Like other IoT applications, the energy consumption optimization of IoT devices in continuous data-flow applications is a challenging problem. Since the anomalous nodes in the network will cause the increase of energy consumption, it is necessary to make continuous data flows bypass these nodes as much as possible. At present, the existing research work related to the performance of continuous data-flow is often optimized from system architecture design and deployment. In this paper, a mathematical programming method is proposed for the first time to optimize the runtime performance of continuous data flow applications. A lightweight anomaly detection method is proposed to evaluate the reliability of nodes. Then the node reliability is input into the optimization algorithm to estimate the task latency. The latency-aware energy consumption optimization for continuous data-flow is modeled as a mixed integer nonlinear programming problem. A block coordinate descend-based max-flow algorithm is proposed to solve this problem. Based on the real-life datasets, the numerical simulation is carried out. The simulation results show that the proposed strategy has better performance than the benchmark strategy.

Veracity handling and instance reduction in big data using interval type-2 fuzzy sets

Within the aspect of big data, veracity refers to the existing uncertainty in the dataset. The continuous flow of unstructured data with unwanted noise may bring abnormality in the dataset making them unusable. In this paper, we propose a novel method to handle the veracity characteristic of the big data using the concept of footprint of uncertainty (FOU) in interval type-2 fuzzy sets (IT2 FSs). The proposed method helps in handling the veracity issue in big data and reduces the instances to a manageable extent. We have compared the results with the existing clustering based methods and examined the relationship between the clusters and the FOUs by comparing their centroids and defuzzified values. To scrutinize the validity of our results, we have also performed a number of additional experiments by appending extra instances to the datasets. To check its consistency and efficacy, the proposed methodology is assessed from three different aspects. Experimental result validates that the proposed method can suitably handle the veracity issue in big datasets and is efficient in reducing the instances.

AdaptFlow: Adaptive Flow Forwarding Scheme for Software-Defined Industrial Networks

Industry 4.0 revolution has emerged as an escalator for increased productivity and cost savings in smart factories. However, they pose a big challenge for network architecture in providing a flexible way for continuous data flow. The necessity of reliable connectivity in the industrial ecosystem has paved the path for the evolution of software-defined industrial networks. However, the nature of data traffic in the industrial environment further pushes the need for adaptive flow forwarding and control. Therefore, in this article, AdaptFlow, an adaptive flow forwarding scheme, is designed for software-defined industrial networks. AdaptFlow includes: 1) application-specific traffic classification using self-organized maps; 2) B+ tree-based flow table management; 3) queuing model for analyzing the waiting time; and 4) energy-aware flow forwarding algorithm. The evaluation of AdaptFlow scheme shows improvements in minimizing delay and energy consumption while increasing network capacity in contrast to the OpenFlow architecture.

Local trend discovery on real-time microblogs with uncertain locations in tight memory environments

This paper presents GeoTrend+; a system approach to support scalable local trend discovery on recent microblogs, e.g., tweets, comments, online reviews, and check-ins, that come in real time. GeoTrend+ discovers top-k trending keywords in arbitrary spatial regions from recent microblogs that continuously arrive with high rates and a significant portion has uncertain geolocations. GeoTrend+ distinguishes itself from existing techniques in different aspects: (1) Discovering trends in arbitrary spatial regions, e.g., city blocks. (2) Considering both exact geolocations, e.g., accurate latitude/longitude coordinates, and uncertain geolocations, e.g., district-level or city-level, that represents a significant portion of past years microblogs. (3) Promoting recent microblogs as first-class citizens and optimizes different components to digest a continuous flow of fast data in main-memory while removing old data efficiently. (4) Providing various main-memory optimization techniques that are able to distinguish useful from useless data to effectively utilize tight memory resources while maintaining accurate query results on relatively large amounts of data. (5) Supporting various trending measures that effectively capture trending items under a variety of definitions that suit different applications. GeoTrend+ limits its scope to real-time data that is posted during the last T time units. To support its queries efficiently, GeoTrend+ employs an in-memory spatial index that is able to efficiently digest incoming data and expire data that is beyond the last T time units. The index also materializes top-k keywords in different spatial regions so that incoming queries can be processed with low latency. In peak times, the main-memory optimization techniques are employed to shed less important data to sustain high query accuracy with limited memory resources. Experimental results based on real data and queries show the scalability of GeoTrend+ to support high arrival rates and low query response time, and at least 90+% query accuracy even under limited memory resources.

A version of Organization of Multithread Processing of Confidential Data on the Basis of Cellular Automata

Рurpose of research is to develop a multithread processing system based on an encryption algorithm using cellular automata and to study statistical performance indicators depending on the hardware components and the size of the input block, and to develop recommendations for improving the cryptostrength of the method. Methods . A mathematical model of the encryption method using a floating window based on cellular automata was considered [3]. To study the speed of confidential data processing, there was developed a variant of the organization of the structure of the software module with an extended block of setting parameters that determine the dimension of the matrix, the line of activation of the bit neighborhood of the processed elements, the number of parallel computations (threads) and the rule of expansion of the boundary elements of the matrix. A method for the development of the dependence curve of the processing time and inital parameters that can be applied both to process individual files and continuous network subscriber data flow, is proposed. Results. The cryptographic module implementing the encryption method on the basis of cellular automata, which specific feature is a multithread mode of operation and dynamic control of the block of initial parameters, was developed. Recommendations for setting the neighborhood of the active elements of the matrix and the number of threads taking into account the architecture of the CPU were formulated. Experimental studies were conducted to confirm the completeness and correctness of the proposed solutions. The expediency of using high-speed hard disk drives and saving the results of encryption in asynchronous segmented mode with working thread-bind results was revealed. Conclusion. The proposed version of the organization of the confidential data processing system in the form of a software module, taking into account the features of the hardware, allows optimization of the processing speed, and the compliance with the recommendations for the expansion of the neighborhood in the block transformation can improve the cryptographic algorithm based on cellular automata with a floating window.

Dynamic alternate buffer switching and congestion control in wireless multimedia sensor networks

Normally sensor nodes are located at unattended environment which has low battery power. The wireless hypermedia sensor based networks have huge capacity and continuous flow of data from source to sink. Congestion arises at the time of incoming data is greater than available resources. The congestion causes packet loss, buffer over flow, energy wastage and increases end to end delay. The proposed method Dynamic Alternate Buffer switching and Congestion Control is used to manage congestion in an efficient manner. In which congestion is detected based on remaining buffer, remaining energy and trust level of the sensor nodes. This approach is based on cost evaluation method which decides primary and spare buffers at run time. This method is used to enable dynamic buffer switching and optimize the effect of congestion. The performance of BETCC compared with various existing protocol like TCEER and TFCC in terms of energy consumption and data loss ratio.

AIS-Clus: A Bio-Inspired Method for Textual Data Stream Clustering

The spread of real-time applications has led to a huge amount of data shared between users. This vast volume of data rapidly evolving over time is referred to as data stream. Clustering and processing such data poses many challenges to the data mining community. Indeed, traditional data mining techniques become unfeasible to mine such a continuous flow of data where characteristics, features, and concepts are rapidly changing over time. This paper presents a novel method for data stream clustering. In this context, major challenges of data stream processing are addressed, namely, infinite length, concept drift, novelty detection, and feature evolution. To handle these issues, the proposed method uses the Artificial Immune System (AIS) meta-heuristic. The latter has been widely used for data mining tasks and it owns the property of adaptability required by data stream clustering algorithms. Our method, called AIS-Clus, is able to detect novel concepts using the performance of the learning process of the AIS meta-heuristic. Furthermore, AIS-Clus has the ability to adapt its model to handle concept drift and feature evolution for textual data streams. Experimental results have been performed on textual datasets where efficient and promising results are obtained.

Heliogeophysical prediction service in Poland:past, present and future

Abstract. The Regional Warning Centre (RWC) of Warsaw operates as the Heliogeophysical Forecasting Centre in the Space Research Centre (SRC) of the Polish Academy of Sciences. It is involved in the current collection of a large volume of data exchanged with the International Space Environment Service (ISES) and received directly from various international observatories and Polish-operated geophysical stations. From this continuous flow of data, daily forecasts are issued of space weather conditions and their influence on the performance of technological systems on the Earth and in the space environment. Special services are prepared for governmental and commercial customers, including the SRC. Software packages have been developed for the processing of data and to implement the prediction and forecast systems.

Fuzzy-Folded Bloom Filter-as-a-Service for Big Data Storage in the Cloud

With the ongoing trend of smart and Internet-connected objects being deployed across a broad range of applications, there is also a corresponding increase in the amount of data movement across different geographical regions. This, in turn, poses a number of challenges with respect to big data storage across multiple locations, including cloud computing platform. For example, the underlying distributed file system has a large number of directories and files in the form of gigantic trees, which are difficult to parse in polynomial time. Moreover, with the exponential increase of big data streams (i.e., unbounded sets of continuous data flows), challenges associated with indexing and membership queries are compounded. The capability to process such significant amount of data with high accuracy can have significant impact on decision-making and formulation of business and risk-related strategies, particularly in our current Industrial Internet of Things environment (IIoT). However, existing storage solutions are deterministic in nature. In other words, they tend to consume considerable memory and CPU time to yield accurate results. This necessitates the design of efficient quality of service-aware IIoT applications that are able to deal with the challenges of data storage and retrieval in the cloud computing environment. In this paper, we present an effective space-effective strategy for massive data storage using bloom filter (BF). Specifically, in the proposed scheme, the standard BF is extended to incorporate fuzzy-enabled folding approach, hereafter referred to as fuzzy folded BF (FFBF). In FFBF, fuzzy operations are used to accommodate the hashed data of one BF into another to reduce storage requirements. Evaluations on UCI ML AReM and Facebook datasets demonstrate the efficacy of FFBF, in terms of dealing with approximately 1.9 times more data as compared to using the standard BF. This is also achieved without affecting the false positive rate and query time.

Thinger.io: An Open Source Platform for Deploying Data Fusion Applications in IoT Environments.

In the last two decades, data and information fusion has experienced significant development due mainly to advances in sensor technology. The sensors provide a continuous flow of data about the environment in which they are deployed, which is received and processed to build a dynamic estimation of the situation. With current technology, it is relatively simple to deploy a set of sensors in a specific geographic area, in order to have highly sensorized spaces. However, to be able to fusion and process the information coming from the data sources of a highly sensorized space, it is necessary to solve certain problems inherent to this type of technology. The challenge is analogous to what we can find in the field of the Internet of Things (IoT). IoT technology is characterized by providing the infrastructure capacity to capture, store, and process a huge amount of heterogeneous sensor data (in most cases, from different manufacturers), in the same way that it occurs in data fusion applications. This work is not simple, mainly due to the fact that there is no standardization of the technologies involved (especially within the communication protocols used by the connectable sensors). The solutions that we can find today are proprietary solutions that imply an important dependence and a high cost. The aim of this paper is to present a new open source platform with capabilities for the collection, management and analysis of a huge amount of heterogeneous sensor data. In addition, this platform allows the use of hardware-agnostic in a highly scalable and cost-effective manner. This platform is called Thinger.io. One of the main characteristics of Thinger.io is the ability to model sensorized environments through a high level language that allows a simple and easy implementation of data fusion applications, as we will show in this paper.

An In-Networking Double-Layered Data Reduction for Internet of Things (IoT).

Due to the ever-increasing number and diversity of data sources, and the continuous flow of data that are inevitably redundant and unused to the cloud, the Internet of Things (IoT) brings several problems including network bandwidth, the consumption of network energy, cloud storage, especially for paid volume, and I/O throughput as well as handling huge amount of stored data in the cloud. These call for data pre-processing at the network edge before data transmission over the network takes place. Data reduction is a method for mitigating such problems. Most state-of-the-art data reduction approaches employ a single tier, such as gateways, or two tiers, such gateways and the cloud data center or sensor nodes and base station. In this paper, an approach for IoT data reduction is proposed using in-networking data filtering and fusion. The proposed approach consists of two layers that can be adapted at either a single tier or two tiers. The first layer of the proposed approach is the data filtering layer that is based on two techniques, namely data change detection and the deviation of real observations from their estimated values. The second layer is the data fusion layer. It is based on a minimum square error criterion and fuses the data of the same time domain for specific sensors deployed in a specific area. The proposed approach was implemented using Python and the evaluation of the approach was conducted based on a real-world dataset. The obtained results demonstrate that the proposed approach is efficient in terms of data reduction in comparison with Least Mean Squares filter and Papageorgiou’s (CLONE) method.

Multiple changepoint detection in categorical data streams

The need for efficient tools is pressing in the era of big data, particularly in streaming data applications. As data streams are ubiquitous, the ability to accurately detect multiple changepoints, without affecting the continuous flow of data, is an important issue. Change detection for categorical data streams is understudied, and existing work commonly introduces fixed control parameters while providing little insight into how they may be chosen. This is ill-suited to the streaming paradigm, motivating the need for an approach that introduces few parameters which may be set without requiring any prior knowledge of the stream. This paper introduces such a method, which can accurately detect changepoints in categorical data streams with fixed storage and computational requirements. The detector relies on the ability to adaptively monitor the category probabilities of a multinomial distribution, where temporal adaptivity is introduced using forgetting factors. A novel adaptive threshold is also developed which can be computed given a desired false positive rate. This method is then compared to sequential and nonsequential change detectors in a large simulation study which verifies the usefulness of our approach. A real data set consisting of nearly 40 million events from a computer network is also investigated.

Big Data and Advanced Analytics in Industry 4.0: a comparative analysis across the European Union

Digital economy and Factory (Industry) 4.0 are among the main challenges in the era of Big Data Analytics. A digital transformation is required since factories and enterprises need to face quick changes in the technological process and continuous flows of massive data, in order to improve the decision making process. First, we want to investigate the actual scenario in Europe. For instance, the data from the Digital Economy Society Index (DESI) by the European commission (available at https://ec.europa.eu/digital-single-market/en/countries-performance-digitisation) gives on overview on countries’ performance digitization. The report measures such factors as connectivity, digital skills, digital public services, and more. In particular, it is of interest to analyze the different incidence of employers with digital skills (Analytics, Cybersecurity, Cloud, for instance) on competitiveness and efficiency among countries and industrial sectors, with a privileged eye on Spain and Italy, and to investigate the profile of those industrial sector that are actually investing in digital skills and those that are not.

Stream Processing Languages in the Big Data Era

This paper is a survey of recent stream processing languages, which are programming languages for writing applications that analyze data streams. Data streams, or continuous data flows, have been around for decades. But with the advent of the big-data era, the size of data streams has increased dramatically. Analyzing big data streams yields immense advantages across all sectors of our society. To analyze streams, one needs to write a stream processing application. This paper showcases several languages designed for this purpose, articulates underlying principles, and outlines open challenges.

Adopting Cloud Computing in Aviation Industry for Flight Safety

Flight Data Recorder (FDR) an aircraft component mounted on board, registers and records all different operations and conditions of aircraft during flight. Video animation and reconstruction of the flight progress can then be visualized by investigators to show the last moments before the accident happened after data retrieval from FDR. There might be difficulties in some cases where it is hard to rescue FDR from accident scene, or if found, data cannot be retrieved due to exposure to extreme heat. Thus, an alternative source of data is highly crucial; therefore cloud computing technology linkage to aircraft systems in the Real Time for data network environment to maintain continuous flow of data and safety has been stressed upon. This research examined how concerned stakeholders in the global aviation industry can improve significantly on aviation safety by linking and saving the aircraft’s data communications to cloud computing on the ground to facilitate the tracking and monitoring of aircrafts in real time, in case the aircraft crashes and loses or suffers damages on its black box component.