Real-time Data Stream Processing Research Articles

A New Architecture for Real Time Data Stream Processing

Processing a data stream in real time is a crucial issue for several applications, however processing a large amount of data from different sources, such as sensor networks, web traffic, social media, video streams and other sources, represents a huge challenge. The main problem is that the big data system is based on Hadoop technology, especially MapReduce for processing. This latter is a high scalability and fault tolerant framework. It also processes a large amount of data in batches and provides perception blast insight of older data, but it can only process a limited set of data. MapReduce is not appropriate for real time stream processing, and is very important to process data the moment they arrive at a fast response and a good decision making. Ergo the need for a new architecture that allows real-time data processing with high speed along with low latency. The major aim of the paper at hand is to give a clear survey of the different open sources technologies that exist for real-time data stream processing including their system architectures. We shall also provide a brand new architecture which is mainly based on previous comparisons of real-time processing powered with machine learning and storm technology.

CLARA: CLAS12 Reconstruction and Analysis Framework

In this paper we present SOA based CLAS12 event Reconstruction and Analyses (CLARA) framework. CLARA design focus is on two main traits: real-time data stream processing, and service-oriented architecture (SOA) in a flow based programming (FBP) paradigm. Data driven and data centric architecture of CLARA presents an environment for developing agile, elastic, multilingual data processing applications. The CLARA framework presents solutions capable of processing large volumes of data interactively and substantially faster than batch systems.

Efficiency of Stream Processing Engines for Processing BIGDATA Streams

Background/Objectives: In this paper, there are real-time stream & big data stream applications running on a large amount of clusters & live migration of data using intensive & interactive tasks. Methods/Statistical Analysis: Traditionally, there are many stream processing engines which are going to manage the processing data in this streaming scenario, ongoing stream processing the data must be handle due to faults & stragglers within a second-scale latency. At present, there are efficient stream processing engines which can avoid these faults & stragglers by using the stream processing engines like APACHE SPARK & APACHE FLINK. Findings: In the real-time data stream processing there are some problem occurrences while processing the data, those are data latency, fault-tolerance, mutable data-sets and more stragglers. To avoid all these issues there is an efficient stream processing engines with additional added mechanisms like dolly retreat mechanism for avoiding stragglers and process data efficiently. Application/Improvements: Now a day’s demand for stream processing is increasing a lot which may be real-time streaming or normal cluster data transformation. Data has to be processed fast, so that companies are integrating these stream processing engines to changing their business conditions effectively in realtime for analyze their data more effectively with help of Fast Stream Processing Engines.

Entity Resolution Approach of Data Stream Management Systems

Owing to the technological advancements in Semantic Web and sensor networks, a large amount of data has been produced in association with the open data policy. However, data stream management systems that process stream data have focused on the processing of a large amount of data with little priority on data identification, integration, and external linkage. Furthermore, entity resolution is focused mainly on static database-based technologies. In this study, a real-time stream data processing architecture that can perform the integration and entity resolution of streaming-type heterogeneous input data and interlink with external data is designed. To achieve this goal, a light adapter to integrate heterogeneous data into standard scheme and blocking technique to reduce comparison candidates are applied. The implemented data adapters shows 4 times higher throughput than open source data parsers and the entity resolution results with streaming data shows similar performance with the static data sets. The proposed streaming data entity resolution architecture is expected to form the basis of data integration research that can integrate various information sources of data efficiently, enrich internal data.

An Evaluation of Data Stream Processing Systems for Data Driven Applications

Real-time data stream processing technologies play an important role in enabling time-critical decision making in many applications. This paper aims at evaluating the performance of platforms that are capable of processing streaming data. Candidate technologies include Storm, Samza, and Spark Streaming. To form the recommendation, a prototype pipeline is designed and implemented in each of the platforms using data collected from sensors used in monitoring heavy-haul railway systems. Through the testing and evaluation of each candidate platform, using both quantitative and qualitative metrics, the paper describes the findings, where Storm is found to be the most appropriate candidate.

A mobile crowd sensing ecosystem enabled by CUPUS: Cloud-based publish/subscribe middleware for the Internet of Things

Mobile crowd sensing (MCS) is a novel class of mobile Internet of Things (IoT) applications for community sensing where sensors and mobile devices jointly collect and share data of interest to observe phenomena over a large geographic area. The inherent device mobility and high sensing frequency has the capacity to produce dense and rich spatiotemporal information about our environment, but also creates new challenges due to device dynamicity and energy constraints, as well as large volumes of generated raw sensor data which need to be processed and analyzed to extract useful information for end users. The paper presents an ecosystem for mobile crowd sensing which relies on the CloUd-based PUblish/Subscribe middleware (CUPUS) to acquire sensor data from mobile devices in a flexible and energy-efficient manner and to perform near real-time processing of Big Data streams. CUPUS has unique features compared to other MCS platforms: It enables management of mobile sensor resources within the cloud, supports filtering and aggregation of sensor data on mobile devices prior to its transmission into the cloud based on global data requirements, and can push information of interest from the cloud to user devices in near real-time. We present our experience with implementation and deployment of an MCS application for air quality monitoring built on top of the CUPUS middleware. Our experimental evaluation shows that CUPUS offers scalable processing performance, both on mobile devices and within the cloud, while its data propagation delay is mainly affected by transmission delay on wireless links.

Data stream management system for moving sensor object data

Sensor and communication development has led to the development of new types of applications. Classic database data storage becomes inadequate when data streams arrive from multiple sensors. Then, data querying and result presentation are not efficient. The desired results are obtained with a delay, and the database is filled with a large amount of unnecessary data. To adequately support the above applications, Data Stream Management System (DSMS) applications are needed. DSMSs provide real-time data stream processing. In this paper, a client-server system is presented with DSMS realized on the Java WebDSMS application server side. WebDSMS functionalities are tested with simulated data and in real-life usage.

Parallel computing on graphics processing units and heterogeneous platforms

Parallel computing on graphics processing units and heterogeneous platforms

Improved Data Stream On-Line Segmentation Based Ultra Short-Term Load Forecasting

To improve both realtime performance and accuracy of ultra short-term load forecasting and to cope with the higher requirement of power grid on realtime load forecasting, based on improved data stream on-line segmentation an ultra short-term load forecasting method is proposed. Based on the time trend of load development, the realtime data stream processing is utilized in the proposed method to perform ultra short-term forecasting, then combining with the results of short-term load forecasting, which contain the whether factors and the effect of load cycle property, the realtime forecasting result at the segmentation point is corrected. The fast segmentation and forecasting ability of the proposed method avoid repeat modeling and improve the speed of forecasting; the realtime correction and handling of the segmentation point increase historical information utilization and decrease the error of segmentation point, thus the forecasting accuracy can be maintained at a better level. Actual load data is adopted to validate the effectiveness of the proposed model, and validation results show that both load forecasting accuracy and speed by the proposed model have an advantage over those by several conventional ultra short-term load forecasting algorithms, besides the forecasting error at the inflection point of load can be decreased and the proposed method can also adapt to the sudden change of the weather.

Feature-Based High Availability Mechanism for Extreme Aggregation Tasks in Real-Time Data Stream Processing

Data stream under Cloud or IoT (Internet of Things) implies real-time and continuous processing, during which availability guarantee is essential but expensive. For extreme aggregation tasks, traditional HA mechanisms require vast space at run-time and much longer recovery latency at fail-time especially under worse input. In this paper, feature-based high availability mechanism is proposed for extreme aggregation tasks, in which space-bounded feature is maintained through random sampling over time-based sliding window and failed tasks can be recovered precisely with high probability in an efficient way. The probabilistic effectiveness has been proved theoretically, and meanwhile the acceptable tradeoff between related overheads and performance has been demonstrated by comprehensive experiments on both synthetic and real data.

SensorStream: a semantic real-time stream management system

As data proliferates at increasing rates, the need for real-time stream processing applications increases as well. In the same way that Data Stream Management Systems (DSMS) have emerged from the database community, there is now a similar concern in managing dynamic knowledge among the Semantic Web community. Unfortunately, early relevant approaches are, to a large extent, theoretical and do not present convincing evidence of their efficiency in real dynamic environments. In this paper, we present a framework for the effective, real-time processing of streaming data and we define and analyse in depth its key components. Our framework serves as a basis for the implementation of the SensorStream prototype, on which we run numerous performance and scalability measurements that outline its behaviour and demonstrate its suitability and scalability for solutions that require real-time information processing from distributed and heterogeneous data sources.

An adaptive γ-ray spectrometer with a high event processing rate

A fast γ-ray spectrometer adaptable to the scintillator type is described. This spectrometer is capable of processing a γ-ray energy spectrum (with a resolution of <4% in the energy range from 20 keV to 10 MeV) in a sequence of time intervals in the real-time mode at a counting rate of up to 106 cps. Digitization of the detector signals by a 14-bit ADC with a sampling rate of 64 MHz and real-time data stream processing are used to separate overlapping events and correctly generate energy spectra.

A Data Filter for Identifying Steady-State Operating Points in Engine Flight Data for Condition Monitoring Applications

Abstract This paper presents an algorithm that automatically identifies and extracts steady-state engine operating points from engine flight data. It calculates the mean and standard deviation of select parameters contained in the incoming flight data stream. If the standard deviation of the data falls below defined constraints, the engine is assumed to be at a steady-state operating point and the mean measurement data at that point are archived for subsequent condition monitoring purposes. The fundamental design of the steady-state data filter is completely generic and applicable for any dynamic system. Additional domain-specific logic constraints are applied to reduce data outliers and variance within the collected steady-state data. The filter is designed for on-line real-time processing of streaming data as opposed to post-processing of the data in batch mode. Results of applying the steady-state data filter to recorded helicopter engine flight data are shown, demonstrating its utility for engine condition monitoring applications.

Interactive multi-scale visualizations of tonal evolution in MuSA.RT Opus 2

The purpose of this article is to describe MuSA.RT Opus 2, an interactive system for tonal visualization of music at multiple scales, and to present examples of the types of musical features and attributes that can be abstracted and visualized by the system. MuSA.RT aims to create an environment by which musical performances can be mapped in real-time to a concrete and visual metaphor for tonal space, wherein we can see the establishment and evolution of the tonal context. In this environment, expert musicians will be able to see the tonal structures of what they play, initiated listeners will be able to visually follow the structures that they hear, and novices can learn to hear the structures that they see. MuSA.RT is both an interactive art installation that can convert musical performances to mathematically elegant graphics and a scientific tool for visualizing the inner workings of tonal induction and tracking algorithms. In this article we describe the mapping strategies for transforming a MIDI stream into tonal structures in 3D space, and our solution for overcoming the challenge of real-time concurrent processing of data streams; we will also give examples and present case studies of visual mappings of music by Pachelbel, Bach, and Barber.

CBP and ART image reconstruction algorithms on media and DSP processors

Computed Tomography (CT) image reconstruction algorithms such as convolution back-projection (CBP) and algebraic reconstruction technique (ART) are highly compute-intensive for today's single processor systems. In this work, we investigate the suitability of TM-1000 media processor and Analog Device's ADSP 21160 as a compute engine for the execution of image reconstruction algorithms. Philips Trimedia TM-1000, a very large instruction word (VLIW) processor, is a high performance media processor optimized for real-time processing of audio, video, graphifcs, and communication data streams. TM-1000 has a high performance digital signal processor (DSP) core, supported by multiple functional units. The DSP core and the functional blocks operate in parallel, driven by a mix of RISC, multimedia, SIMD-type DSP and floating point instructions. A typical DSP such as Analog Device's ADSP 21160 is based on super harvard architecture (SHARC) and is an optimized processor for digital signal processing applications. It has two sets of computation units. Each computation unit comprises of three functional blocks namely, arithmetic and logic unit (ALU), multiplier and shifter unit. ADSP 21160 supports single instruction multiple data (SIMD) computation model to handle dual computation units. Both sets of computation units operate concurrently. We compare the performance of TM-1000 media processor and ADSP 21160 DSP processor to execute the image reconstruction algorithms by comparing the execution time of CBP and ART algorithms on them. The image reconstruction algorithms normally break down to a repetitive multiply-accumulate operation (MAC). All DSP processors support single-cycle MAC and zero-overhead loop instructions. The media processors normally do not support single-cycle MAC instruction and zero-overhead loop functionality. However, media processors are equipped with multiple functional units that perform multiple operations in a single instruction time. A DSP processor is expected to execute image reconstruction algorithms much faster than a multimedia processor. However, the experimental results show that the execution time on DSP and media processor are more or less same when 16-bit representation is used for data. When floating point data is used for implementation, DSP processor has an edge. ADSP 21160 gives same execution time for both floating point and 16-bit fixed point data. But, the execution time almost doubles when data is in floating point format on the media processor, compared to a 16-bit implementation. This can be attributed to the fact that TM-1000 processes two sets of operands in a single instruction time when data is in 16-bit format. The executable code for ADSP 21160 was generated from an optimized assembly language program whereas the executable code for TM-1000 was generated from an optimized ‘C’ with a few custom operations.