Real-time Data Processing System Research Articles

Real-time intelligent big data processing: technology, platform, and applications

Human beings keep exploring the physical space using information means. Only recently, with the rapid development of information technologies and the increasing accumulation of data, human beings can learn more about the unknown world with data-driven methods. Given data timeliness, there is a growing awareness of the importance of real-time data. There are two categories of technologies accounting for data processing: batching big data and streaming processing, which have not been integrated well. Thus, we propose an innovative incremental processing technology named after Stream Cube to process both big data and stream data. Also, we implement a real-time intelligent data processing system, which is based on real-time acquisition, real-time processing, real-time analysis, and real-time decision-making. The real-time intelligent data processing technology system is equipped with a batching big data platform, data analysis tools, and machine learning models. Based on our applications and analysis, the real-time intelligent data processing system is a crucial solution to the problems of the national society and economy.

NI P2P powered multi FPGA real-time data processing on tokamak diagnosis systems

On tokamaks, there are many diagnostics, which need real-time data acquisition and processing to provide useful information for plasma control. Some of the diagnostics required fast processing of multiple very high sampling rate signals. It is often difficult to achieve even with modern multi-core CPUs or GPUs. Moving large amount of data from the digitizers to the system ram will hurt the deterministic performance significantly. FPGA data pre-processing is a better choice when the data large and deterministic performance is required. When the task is too heavy for a single FPGA to implement, transfer large data between FPGAs is also required. On the polarimeter-interferometer diagnostics on J-TEXT, in order to calculate the density profile of the plasma in real-time, multiple chords of line-integrated density must be calculated first. The data acquisition system for the polarimeter-interferometer is equipped with 5 FlexRIO FPGAs, only one is acquiring the reference signal. Density calculation for every probing chord needs the reference signal. So this signal needs to be transferred to other FPGAs with strict real-time requirement. Mismatching a single sample, the result would be completely incorrect. NI P2P streaming allow the FPGA to transfer data without burdening the process. In the paper, the determinism characteristic and other performance of the P2P streaming is tested, and a mechanism to align local digitized samples with P2P streamed samples is presented. These results can be applied to other real-time data processing system that requires multiple FPGAs.

Customer behavior analysis using real-time data processing

PurposeThe purpose of this paper is to propose customer behavior analysis based on real-time data processing and association rule for digital signage-based online store (DSOS). The real-time data processing based on big data technology (such as NoSQL MongoDB and Apache Kafka) is utilized to handle the vast amount of customer behavior data.Design/methodology/approachIn order to extract customer behavior patterns, customers’ browsing history and transactional data from digital signage (DS) could be used as the input for decision making. First, the authors developed a DSOS and installed it in different locations, so that customers could have the experience of browsing and buying a product. Second, the real-time data processing system gathered customers’ browsing history and transaction data as it occurred. In addition, the authors utilized the association rule to extract useful information from customer behavior, so it may be used by the managers to efficiently enhance the service quality.FindingsFirst, as the number of customers and DS increases, the proposed system was capable of processing a gigantic amount of input data conveniently. Second, the data set showed that as the number of visit and shopping duration increases, the chance of products being purchased also increased. Third, by combining purchasing and browsing data from customers, the association rules from the frequent transaction pattern were achieved. Thus, the products will have a high possibility to be purchased if they are used as recommendations.Research limitations/implicationsThis research empirically supports the theory of association rule that frequent patterns, correlations or causal relationship found in various kinds of databases. The scope of the present study is limited to DSOS, although the findings can be interpreted and generalized in a global business scenario.Practical implicationsThe proposed system is expected to help management in taking decisions such as improving the layout of the DS and providing better product suggestions to the customer.Social implicationsThe proposed system may be utilized to promote green products to the customer, having a positive impact on sustainability.Originality/valueThe key novelty of the present study lies in system development based on big data technology to handle the enormous amounts of data as well as analyzing the customer behavior in real time in the DSOS. The real-time data processing based on big data technology (such as NoSQL MongoDB and Apache Kafka) is used to handle the vast amount of customer behavior data. In addition, the present study proposed association rule to extract useful information from customer behavior. These results can be used for promotion as well as relevant product recommendations to DSOS customers. Besides in today’s changing retail environment, analyzing the customer behavior in real time in DSOS helps to attract and retain customers more efficiently and effectively, and retailers can get a competitive advantage over their competitors.

Real-time Big Data Processing System to Improve Semiconductor Production Efficiency in Smart Factory

Semiconductor production efficiency is closely related to the defect rate in the production process. The temperature and humidity control in the production line are very important because these affect the defect rate. So many smart factory of semiconductor production uses sensor. It is installed in the semiconductor process, which send huge amounts of data per second to a central server to carry out temperature and humidity control in each production line. However, big data processing systems that analyze and process large-scale data are subject to frequent delays in processing, and transmitted data are lost owing to bottlenecks and insufficient memory caused by traffic concentrated in the central server. In this paper, we propose a real-time big data processing system to improve semiconductor production efficiency. The proposed system consists of a production line collection system, task processing system and data storage system, and improves the productivity of the semiconductor manufacturing process by reducing data processing delays as well as data loss and discarded data.

Data acquisition architecture and online processing system for the HAWC gamma-ray observatory

The High Altitude Water Cherenkov observatory (HAWC) is an air shower array devised for TeV gamma-ray astronomy. HAWC is located at an altitude of 4100 m a.s.l. in Sierra Negra, Mexico. HAWC consists of 300 Water Cherenkov Detectors, each instrumented with 4 photomultiplier tubes (PMTs). HAWC re-uses the Front-End Boards from the Milagro experiment to receive the PMT signals. These boards are used in combination with Time to Digital Converters (TDCs) to record the time and the amount of light in each PMT hit (light flash). A set of VME TDC modules (128 channels each) is operated in a continuous (dead time free) mode. The TDCs are read out via the VME bus by Single-Board Computers (SBCs), which in turn are connected to a gigabit Ethernet network. The complete system produces ≈500 MB/s of raw data. A high-throughput data processing system has been designed and built to enable real-time data analysis. The system relies on off-the-shelf hardware components, an open-source software technology for data transfers (ZeroMQ) and a custom software framework for data analysis (AERIE). Multiple trigger and reconstruction algorithms can be combined and run on blocks of data in a parallel fashion, producing a set of output data streams which can be analyzed in real time with minimal latency (<5 s). This paper provides an overview of the hardware set-up and an in-depth description of the software design, covering both the TDC data acquisition system and the real-time data processing system. The performance of these systems is also discussed.

The Key Technology Research and Application of Re-configurable Real-time Data Process System

In aerospace launching mission, the requirements of the tracking and control system are frequently changing, which cause the high costs of software maintenance and low software reliability. To solve this problem, we research some key technologies in designing a reconfigurable real-time data process system. According to the characteristics of the tracking and control information, an information description model, a processing method model and a processing tree model are presented. Based on these models, re-configurable real-time data processing system architecture and resource configuration methods are proposed. The proposed methods improve the data processing system’s rapid adaption ability to new mission and new processing requirements.

Hyperspectral Imaging for Real-Time Unmanned Aerial Vehicle Maritime Target Detection

This work address hyperspectral imaging systems use for maritime target detection using unmanned aerial vehicles. Specifically, by working in the creation of a hyperspectral real-time data processing system pipeline. We develop a boresight calibration method that allows to calibrate the position of the navigation sensor related to the camera imaging sensor, and improve substantially the accuracy of the target geo-reference. We also develop an unsupervised method for segmenting targets (boats) from their dominant background in real-time. We evaluated the performance of our proposed system for target detection in real-time with UAV flight data and present detection results comparing favorably our approach against other state-of- the-art method.

XSTREAM: A Highly Efficient High Speed Real-time Satellite Data Acquisition and Processing System using Heterogeneous Computing

Abstract. In the last decade, the remote sensing community has observed a significant growth in number of satellites, sensors and their resolutions, thereby increasing the volume of data to be processed each day. Satellite data processing is a complex and time consuming activity. It consists of various tasks, such as decode, decrypt, decompress, radiometric normalization, stagger corrections, ephemeris data processing for geometric corrections etc., and finally writing of the product in the form of an image file. Each task in the processing chain is sequential in nature and has different computing needs. Conventionally the processes are cascaded in a well organized workflow to produce the data products, which are executed on general purpose high-end servers / workstations in an offline mode. Hence, these systems are considered to be ineffective for real-time applications that require quick response and just-intime decision making such as disaster management, home land security and so on. This paper discusses anovel approach to processthe data online (as the data is being acquired) using a heterogeneous computing platform namely XSTREAM which has COTS hardware of CPUs, GPUs and FPGA. This paper focuses on the process architecture, re-engineering aspects and mapping of tasks to the right computing devicewithin the XSTREAM system, which makes it an ideal cost-effective platform for acquiring, processing satellite payload data in real-time and displaying the products in original resolution for quick response. The system has been tested for IRS CARTOSAT and RESOURCESAT series of satellites which have maximum data downlink speed of 210 Mbps.

Design of FPGA based high-speed data acquisition and real-time data processing system on J-TEXT tokamak

Tokamak experiment requires high-speed data acquisition and processing systems. In traditional data acquisition system, the sampling rate, channel numbers and processing speed are limited by bus throughput and CPU speed. This paper presents a data acquisition and processing system based on FPGA. The data can be processed in real-time before it is passed to the CPU. It provides processing ability for more channels with higher sampling rates than the traditional data acquisition system while ensuring deterministic real-time performance. A working prototype is developed for the newly built polarimeter–interferometer diagnostic system on the Joint Texas Experimental Tokamak (J-TEXT). It provides 16 channels with 120MHz maximum sampling rate and 16bit resolution. The onboard FPGA is able to calculate the plasma electron density and Faraday rotation angel. A RAID 5 storage device is adopted providing 700MB/s read–write speed to buffer the data to the hard disk continuously for better performance.

제조 및 유통산업을 위한 RFID 기반 자동 검수 시스템의 설계 및 구현

현재 제조, 유통산업에서는 입출고 운영결과를 수기 작성, 전산 입력 또는 엑셀파일에 의존하는 물류를 운영하고 있다. 이에 수기 입력 오류, 전산 입력 오류, 실시간 데이터 처리 불가, 수량 부족/초과 문제 발생과 같이 물류를 운영하는데 문제가 발생하고 있다. 이에 본 논문은 RFID 기술을 활용하여 입출고 데이터 처리를 통한 입고 자동 검수 시스템을 제안한다. 또한 실시간 데이터 처리 및 자동 검수를 통해 기존 물류 운영 대비 RFID를 적용한 시스템의 비용 절감효과를 제안한다. RFID 기술을 적용하여 입출고 데이터를 현장에서 실시간으로 처리할 수 있으며 수량 부족/초과 문제도 현장에서 바로 조치할 수 있게 된다. 그럼으로써 현재 운영 시스템 대비 RFID 시스템을 통해 물류비를 절감할 수 있다. Current manufacturing and distribution industries work with handwritten shipping information or receive operation results in excel files and have to go through computer processing for their logistics. Thus, hand writing error or computer entry error, non-real-time data processing, quantity shortage and excess operating problems have been frequently occurring in many logistics points. In this paper, we will be proposing a RFID based automatic inspection system that will ensure more accurate data for inbound and outbound. Real-time data processing and automatic inspection system will show cost saving effect compared to the existing system.

Application Research on Real-Time Database System Model Based on Distributed System of Cloud Manufacturing

This paper analyzed on real-time database system, and applied the cloud manufacturing technology and distributed system technology to the database system model. Through the analysis on the real time database, we can got the distributed real-time data system types and data storage model, establish two kinds of memory T-mode and S-mode, database storage model and make the production of block. Finally establish the work flow of distributed real-time database system, get relevant application program. A large number of real-time data collection and administration can be realized, which can improve the real-time information data processing efficiency and system model performance. Through the establishment of model and analysis of its working process in the real-time distributed system real time database application path, it can provide guidance for the establishment of distributed real-time database system model.

Research on Real-Time Data Processing Based on the Linux Shared Memory

In order to improve the efficiency, performance, concurrency of real-time data processing, the paper uses the technology of shared memory, multithreading, double-buffer queue to carry out process of very large-data for real-time by analyzing the structure of Linux shared memory mechanism and traditional real-time data processing system.

Airborne SAR Real-time Imaging Algorithm Design and Implementation with CUDA on NVIDIA GPU

合成孔径雷达(SAR)成像处理的运算量较大，在基于中央处理器(Central Processing Unit, CPU)的工作站或服务器上一般需要耗费较长的时间，无法满足实时性要求。借助于通用并行计算架构(CUDA)编程架构，该文提出一种基于图形处理器(GPU)的SAR 成像处理算法实现方案。该方案解决了GPU 显存不足以容纳一景SAR 数据时数据处理环节与内存/显存间数据传输环节的并行化问题，并能够支持多GPU 设备的并行处理，充分利用了GPU设备的计算资源。在NVIDIA K20C 和INTEL E5645 上的测试表明，与传统基于GPU 的SAR 成像处理算法相比，该方案能够达到数十倍的速度提升，显著降低了处理设备的功耗，提高了处理设备的便携性，能够达到每秒约36兆采样点的实时处理速度。

Design of Real-Time Image Data Acquisition and Processing System Based on SOPC

A real-time image data acquisition and processing system based on SOPC in intelligent cart with automatic tracking function was designed. In this paper, the basic design theory and the system structure was analyzed. The Quartus II and Nios II software was used to make the system with real-time image acquisition function. And the data in SDRAM of two CMOS cameras based on SOPC was read. The design was tested with the system, and results showed that this module had the characters of design flexibility, fast image processing and expansibility.

FPGA+USB-Based Real-Time Data Processing System for Defects Inspection by CCD Scanning

In this paper, the real-time defects inspection was implemented via use of paralleled structure and high-speed operation of FPGA. The hardware circuit based on FPGA was established. According to signal characteristics of polymeric film defects, the preprocessing scheme of defect images based on FPGA was designed. The defect data were packed according to the defined format. Data processed were transferred to PC through USB2.0 real-timely to reconstruct defect microscopic images. The quantity of transferred data was decreased tremendously by this method. The inspecting speed was greatly improved.

Clinical Validation of a Real-Time Data Processing System for Cardiac Output and Arterial Pressure Measurement During Intraoperative Biventricular Pacing Optimization

Biventricular pacing (BiVP) improves cardiac output (CO) and mean arterial pressure (MAP) after cardiopulmonary bypass (CPB) in selected patients at risk for acute left heart failure after cardiac surgery. Optimization of atrioventricular delay (AVD) and interventricular delay (VVD) to maximize the hemodynamic effect of pacing requires rapid and accurate data processing. Conventional post hoc data processing (PP) is accurate but time-consuming, and infeasible in the intraoperative setting. We created a customized, real-time data processing (RTP) system to improve data processing efficiency, while maintaining accuracy. Biventricular pacing optimization was performed within 1 hour of the conclusion of CPB in 10 patients enrolled in the Biventricular Pacing After Cardiac Surgery trial. Cardiac output, measured by an electromagnetic flow meter, and arterial pressure were recorded as AVD was randomly varied across seven settings and VVD across nine settings. Post hoc data processing values calculated by two observers were compared to RTP-generated outputs for CO and MAP. Interexaminer reliability coefficients were generated to access the dependability of RTP. Interexaminer reliability coefficient values ranged from 0.997 to 0.999, indicating RTP is as reliable as PP for optimization. Real-time data processing is instantaneous and therefore is more practical in a clinical setting than the PP method. Real-time data processing is useful for guiding intraoperative BiVP optimization and merits further development.

Characteristics of Reprocessed Hydrometeorological Automated Data System (HADS) Hourly Precipitation Data

Abstract The Hydrometeorological Automated Data System (HADS) is a real-time data acquisition, processing, and distribution system operated by the Office of Hydrologic Development (OHD) of NOAA’s National Weather Service (NWS). The initial reprocessing of HADS data from its original format since its inception in July 1996 has been completed at NOAA’s National Climatic Data Center (NCDC). The quality of the reprocessed HADS hourly precipitation data from rain gauges is assessed by two objective metrics: the average fraction of missing values and the percentage of top-of-the-hour observations for a 3-yr period (2003–05). Pairwise comparisons between the reprocessed product and the real-time product are made using representative samples (about 13%) from the 48 contiguous United States. The monthly average of missing values varies from 0.5% to 2% in the reprocessed product and from 1.7% to 10.1% in the real-time product. Except for January 2003, the reprocessed product consistently reduced missing values, by as much as 9.4% in October 2004. The availability of top-of-the-hour observations is about 85% in the reprocessed product, while the real-time product has top-of-the-hour observations only about 50% of the time. This paper discusses real-time product quality issues, additional quality assurance algorithms used in the reprocessing environment, and the design of system-wide performance comparisons. Thus, the benefits to users of reprocessing the HADS data are the correction of 4-h observation time errors during 1 July–11 August 2005 and the demonstration of diurnals pattern of precipitation frequencies in regional domains. A Web-based interactive quality assessment tool for reprocessed HADS hourly precipitation data and access to the data are also presented.

地震情報の現状と課題

Japan is a seismically active country, and earthquake information is indispensable in Japan. There are various systems or aspects concerning earthquake information, such as earthquake observations on scientific basis, real-time seismic data processing systems, dissemination systems of the results of processed data as earthquake information, and citizens’ understandings and utilization of the information. If the above mentioned whole flow can be called a “mechanism”, Japan is the only one that has the unique and advanced “mechanism” on earthquake information in the world. This mechanism, that can also be called one of the Japanese cultures, has not been built up in a reasonable way from the beginning but it took very long time in establishing the current mechanism. In addition, a new earthquake information provision service has been started that has the nature of earthquake forecast, not observation. Like this, the character of the mechanism has been changing. In this paper, the current Japanese earthquake information is outlined, and its issues are also described.

Failure and Coverage Factors Based Markoff Models: A New Approach for Improving the Dependability Estimation in Complex Fault Tolerant Systems Exposed to SEUs

Dependability estimation of a fault tolerant computer system (FTCS) perturbed by single event upsets (SEUs) requires obtaining first the probability distribution functions for the time to recovery (TTR) and the time to failure (TTF) random variables. The application cross section (sigmaAP) approach does not give directly all the required information. This problem can be solved by means of the construction of suitable Markoff models. In this paper, a new method for constructing such models based on the system's failure and coverage factors is presented. Analytical dependability estimation is consistent with fault injection experiments performed in a fault tolerant operating system developed for a complex, real time data processing system.

Proposal of Hybrid Tsunami Monitoring Network System Consisted of Offshore, Coastal and On-Site Wave Sensors

Development and improvement of the International Tsunami Monitoring Monitoring System is getting more important after the 2004 Sumatra-Off-Earthquake Tsunami disaster. Till now, tsunami monitoring system has been developed and established based no observation net-work of strong ground motion only. Nevertheless, earthquake vibration data may give us incorrect tsunami forecasting, for the strength of the vibration and the tsunami-wave profile basic design of the future tsunami monitoring system using newly developed GPS buoy system and other coastal and on-site sensors. Method of real-time tsunami data processing system is also introduced.