Real-time Data Processing System Research Articles

Real-Time Data Processing Method for CO2 Dispersion Interferometer on EAST

Abstract In this paper, a real-time data processing system is designed for the carbon dioxide dispersion interferometer (CO2-DI) on EAST to measure electron density and provide real-time feedback control. The system utilizes the parallel and pipelining capabilities of Field-Programmable Gate Array (FPGA) to digitize and process the intensity of signals from the detector. Finally, the real-time electron density signals are exported through a Digital-to-Analog Converter (DAC) module in the form of analog signals. The system has been successfully applied in the CO2-DI system to provide low-latency electron density data to the plasma control system on EAST. Experimental results of the latest campaign with long-pulse discharges on EAST (2022-2023) demonstrate that the system can respond effectively in the case of rapid density changes, which means it is reliable and accurate for future electron density calculation.

Development of the logical system for forecasting wind characteristics in the urban conditions

This study is devoted to the development of a logical system to improve the efficiency of wind turbines with the identification of patterns and relationships between wind speeds and directions in urban conditions. The results of the study made it possible to identify approaches to use wind energy efficiently to produce electricity with minimal losses for the power supply of urban facilities. Data Mining methods were applied, the dependencies between weather data were examined. Machine Learning algorithms have also been applied to forecast wind speed and direction in order to increase the efficiency of power generation. A thorough exploratory data analysis was carried out, including visualization of meteorological data and the study of their statistical indicators. To predict wind speed, numerical indicators such as air temperature, wind direction, pressure tendency and atmospheric pressure were taken. Linear Regression, Decision Tree and Support Vector Machine were taken as Machine Learning models. The results revealed that the Random Forest turned out to be the most effective, with a mean squared error of 0.302. According to the results of the research, it was found that the initial data on the characteristics of the wind are highly distorted when it circulates in the urban environment due to the influence of urban development. However, the use of a logical system made it possible to predict and adapt the operation of wind turbines to changes in wind characteristics with minimal losses. A pivotal outcome of this study is the creation of a real-time data processing system, enabling accurate predictions of wind characteristics from speed and direction measuring devices. Additionally, a MATLAB Simulink-based computer model was developed to explore the logical system's influence on wind turbine performance. The findings underscore the efficacy of employing forecasting within a logical system framework to harness wind energy efficiently, contributing to sustainable urban energy solutions

Spark Distributed Real-Time Data and GPU Parallel Computing Based on 5G Virtual Reality

5G virtual reality has attracted many manufacturers and users with its unique immersion, interactivity and imagination characteristics, which has become the focus of new markets. The purpose of this article is to use Spark distributed real-time data system and GPU parallel computing to quickly process and analyze data. This article mainly designs a general-purpose real-time data analysis and processing system based on Spark, which mainly includes new ETL and real-time processing engine modules, and is committed to achieving higher real-time performance than traditional Hadoop. And realize fast calculation. At the same time there is universality and stability. Includes real-time flow calculations. Fast batch processing and machine learning The various types of data computers are included in this article by preparing the cutting device and adjusting the cutting output. The device is ready to effectively terminate the CUDA environment. The cudamalloc function is used to allocate a linear space of bytes to the device, and then transfer the data from the host to the device to determine the number of GPU blocks and threads. GPU parallel computing can increase the data processing speed by 27%, while the secondary programming algorithm can reduce the optimization time of the cup by 12%.

Development of a multifunctional real-time data processing system for interferometers on EAST

In the latest campaign of EAST experiments, two new interferometers were installed, a dispersion interferometer (DI) based on a Carbon Dioxide laser and a solid source interferometer (SSI) based on microwave multiplier sources. To make them available for the Plasma Control System (PCS) system, each of them needs to be provided with a real-time processing system to extract the detector output signal and afterward obtain the electron density information through signal processing. To obtain interferometer data quickly and reliably, a unified hardware template was applied to both interferometers. Three main parts are included in this hardware template — digitization, digital signal processing, and output modules. In particular, the digitization section uses a multi-channel Analog-to-Digital Convertor (ADC) to acquire the signals that need to be calculated. The digital signal processing part is implemented by an FPGA, which is the hardware basis of this multifunctional template that can be used for multiple interferometers. This section includes two main parts, wrapped density signal extraction and signal unwrapping. For DI it is a phase calculation based on the intensity ratio, while for SSI it is a phase calculation by means of FFT or phase demodulation. Both computational approaches form independent IP cores to build systems quickly, while these IP cores provide parameter interfaces to adapt to different application scenarios. The output module provides a variety of data transmission methods, including fiber optic transmission and Digital-Analog-Convertor (DAC) based analog transmission, to interface with PCS or other subsequent systems.This data processing system template has been applied to each of the above two interferometers and valid data were obtained in a recent EAST experiment campaign. It demonstrates the usefulness of the template and provides a reference for the design of data processing systems on future devices.

Robust and provably monotonic networks

The Lipschitz constant of the map between the input and output space represented by a neural network is a natural metric for assessing the robustness of the model. We present a new method to constrain the Lipschitz constant of dense deep learning models that can also be generalized to other architectures. The method relies on a simple weight normalization scheme during training that ensures the Lipschitz constant of every layer is below an upper limit specified by the analyst. A simple monotonic residual connection can then be used to make the model monotonic in any subset of its inputs, which is useful in scenarios where domain knowledge dictates such dependence. Examples can be found in algorithmic fairness requirements or, as presented here, in the classification of the decays of subatomic particles produced at the CERN Large Hadron Collider. Our normalization is minimally constraining and allows the underlying architecture to maintain higher expressiveness compared to other techniques which aim to either control the Lipschitz constant of the model or ensure its monotonicity. We show how the algorithm was used to train a powerful, robust, and interpretable discriminator for heavy-flavor-quark decays, which has been adopted for use as the primary data-selection algorithm in the LHCb real-time data-processing system in the current LHC data-taking period known as Run 3. In addition, our algorithm has also achieved state-of-the-art performance on benchmarks in medicine, finance, and other applications.

Application of real-time data processing system of Internet of Things based on blockchain technology in the financial field of Yangtze River Delta urban agglomeration

After more than 40 years of long-term development, as a pioneer of China's reform and opening up, the Yangtze River Delta region has made tremendous economic progress and strong scientific and technological innovation strength, which has made the economy flourish and scientific and technological innovation strength strong. However, in the process of financial growth in the Yangtze River Delta, there are also financial structures that do not match the optimization of industrial structure, thus hindering the quality and efficiency of industrial structure optimization, thus affecting the strategic function of industrial development. For the development of the financial field, this paper applies the real-time data processing system of the Internet of Things based on blockchain technology to analyze the financial industry of the Yangtze River Delta urban agglomeration. The Internet of Things, based on the technology of real-time data processing system, can store different types of financial assets. These assets are a common component of supporting the application of the Internet of Things. They mix and build a powerful back-end process that can support the data processing application of the Internet of Things. Therefore, the Internet of Things data network and other information are stored in the data structure, so that data analysis and consolidation can be carried out with consensus, Thus, the processing efficiency of relevant financing data in the Yangtze River Delta region has been greatly improved, and the development process of the financial sector in the region has been effectively accelerated. Therefore, the research on this issue will enable the financial industry to give full play to its potential as a major component of national economic development, so as to maximize the improvement of the industrial structure and promote the development of blockchain and Internet of Things technologies.

Quantitative evaluation of attraction intensity of highway landscape visual elements based on dynamic perception

In order to quantify the aesthetic attraction of visual elements of highway landscape space, the concept of visual attraction intensity of highway landscape space is proposed, and the factors and indicators of visual attraction elements are clarified. Based on the perspective of human factors, taking passengers as the main attraction, taking 30 landscape elements from three typical highway landscape scenes, namely mountain highway landscape, plain highway landscape and elevated highway landscape, as the research object. The real-time evaluation and data processing system of subjective landscape preference has been independently developed, and combined with the processing and analysis of human-machine-environment synchronous quantitative analysis technology and in-depth learning, a “subjective+objective” quantitative evaluation model of highway landscape visual element attraction intensity based on dynamic perception has been constructed, revealing the quantitative relationship between highway landscape visual attraction elements and attraction intensity, The quantitative method and action law of the attraction intensity of highway landscape and the dynamic aesthetic perception of the attraction subject is clarified. It is found that the change of highway planning alignment, high green view rate and low enclosure have positive effects on the subjective preference score of the landscape (0.690), and there is a significant positive correlation between green view rate and enclosure; the richer the landscape elements such as mountains, trees and water bodies, the stronger the visual attraction, but the complexity of the landscape elements increases the cognitive load of the area of interest. Among the physiological indicators, there are significant correlations between EEG and ECG (0.790), EDA and ECG (0.632), and subjective preference and passengers' eye-movement behavior and EDA; as well as the change pattern of attraction intensity based on the dynamic perception of tunnel area. The results show that the attraction intensity of visual elements of highway landscape space based on dynamic perception is consistent with human aesthetic preference, visual perception and physiological response, which verifies the scientificity, appropriateness and accuracy of eye-movement analysis method and physiological analysis method for quantitative evaluation of aesthetic preference of highway landscape.

Healthcare Monitoring System for Diabetic Patients Using Machine Learning

Abstract: Present-day technology provides quite an efficient way of monitoring an individual’s health. Bluetooth Low Energy (BLE)-based sensors can be considered as one of the solutions for checking personal vital signs data such as blood pressure, heart rate, weight, and blood glucose (BG). In this study, we propose a personalized healthcare monitoring system by utilizing a BLE-based sensor device dataset, data processing, and machine learning-based algorithms to help diabetic patients to better self-manage their chronic condition. The proposed real-time data processing system utilizes machine Learning algorithms to train the model. Machine learning–based classification methods were tested on a diabetes dataset in order to show that a Multilayer Perceptron can provide early prediction of diabetes given the user’s sensor data as input. Furthermore, the proposed diabetes classification and prediction might be integrated with individualized diet and physical activity recommendations to improve patients' health quality and avert severe circumstances in the future.

High-precision luminosity instrumentation for the CMS detector at the HL-LHC

The high-luminosity upgrade of the LHC (HL-LHC) is foreseen to reach an instantaneous luminosity a factor of five to seven times the nominal LHC design value. This creates the need for new high-precision instrumentation at CMS, using radiation-hard detector technologies. This contribution introduces the instrumentation for bunch-by-bunch online luminosity and beam-induced background measurements in CMS. The Tracker Endcap Pixel Detector (TEPX) will be adapted for precision luminometry by implementing a dedicated trigger type and a real-time data processing system performing clustering on FPGA. The innermost ring of the last TEPX layer (D4R1) will be operated independently from the rest of TEPX, enabling beam monitoring during the LHC ramp and during unqualified beam conditions with a dedicated timing and trigger infrastructure. A new standalone luminometer, the Fast Beam Condition Monitor (FBCM), which will be fully independent from the central trigger and data acquisition services and will be able to operate during all times is also introduced. FBCM is foreseen to utilize silicon-pad sensors with sub-bunch-crossing time resolution enabling also the measurement of beam-induced background. The potential of the exploitation of other CMS subsystems such as the outer tracker, the barrel muon detectors and the 40 MHz L1 trigger scouting system using a common histogramming firmware is also discussed.

Sboing4Real: A real-time crowdsensing-based traffic management system

This work describes the architecture of the back-end engine of a real-time traffic data processing and satellite navigation system. The role of the engine is to process real-time feedback, such as speed and travel time, provided by in-vehicle devices and derive real-time reports and traffic predictions through leveraging historical data as well. We present the main building blocks and the versatile set of data sources and processing platforms that need to be combined together to form a fully-functional and scalable solution. We also present performance results focusing on meeting system requirements while keeping the need for computing resources low. The lessons and results presented are of value to additional real-time applications that rely on both recent and historical data. Finally, we discuss the application of the aforementioned solution to a successful pilot study, where the full system was deployed and processed data from 800 taxis for a period of 3 months.

Design and Implementation of Real Time Data Processing System Based on Spark Streaming

With the rapid development of Internet technology, people have more abundant ways to obtain information through the Internet.More and more information is spread in the network, which will produce a large amount of data.The big data type is more complicated. In order to implement data implementation, the industry is generally combined with different data types and business scenes, special development and design of different processing subsystems.This method has a large research cost and learning cost, and a unified computing system platform is relatively lacking, which further makes developers unable to maintain and expand the system formed by different sets of technology systems.Based on the above situation, this paper designs a universal real-time data analysis and processing system based on Spark Streaming.

Real-Time Data Acquisition and Processing System for MHz Repetition Rate Image Sensors

An electro-optic detector is one of the diagnostic setups used in particle accelerators. It employs an electro-optic crystal to encode the longitudinal beam charge profile in the spectrum of a light pulse. The charge distribution is then reconstructed using data captured by a fast spectrometer. The measurement repetition rate should match or exceed the machine bunching frequency, which is often in the range of several MHz. A high-speed optical line detector (HOLD) is a linear camera designed for easy integration with scientific experiments. The use of modern FPGA circuits helps in the efficient collection and processing of data. The solution is based on Xilinx 7-Series FPGA circuits and implements a custom latency-optimized architecture utilizing the AXI4 family of interfaces. HOLD is one of the fastest line cameras in the world. Thanks to its hardware architecture and a powerful KALYPSO sensor from KIT, it outperforms the fastest comparable commercial devices.

A New Method for Compaction Quality Evaluation of Asphalt Mixtures with the Intelligent Aggregate (IA).

To provide a new method for the evaluation of the compaction quality of asphalt mixture, a real-time data acquisition and processing system (RDAPS) for the motion state of aggregate with a small volume and high precision is developed. The system consists of an intelligent aggregate (IA), analysis software and hardware equipment. The performance of the IA was tested by regarding data sensitivity, high-temperature resistance, and mechanical properties. A new evaluation method was proposed for evaluating the compaction quality of AC-25 and SMA-25 asphalt mixtures based on an IA. The results show that the best transmission baud rate for the IA was 9600 bps, and the corresponding signal transmission distance was 380 m. Only one IA was needed to complete the state data collection for the aggregate within the asphalt mixture in a circular area, with the IA layout point as the center of the circle and a radius of 5 m. The IA conducted reliable data transmission up to 200 °C; however, its compressive strength decreased with increasing temperature until reaching stability. Traditional aggregate could be replaced by an IA to withstand external forces and internal load transfer. Embedding an IA into AC-25 or SMA-25 asphalt mixtures did not have a significant impact on the original mechanical properties of the mixture. The effect of the gradation type of the asphalt mixture on the IA motion state was not significant. When the compaction degree met the specification requirements, the motion data of the IA did not reach a stable state, and the interlocking effect between aggregates in the asphalt mixture could be further optimized. An evaluation method is proposed based on the IA for the compaction quality of AC-25 and SMA-25 asphalt mixtures with the compaction degree as the main index and the spatial attitude angle and spatial acceleration of the IA as the auxiliary indexes.

Research on Application of Computer Big Data in Mathematical Polymorphic Game Algorithm Theory

In large-scale game activities, the computer big data system is collected, and then large-scale real-time data collection, processing and control systems are used to bring mathematical algorithm models, and state information reliability algorithms are used to illustrate the system control machine design and its Implementation based on VMS. This article calculates the most by calculation. The main design ideas in the game mathematical model, the research found that the system controller proposed by the system is a method to solve the multi-state process synchronization problem in the Client/Server computing system, and can effectively deal with the algorithm application research in the model.

DOA Estimation for Coherent Signals With Improved Sparse Representation in the Presence of Unknown Spatially Correlated Gaussian Noise

In this paper, a new method is proposed for direction-of-arrival (DOA) estimation of coherent signals with improved sparse representation in unknown spatially correlated Gaussian noise. To be specific, leveraging a symmetric uniform linear array, the entries of the signal covariance matrix is firstly recasted to eliminate the spatially correlated noise. Subsequently, it is shown that an equivalent source vector can be obtained by squaring any row of the noise-free signal covariance matrix, irrespective of the coherency between the signals. Finally, an improved sparse representation, which enhances signal sparsity via utilizing a designed weight vector, is derived to determine the DOAs of the signals. Numerical examples are provided to demonstrate its superiority of DOA estimation performance in low signal-to-noise ratio (SNR) environments. Besides, it is computationally efficient, which is critical for large array and/or real-time data processing systems.

Palomar Gattini-IR: Survey Overview, Data Processing System, On-sky Performance and First Results

Palomar Gattini-IR is a new wide-field, near-infrared (NIR) robotic time domain survey operating at Palomar Observatory. Using a 30 cm telescope mounted with a H2RG detector, Gattini-IR achieves a field of view (FOV) of 25 sq. deg. with a pixel scale of 8.″7 in J-band. Here, we describe the system design, survey operations, data processing system and on-sky performance of Palomar Gattini-IR. As a part of the nominal survey, Gattini-IR scans ≈7500 square degrees of the sky every night to a median 5σ depth of 15.7 AB mag outside the Galactic plane. The survey covers ≈15,000 square degrees of the sky visible from Palomar with a median cadence of 2 days. A real-time data processing system produces stacked science images from dithered raw images taken on sky, together with point-spread function (PSF)-fit source catalogs and transient candidates identified from subtractions within a median delay of ≈4 hr from the time of observation. The calibrated data products achieve an astrometric accuracy (rms) of ≈0.″7 with respect to Gaia DR2 for sources with signal-to-noise ratio > 10, and better than ≈0.″35 for sources brighter than ≈12 Vega mag. The photometric accuracy (rms) achieved in the PSF-fit source catalogs is better than ≈3% for sources brighter than ≈12 Vega mag and fainter than the saturation magnitude of ≈8.5 Vega mag, as calibrated against the Two Micron All Sky Survey catalog. The detection efficiency of transient candidates injected into the images is better than 90% for sources brighter than the 5σ limiting magnitude. The photometric recovery precision of injected sources is 3% for sources brighter than 13 mag, and the astrometric recovery rms is ≈0.″9. Reference images generated by stacking several field visits achieve depths of ≳16.5 AB mag over 60% of the sky, while it is limited by confusion in the Galactic plane. With a FOV ≈40× larger than any other existing NIR imaging instrument, Gattini-IR is probing the reddest and dustiest transients in the local universe such as dust obscured supernovae in nearby galaxies, novae behind large columns of extinction within the galaxy, reddened microlensing events in the Galactic plane and variability from cool and dust obscured stars. We present results from transients and variables identified since the start of the commissioning period.

Real-time fast Fourier transform

The paper presents a method of implementation of fast Fourier transform in real-time data processing and measurement systems. The method is based on software and hardware package «UMIKON», supporting input-output drivers. Classic Cooley and Tukey algorithm apply to real-time data vector, which formed by successive shift after every clock cycle time. This approach to data processing in real-time allows reaching high access time in information measurement systems when analyzing dynamic processes.

Real-time intelligent big data processing: technology, platform, and applications

Human beings keep exploring the physical space using information means. Only recently, with the rapid development of information technologies and the increasing accumulation of data, human beings can learn more about the unknown world with data-driven methods. Given data timeliness, there is a growing awareness of the importance of real-time data. There are two categories of technologies accounting for data processing: batching big data and streaming processing, which have not been integrated well. Thus, we propose an innovative incremental processing technology named after Stream Cube to process both big data and stream data. Also, we implement a real-time intelligent data processing system, which is based on real-time acquisition, real-time processing, real-time analysis, and real-time decision-making. The real-time intelligent data processing technology system is equipped with a batching big data platform, data analysis tools, and machine learning models. Based on our applications and analysis, the real-time intelligent data processing system is a crucial solution to the problems of the national society and economy.

NI P2P powered multi FPGA real-time data processing on tokamak diagnosis systems

On tokamaks, there are many diagnostics, which need real-time data acquisition and processing to provide useful information for plasma control. Some of the diagnostics required fast processing of multiple very high sampling rate signals. It is often difficult to achieve even with modern multi-core CPUs or GPUs. Moving large amount of data from the digitizers to the system ram will hurt the deterministic performance significantly. FPGA data pre-processing is a better choice when the data large and deterministic performance is required. When the task is too heavy for a single FPGA to implement, transfer large data between FPGAs is also required. On the polarimeter-interferometer diagnostics on J-TEXT, in order to calculate the density profile of the plasma in real-time, multiple chords of line-integrated density must be calculated first. The data acquisition system for the polarimeter-interferometer is equipped with 5 FlexRIO FPGAs, only one is acquiring the reference signal. Density calculation for every probing chord needs the reference signal. So this signal needs to be transferred to other FPGAs with strict real-time requirement. Mismatching a single sample, the result would be completely incorrect. NI P2P streaming allow the FPGA to transfer data without burdening the process. In the paper, the determinism characteristic and other performance of the P2P streaming is tested, and a mechanism to align local digitized samples with P2P streamed samples is presented. These results can be applied to other real-time data processing system that requires multiple FPGAs.

Customer behavior analysis using real-time data processing

PurposeThe purpose of this paper is to propose customer behavior analysis based on real-time data processing and association rule for digital signage-based online store (DSOS). The real-time data processing based on big data technology (such as NoSQL MongoDB and Apache Kafka) is utilized to handle the vast amount of customer behavior data.Design/methodology/approachIn order to extract customer behavior patterns, customers’ browsing history and transactional data from digital signage (DS) could be used as the input for decision making. First, the authors developed a DSOS and installed it in different locations, so that customers could have the experience of browsing and buying a product. Second, the real-time data processing system gathered customers’ browsing history and transaction data as it occurred. In addition, the authors utilized the association rule to extract useful information from customer behavior, so it may be used by the managers to efficiently enhance the service quality.FindingsFirst, as the number of customers and DS increases, the proposed system was capable of processing a gigantic amount of input data conveniently. Second, the data set showed that as the number of visit and shopping duration increases, the chance of products being purchased also increased. Third, by combining purchasing and browsing data from customers, the association rules from the frequent transaction pattern were achieved. Thus, the products will have a high possibility to be purchased if they are used as recommendations.Research limitations/implicationsThis research empirically supports the theory of association rule that frequent patterns, correlations or causal relationship found in various kinds of databases. The scope of the present study is limited to DSOS, although the findings can be interpreted and generalized in a global business scenario.Practical implicationsThe proposed system is expected to help management in taking decisions such as improving the layout of the DS and providing better product suggestions to the customer.Social implicationsThe proposed system may be utilized to promote green products to the customer, having a positive impact on sustainability.Originality/valueThe key novelty of the present study lies in system development based on big data technology to handle the enormous amounts of data as well as analyzing the customer behavior in real time in the DSOS. The real-time data processing based on big data technology (such as NoSQL MongoDB and Apache Kafka) is used to handle the vast amount of customer behavior data. In addition, the present study proposed association rule to extract useful information from customer behavior. These results can be used for promotion as well as relevant product recommendations to DSOS customers. Besides in today’s changing retail environment, analyzing the customer behavior in real time in DSOS helps to attract and retain customers more efficiently and effectively, and retailers can get a competitive advantage over their competitors.