Real-time Data Processing System Research Articles

Time series forecasting of price of the agricultural products using data science

Purpose. The purpose of our article is to research and forecast prices for agricultural products using the example of potato prices based on the most effective models using data science techniques. Methodology / approach. Various forecasting models are explored, starting from baseline models like decomposition and exponential smoothing models to more advanced techniques such as ARIMA, SARIMA, as well as deep learning models including neural network. The data is split into training and testing sets, and models are validated using cross-validation techniques and optimised through hyperparameter tuning. Model performance is evaluated using metrics such as MAE, MSE, RMSE, and MAPE. The selected model is then used to generate future price forecasts, with uncertainty quantified through confidence intervals. Results. The study successfully applied advanced data science techniques to forecast potato prices, leveraging a range of effective models. By analysing historical price data and using various forecasting methods, the research identified the most accurate models for predicting future price trends. The results demonstrate that the selected models can provide reliable forecasts. In particular, the results showed that the model could achieve good forecast results when applied to real problems and, thus, can be effectively used for forecasting tasks especially considering seasonality. In addition, it should be noted that the model has a higher prediction accuracy at the time intervals closest to the original data. The obtained results support using both models simultaneously for forecasting, which can compensate for the shortcomings of each of them. The models can be used separately, to more accurately predict the values for the required period, or a combination of them is also possible. Originality / scientific novelty. The study’s originality lies in development of methods for effectively accounting for seasonality in agricultural price data, such as using seasonal decomposition techniques or more advanced techniques that combine statistical and data science approaches. The novelty implies the implementation of real-time data processing and forecasting system allows for the timely prediction of price changes, enabling stakeholders to make more informed decisions. Practical value / implications. Forecasting potato prices holds significant practical value for various stakeholders. For farmers, accurate forecasts enable informed decisions on the optimal times to plant, harvest, and sell their crops, thereby optimising their profits. In the supply chain, distributors and retailers can use these forecasts to manage inventory more effectively and plan contracts, reducing waste and avoiding shortages. Policymakers benefit from forecasts by anticipating market fluctuations and stabilising prices, which supports both consumers and producers. For consumers, stable pricing ensures better budgeting and helps avoid sudden price spikes, making essential foods more affordable. Overall, accurate price forecasting enhances market efficiency by reducing uncertainty and aiding investors in managing risk.

Retraction Note: Application of real-time data processing system of internet of things based on blockchain technology in the financial field of Yangtze River Delta urban agglomeration

Retraction Note: Application of real-time data processing system of internet of things based on blockchain technology in the financial field of Yangtze River Delta urban agglomeration

Real-time data processing method for CO2 dispersion interferometer on EAST

A real-time data processing system is designed for the carbon dioxide dispersion interferometer (CO2-DI) on EAST. The system utilizes the parallel and pipelining capabilities of an field-programmable gate array (FPGA) to digitize and process the intensity of signals from the detector. Finally, the real-time electron density signals are exported through a digital-to-analog converter (DAC) module in the form of analog signals. The system has been successfully applied in the CO2-DI system to provide low-latency electron density input to the plasma control system on EAST. Experimental results of the latest campaign with long-pulse discharges on EAST (2022–2023) demonstrate that the system can respond effectively in the case of rapid density changes, proving its reliability and accuracy for future electron density calculation.

AI Revolution in Exit Polls: Enhancing Accuracy, Efficiency, and Inclusivity

Exit polls have long been integral to understanding electoral dynamics, offering insights into voter behavior and preferences. Traditional exit poll methodologies, involving manual data collection and in-person interviews, face significant challenges in accuracy, sample size, and inclusivity. This paper explores the transformative impact of Artificial Intelligence (AI) on the exit polling process. By leveraging neural networks for enhanced sampling, employing automated Voice-AI for data collection, and utilizing Natural Language Processing (NLP) for real-time data analysis, AI significantly improves the efficiency, accuracy, and inclusivity of exit polls. The paper discusses the technical aspects of implementing AI-driven exit polls, including the architecture of neural networks, real-time data processing systems, and machine learning models for prediction. Additionally, it addresses the advantages of AI, such as cost-effectiveness and reduced human biases, while acknowledging challenges like algorithmic bias, privacy concerns, and the complexities of accurately interpreting voter behavior. As AI continues to evolve, it promises to further enhance the reliability and comprehensiveness of exit polls, providing valuable tools for understanding public opinion in democratic processes

Development of the logical system for forecasting wind characteristics in the urban conditions

This study is devoted to the development of a logical system to improve the efficiency of wind turbines with the identification of patterns and relationships between wind speeds and directions in urban conditions. The results of the study made it possible to identify approaches to use wind energy efficiently to produce electricity with minimal losses for the power supply of urban facilities. Data Mining methods were applied, the dependencies between weather data were examined. Machine Learning algorithms have also been applied to forecast wind speed and direction in order to increase the efficiency of power generation. A thorough exploratory data analysis was carried out, including visualization of meteorological data and the study of their statistical indicators. To predict wind speed, numerical indicators such as air temperature, wind direction, pressure tendency and atmospheric pressure were taken. Linear Regression, Decision Tree and Support Vector Machine were taken as Machine Learning models. The results revealed that the Random Forest turned out to be the most effective, with a mean squared error of 0.302. According to the results of the research, it was found that the initial data on the characteristics of the wind are highly distorted when it circulates in the urban environment due to the influence of urban development. However, the use of a logical system made it possible to predict and adapt the operation of wind turbines to changes in wind characteristics with minimal losses. A pivotal outcome of this study is the creation of a real-time data processing system, enabling accurate predictions of wind characteristics from speed and direction measuring devices. Additionally, a MATLAB Simulink-based computer model was developed to explore the logical system's influence on wind turbine performance. The findings underscore the efficacy of employing forecasting within a logical system framework to harness wind energy efficiently, contributing to sustainable urban energy solutions

Spark Distributed Real-Time Data and GPU Parallel Computing Based on 5G Virtual Reality

5G virtual reality has attracted many manufacturers and users with its unique immersion, interactivity and imagination characteristics, which has become the focus of new markets. The purpose of this article is to use Spark distributed real-time data system and GPU parallel computing to quickly process and analyze data. This article mainly designs a general-purpose real-time data analysis and processing system based on Spark, which mainly includes new ETL and real-time processing engine modules, and is committed to achieving higher real-time performance than traditional Hadoop. And realize fast calculation. At the same time there is universality and stability. Includes real-time flow calculations. Fast batch processing and machine learning The various types of data computers are included in this article by preparing the cutting device and adjusting the cutting output. The device is ready to effectively terminate the CUDA environment. The cudamalloc function is used to allocate a linear space of bytes to the device, and then transfer the data from the host to the device to determine the number of GPU blocks and threads. GPU parallel computing can increase the data processing speed by 27%, while the secondary programming algorithm can reduce the optimization time of the cup by 12%.

Development of a multifunctional real-time data processing system for interferometers on EAST

In the latest campaign of EAST experiments, two new interferometers were installed, a dispersion interferometer (DI) based on a Carbon Dioxide laser and a solid source interferometer (SSI) based on microwave multiplier sources. To make them available for the Plasma Control System (PCS) system, each of them needs to be provided with a real-time processing system to extract the detector output signal and afterward obtain the electron density information through signal processing. To obtain interferometer data quickly and reliably, a unified hardware template was applied to both interferometers. Three main parts are included in this hardware template — digitization, digital signal processing, and output modules. In particular, the digitization section uses a multi-channel Analog-to-Digital Convertor (ADC) to acquire the signals that need to be calculated. The digital signal processing part is implemented by an FPGA, which is the hardware basis of this multifunctional template that can be used for multiple interferometers. This section includes two main parts, wrapped density signal extraction and signal unwrapping. For DI it is a phase calculation based on the intensity ratio, while for SSI it is a phase calculation by means of FFT or phase demodulation. Both computational approaches form independent IP cores to build systems quickly, while these IP cores provide parameter interfaces to adapt to different application scenarios. The output module provides a variety of data transmission methods, including fiber optic transmission and Digital-Analog-Convertor (DAC) based analog transmission, to interface with PCS or other subsequent systems.This data processing system template has been applied to each of the above two interferometers and valid data were obtained in a recent EAST experiment campaign. It demonstrates the usefulness of the template and provides a reference for the design of data processing systems on future devices.

Robust and provably monotonic networks

The Lipschitz constant of the map between the input and output space represented by a neural network is a natural metric for assessing the robustness of the model. We present a new method to constrain the Lipschitz constant of dense deep learning models that can also be generalized to other architectures. The method relies on a simple weight normalization scheme during training that ensures the Lipschitz constant of every layer is below an upper limit specified by the analyst. A simple monotonic residual connection can then be used to make the model monotonic in any subset of its inputs, which is useful in scenarios where domain knowledge dictates such dependence. Examples can be found in algorithmic fairness requirements or, as presented here, in the classification of the decays of subatomic particles produced at the CERN Large Hadron Collider. Our normalization is minimally constraining and allows the underlying architecture to maintain higher expressiveness compared to other techniques which aim to either control the Lipschitz constant of the model or ensure its monotonicity. We show how the algorithm was used to train a powerful, robust, and interpretable discriminator for heavy-flavor-quark decays, which has been adopted for use as the primary data-selection algorithm in the LHCb real-time data-processing system in the current LHC data-taking period known as Run 3. In addition, our algorithm has also achieved state-of-the-art performance on benchmarks in medicine, finance, and other applications.

Application of real-time data processing system of Internet of Things based on blockchain technology in the financial field of Yangtze River Delta urban agglomeration

After more than 40 years of long-term development, as a pioneer of China's reform and opening up, the Yangtze River Delta region has made tremendous economic progress and strong scientific and technological innovation strength, which has made the economy flourish and scientific and technological innovation strength strong. However, in the process of financial growth in the Yangtze River Delta, there are also financial structures that do not match the optimization of industrial structure, thus hindering the quality and efficiency of industrial structure optimization, thus affecting the strategic function of industrial development. For the development of the financial field, this paper applies the real-time data processing system of the Internet of Things based on blockchain technology to analyze the financial industry of the Yangtze River Delta urban agglomeration. The Internet of Things, based on the technology of real-time data processing system, can store different types of financial assets. These assets are a common component of supporting the application of the Internet of Things. They mix and build a powerful back-end process that can support the data processing application of the Internet of Things. Therefore, the Internet of Things data network and other information are stored in the data structure, so that data analysis and consolidation can be carried out with consensus, Thus, the processing efficiency of relevant financing data in the Yangtze River Delta region has been greatly improved, and the development process of the financial sector in the region has been effectively accelerated. Therefore, the research on this issue will enable the financial industry to give full play to its potential as a major component of national economic development, so as to maximize the improvement of the industrial structure and promote the development of blockchain and Internet of Things technologies.

Quantitative evaluation of attraction intensity of highway landscape visual elements based on dynamic perception

In order to quantify the aesthetic attraction of visual elements of highway landscape space, the concept of visual attraction intensity of highway landscape space is proposed, and the factors and indicators of visual attraction elements are clarified. Based on the perspective of human factors, taking passengers as the main attraction, taking 30 landscape elements from three typical highway landscape scenes, namely mountain highway landscape, plain highway landscape and elevated highway landscape, as the research object. The real-time evaluation and data processing system of subjective landscape preference has been independently developed, and combined with the processing and analysis of human-machine-environment synchronous quantitative analysis technology and in-depth learning, a “subjective+objective” quantitative evaluation model of highway landscape visual element attraction intensity based on dynamic perception has been constructed, revealing the quantitative relationship between highway landscape visual attraction elements and attraction intensity, The quantitative method and action law of the attraction intensity of highway landscape and the dynamic aesthetic perception of the attraction subject is clarified. It is found that the change of highway planning alignment, high green view rate and low enclosure have positive effects on the subjective preference score of the landscape (0.690), and there is a significant positive correlation between green view rate and enclosure; the richer the landscape elements such as mountains, trees and water bodies, the stronger the visual attraction, but the complexity of the landscape elements increases the cognitive load of the area of interest. Among the physiological indicators, there are significant correlations between EEG and ECG (0.790), EDA and ECG (0.632), and subjective preference and passengers' eye-movement behavior and EDA; as well as the change pattern of attraction intensity based on the dynamic perception of tunnel area. The results show that the attraction intensity of visual elements of highway landscape space based on dynamic perception is consistent with human aesthetic preference, visual perception and physiological response, which verifies the scientificity, appropriateness and accuracy of eye-movement analysis method and physiological analysis method for quantitative evaluation of aesthetic preference of highway landscape.

Healthcare Monitoring System for Diabetic Patients Using Machine Learning

Abstract: Present-day technology provides quite an efficient way of monitoring an individual’s health. Bluetooth Low Energy (BLE)-based sensors can be considered as one of the solutions for checking personal vital signs data such as blood pressure, heart rate, weight, and blood glucose (BG). In this study, we propose a personalized healthcare monitoring system by utilizing a BLE-based sensor device dataset, data processing, and machine learning-based algorithms to help diabetic patients to better self-manage their chronic condition. The proposed real-time data processing system utilizes machine Learning algorithms to train the model. Machine learning–based classification methods were tested on a diabetes dataset in order to show that a Multilayer Perceptron can provide early prediction of diabetes given the user’s sensor data as input. Furthermore, the proposed diabetes classification and prediction might be integrated with individualized diet and physical activity recommendations to improve patients' health quality and avert severe circumstances in the future.

High-precision luminosity instrumentation for the CMS detector at the HL-LHC

The high-luminosity upgrade of the LHC (HL-LHC) is foreseen to reach an instantaneous luminosity a factor of five to seven times the nominal LHC design value. This creates the need for new high-precision instrumentation at CMS, using radiation-hard detector technologies. This contribution introduces the instrumentation for bunch-by-bunch online luminosity and beam-induced background measurements in CMS. The Tracker Endcap Pixel Detector (TEPX) will be adapted for precision luminometry by implementing a dedicated trigger type and a real-time data processing system performing clustering on FPGA. The innermost ring of the last TEPX layer (D4R1) will be operated independently from the rest of TEPX, enabling beam monitoring during the LHC ramp and during unqualified beam conditions with a dedicated timing and trigger infrastructure. A new standalone luminometer, the Fast Beam Condition Monitor (FBCM), which will be fully independent from the central trigger and data acquisition services and will be able to operate during all times is also introduced. FBCM is foreseen to utilize silicon-pad sensors with sub-bunch-crossing time resolution enabling also the measurement of beam-induced background. The potential of the exploitation of other CMS subsystems such as the outer tracker, the barrel muon detectors and the 40 MHz L1 trigger scouting system using a common histogramming firmware is also discussed.

Sboing4Real: A real-time crowdsensing-based traffic management system

This work describes the architecture of the back-end engine of a real-time traffic data processing and satellite navigation system. The role of the engine is to process real-time feedback, such as speed and travel time, provided by in-vehicle devices and derive real-time reports and traffic predictions through leveraging historical data as well. We present the main building blocks and the versatile set of data sources and processing platforms that need to be combined together to form a fully-functional and scalable solution. We also present performance results focusing on meeting system requirements while keeping the need for computing resources low. The lessons and results presented are of value to additional real-time applications that rely on both recent and historical data. Finally, we discuss the application of the aforementioned solution to a successful pilot study, where the full system was deployed and processed data from 800 taxis for a period of 3 months.

Design and Implementation of Real Time Data Processing System Based on Spark Streaming

With the rapid development of Internet technology, people have more abundant ways to obtain information through the Internet.More and more information is spread in the network, which will produce a large amount of data.The big data type is more complicated. In order to implement data implementation, the industry is generally combined with different data types and business scenes, special development and design of different processing subsystems.This method has a large research cost and learning cost, and a unified computing system platform is relatively lacking, which further makes developers unable to maintain and expand the system formed by different sets of technology systems.Based on the above situation, this paper designs a universal real-time data analysis and processing system based on Spark Streaming.

Real-Time Data Acquisition and Processing System for MHz Repetition Rate Image Sensors

An electro-optic detector is one of the diagnostic setups used in particle accelerators. It employs an electro-optic crystal to encode the longitudinal beam charge profile in the spectrum of a light pulse. The charge distribution is then reconstructed using data captured by a fast spectrometer. The measurement repetition rate should match or exceed the machine bunching frequency, which is often in the range of several MHz. A high-speed optical line detector (HOLD) is a linear camera designed for easy integration with scientific experiments. The use of modern FPGA circuits helps in the efficient collection and processing of data. The solution is based on Xilinx 7-Series FPGA circuits and implements a custom latency-optimized architecture utilizing the AXI4 family of interfaces. HOLD is one of the fastest line cameras in the world. Thanks to its hardware architecture and a powerful KALYPSO sensor from KIT, it outperforms the fastest comparable commercial devices.

Map Matching Algorithm for Real-Time Data Processing of Non-route GPS Data in Seoul

Global Positioning System (GPS) data are satellite-based longitude and latitude coordinates for which map matching with road links, which are electronic maps, may be used in traffic operation management. Various studies have been conducted to improve the accuracy of map matching; however, research on the application of map matching algorithms for real-time data processing systems has not been widely conducted. Therefore, this study investigated a map matching algorithm that can shorten the processing time required for application to systems that process large volumes of GPS data collected for a wide network. To reduce the processing time, a candidate link selection method and a grid-type geospatial indexing process were designed. The processing time was analyzed based on an emulator before applying the map matching algorithm to the Transport Operation and Information Service Center, which generates the real-time travel speed data in Seoul City. The results obtained from comparing the prior and new algorithms showed a 96% decrease in the processing time. We believe that the findings of this study will benefit the real-time processing of GPS data, which are expected to be used in various ways in traffic operations and management in the future.

A New Method for Compaction Quality Evaluation of Asphalt Mixtures with the Intelligent Aggregate (IA).

To provide a new method for the evaluation of the compaction quality of asphalt mixture, a real-time data acquisition and processing system (RDAPS) for the motion state of aggregate with a small volume and high precision is developed. The system consists of an intelligent aggregate (IA), analysis software and hardware equipment. The performance of the IA was tested by regarding data sensitivity, high-temperature resistance, and mechanical properties. A new evaluation method was proposed for evaluating the compaction quality of AC-25 and SMA-25 asphalt mixtures based on an IA. The results show that the best transmission baud rate for the IA was 9600 bps, and the corresponding signal transmission distance was 380 m. Only one IA was needed to complete the state data collection for the aggregate within the asphalt mixture in a circular area, with the IA layout point as the center of the circle and a radius of 5 m. The IA conducted reliable data transmission up to 200 °C; however, its compressive strength decreased with increasing temperature until reaching stability. Traditional aggregate could be replaced by an IA to withstand external forces and internal load transfer. Embedding an IA into AC-25 or SMA-25 asphalt mixtures did not have a significant impact on the original mechanical properties of the mixture. The effect of the gradation type of the asphalt mixture on the IA motion state was not significant. When the compaction degree met the specification requirements, the motion data of the IA did not reach a stable state, and the interlocking effect between aggregates in the asphalt mixture could be further optimized. An evaluation method is proposed based on the IA for the compaction quality of AC-25 and SMA-25 asphalt mixtures with the compaction degree as the main index and the spatial attitude angle and spatial acceleration of the IA as the auxiliary indexes.

Research on Application of Computer Big Data in Mathematical Polymorphic Game Algorithm Theory

In large-scale game activities, the computer big data system is collected, and then large-scale real-time data collection, processing and control systems are used to bring mathematical algorithm models, and state information reliability algorithms are used to illustrate the system control machine design and its Implementation based on VMS. This article calculates the most by calculation. The main design ideas in the game mathematical model, the research found that the system controller proposed by the system is a method to solve the multi-state process synchronization problem in the Client/Server computing system, and can effectively deal with the algorithm application research in the model.

DOA Estimation for Coherent Signals With Improved Sparse Representation in the Presence of Unknown Spatially Correlated Gaussian Noise

In this paper, a new method is proposed for direction-of-arrival (DOA) estimation of coherent signals with improved sparse representation in unknown spatially correlated Gaussian noise. To be specific, leveraging a symmetric uniform linear array, the entries of the signal covariance matrix is firstly recasted to eliminate the spatially correlated noise. Subsequently, it is shown that an equivalent source vector can be obtained by squaring any row of the noise-free signal covariance matrix, irrespective of the coherency between the signals. Finally, an improved sparse representation, which enhances signal sparsity via utilizing a designed weight vector, is derived to determine the DOAs of the signals. Numerical examples are provided to demonstrate its superiority of DOA estimation performance in low signal-to-noise ratio (SNR) environments. Besides, it is computationally efficient, which is critical for large array and/or real-time data processing systems.

Palomar Gattini-IR: Survey Overview, Data Processing System, On-sky Performance and First Results

Palomar Gattini-IR is a new wide-field, near-infrared (NIR) robotic time domain survey operating at Palomar Observatory. Using a 30 cm telescope mounted with a H2RG detector, Gattini-IR achieves a field of view (FOV) of 25 sq. deg. with a pixel scale of 8.″7 in J-band. Here, we describe the system design, survey operations, data processing system and on-sky performance of Palomar Gattini-IR. As a part of the nominal survey, Gattini-IR scans ≈7500 square degrees of the sky every night to a median 5σ depth of 15.7 AB mag outside the Galactic plane. The survey covers ≈15,000 square degrees of the sky visible from Palomar with a median cadence of 2 days. A real-time data processing system produces stacked science images from dithered raw images taken on sky, together with point-spread function (PSF)-fit source catalogs and transient candidates identified from subtractions within a median delay of ≈4 hr from the time of observation. The calibrated data products achieve an astrometric accuracy (rms) of ≈0.″7 with respect to Gaia DR2 for sources with signal-to-noise ratio > 10, and better than ≈0.″35 for sources brighter than ≈12 Vega mag. The photometric accuracy (rms) achieved in the PSF-fit source catalogs is better than ≈3% for sources brighter than ≈12 Vega mag and fainter than the saturation magnitude of ≈8.5 Vega mag, as calibrated against the Two Micron All Sky Survey catalog. The detection efficiency of transient candidates injected into the images is better than 90% for sources brighter than the 5σ limiting magnitude. The photometric recovery precision of injected sources is 3% for sources brighter than 13 mag, and the astrometric recovery rms is ≈0.″9. Reference images generated by stacking several field visits achieve depths of ≳16.5 AB mag over 60% of the sky, while it is limited by confusion in the Galactic plane. With a FOV ≈40× larger than any other existing NIR imaging instrument, Gattini-IR is probing the reddest and dustiest transients in the local universe such as dust obscured supernovae in nearby galaxies, novae behind large columns of extinction within the galaxy, reddened microlensing events in the Galactic plane and variability from cool and dust obscured stars. We present results from transients and variables identified since the start of the commissioning period.