Data Processing Platform Research Articles

A ground-based remote sensing system for high-frequency and real-time monitoring of phytoplankton blooms

The worldwide expansion of phytoplankton blooms has severely threatened water quality, food webs, habitat stability and human health. Due to the rapidity of phytoplankton migration and reproduction, high-frequency information on phytoplankton bloom dynamics is crucial for their forecasting, treatment, and management. While several approaches involving satellites, in situ observations and automated underwater monitoring stations have been widely used in the past several decades, they cannot fully provide high-frequency and continuous observations of phytoplankton blooms at low cost and with high accuracy. Thus, we propose a novel ground-based remote sensing system (GRSS) that can monitor real-time chlorophyll a concentrations (Chla) in inland waters with a high frequency. The GRSS mainly consists of three platforms: the spectral measurement platform, the data-processing platform, and the remote access control, display and storage platform. The GRSS is capable of obtaining a remote sensing irradiance ratio (R(λ)) of 400–1000 nm at a high frequency of 20 s. Eight different Chla retrieval algorithms were calibrated and validated using a dataset of 481 pairs of GRSS R(λ) and in situ Chla measurements collected from four inland waters. The results showed that random forest regression achieved the best performance in deriving Chla (R2 = 0.95, root mean square error = 13.40 μg/L, and mean relative error = 25.7%). The GRSS successfully captured two typical phytoplankton bloom events in August 2021 with rapid changes in Chla from 20 μg/L to 325 μg/L at the minute level, highlighting the critical role that this GRSS can play in the high-frequency monitoring of phytoplankton blooms. Although the algorithm embedded into the GRSS may be limited by the size of the training dataset, the high-frequency, continuous and real-time data acquisition capabilities of the GRSS can effectively compensate for the limitations of traditional observations. The initial application demonstrated that the GRSS can capture rapid changes of phytoplankton blooms in a short time and thus will play a critical role in phytoplankton bloom management. From a broader perspective, this approach can be extended to other carriers, such as aircraft, ships and unmanned aerial vehicles, to achieve the networked monitoring of phytoplankton blooms.

A novel short-term load forecasting method based on mini-batch stochastic gradient descent regression model

Short-term Load Forecasting (STLF) is the basis of smart distribution network system operation, planning, and dispatching. The traditional linear regression prediction method has the problems of slow prediction speed and low prediction accuracy. In order to solve the problem, an improved regression model based on mini-batch stochastic gradient descent is proposed in this paper. Combined with the big data analysis and processing platform, the collected data is conformed, and the parallel computing model Map-Reduce is used to parallelize mini-batch stochastic gradient descent algorithm for improving the processing ability of mini-batch stochastic gradient descent algorithm in big data load forecasting, and shorten load forecasting time. Meanwhile, in order to clean up the duplicated data and bad data generated by the smart meter and sensor before calculation, an adaptive sorted neighborhood method is proposed to detect the repeatedly recorded data, and the K-means clustering method is used to eliminate the noise data .The experimental results show that the parallelized mini-batch stochastic gradient descent algorithm is much faster than the traditional regression analysis algorithm when the data volume is large. The average absolute percentage error of the load forecasting model for Belgium and a transformer station in Baiyin city of Gansu Province in China is 1.902% and 2.058% respectively, which satisfies the requirements of load forecasting.

Construction and application of LHAASO data processing platform

Construction and application of LHAASO data processing platform

SIMILARITY ALGORITHMS FOR FUZZY JOIN COMPUTATION IN BIG DATA PROCESSING ENVIRONMENT

Big data processing is attracting the interest of many researchers to process large-scale datasets and extract useful information for supporting and providing decisions. One of the biggest challenges is the problem of querying large datasets. It becomes even more complicated with similarity queries instead of exact match queries. A fuzzy join operation is a typical operation frequently used in similarity queries and big data analysis. Currently, there is very little research on this issue, thus it poses significant barriers to the efforts of improving query operations on big data efficiently. As a result, this study overviews the similarity algorithms for fuzzy joins, in which the data at the join key attributes may have slight differences within a fuzzy threshold. We analyze six similarity algorithms including Hamming, Levenshtein, LCS, Jaccard, Jaro, and Jaro - Winkler, to show the difference between these algorithms through the three criteria: output enrichment, false positives/negatives, and the processing time of the algorithms. Experiments of fuzzy joins algorithms are implemented in the Spark environment, a popular big data processing platform. The algorithms are divided into two groups for evaluation: group 1 (Hamming, Levenshtein, and LCS) and group 2 (Jaccard, Jaro, and Jaro - Winkler). For the former, Levenshtein has an advantage over the other two algorithms in terms of output enrichment, high accuracy in the result set (false positives/negatives), and acceptable processing time. In the letter, Jaccard is considered the worst algorithm considering all three criteria mean while Jaro - Winkler algorithm has more output richness and higher accuracy in the result set. The overview of the similarity algorithms in this study will help users to choose the most suitable algorithm for their problems.

Probabilistic Statistics-Based Endurance Life Prediction of Bridge Structures.

With the massive construction of bridge infrastructures, bridge health monitoring systems have gradually matured in application and research, but previous research has primarily focused on structural damage detection and bridge safety warnings based on valid data. The structural details of steel bridge panels and structural systems are determined by the coupling effects of many intrinsic and extrinsic uncertainties, such as material properties, structural characteristics, manufacturing processes, and random traffic loads. The evaluation of fatigue is a difficult task. This article first builds a big data platform, utilizing its high-efficiency parallel computing capability and highly fault-tolerant distributed file system to achieve second-level monitoring data processing; ensuring real-time data cleaning, data analysis, and safety warning; and building a big data analysis and processing platform with high reliability, high availability, high storage efficiency, and high scalability of bridge health monitoring. The big data platform chooses HDFS for offline data storage and Spark for data analysis and modelling after comparing and analysing the benefits and drawbacks of various big data technologies. Kafka is used for caching real-time data, and Spark-streaming is used for reading data and real-time processing. Finally, the platform's superiority and reliability in terms of offline computing performance, real-time online performance, scalability, and fault tolerance are confirmed through experimental analysis; the optimal data cleaning method is derived by comparing and analysing monitoring data noise, jump point, and drift phenomena. This part of the research is based on bridge temperature data with stable signals and bridge strain data with fluctuating signals, taking into account the influence of different data types; the corresponding data missing repair algorithms are proposed for different types of data to form a complete and general data patching method process. The probabilistic fracture mechanics theory, in comparison to the traditional deterministic fatigue assessment method, can better reflect the essential uncertainty of fatigue problems and is an effective way to assess the fatigue performance of orthotropic steel bridge decks. The goal of data patching is to ensure data recovery accuracy of over 90%, with no patching repair required for monitoring data with too much missing data. The endurance life of bridge structures is predicted using a big data probabilistic statistics approach based on a variety of factors such as material properties, construction characteristics, manufacturing processes, and random traffic loads.

Hadoop Small Image Processing Technology Based on Big Data Processing and Its Application Effect in Face Feature Extraction and Face Recognition System Design

From standard software to crucial applications, face recognition (FR) is always at the core of many unique advances over the past 20 years. Big data is a rapidly growing collection of data. It is important in many applications such as medical, academic, and industries, and it refers to vast data sources that are hard to analyze, store, and interpret for subsequent procedures. Hadoop is an open-source large data processing platform that stores and analyses data in scalable computer clusters. FR technology is built on Hadoop processing to boost the effectiveness of recognition using MapReduce computing. This paper presents a novel adaptive fine-tuned AdaBoost (AFTA) algorithm to enhance the FR in the Hadoop processing. Collected face data sets are employed in preprocessing stage to normalize and enhance the superiority through the median filter (MF) and contour-based image enhancement (CIE), correspondingly. To handle vast quantities of the data, boosted k-means clustering (BKC) approach is used over the Hadoop servers for the MapReduce process. The binary partition tree (BPT) approach is employed in the segmentation stage to split the data into many subgroups. To lessen the dimension of the data, we use the Gabor filter. To select the consistent features, threshold-based binary particle swarm optimization (T-BPSO) is applied. Then, our proposed technique is utilized in the FR process. Finally, the performance metrics of the proposed technique are examined and compared with existing techniques to accomplish our research with the greatest effectiveness. For accuracy, precision, recall, and F-score, we attained the proposed (AFTA) values of 99.67, 97.13, 95.21, and 94.11, respectively. The outcomes are depicted in graphical representation by employing the MATLAB tool.

Intra-set correlation analysis of medical records of thyroid cancer treated by traditional Chinese medicine Master ZHOU Zhongying

ObjectiveBased on intra-set correlation analysis, this paper deconstructs the clinical medical records of traditional Chinese medicine (TCM) Master ZHOU Zhongying in treating thyroid cancer, and analyzes the experience in “mechanism-syndrome-medicine-prescription” for thyroid cancer. MethodsThrough Medcase data processing platform, based on Frequent Pattern (FP)-Growth enhanced correlation analysis algorithm, the medical records of Professor ZHOU Zhongying for the treatment of thyroid cancer from June 1, 2001 to February 28, 2015 were analyzed within the set. ResultsThis study involved 43 medical records, 43 patients, and 167 visits. After processing intra-set correlations, 28 groups of highly correlated symptoms, 21 groups of highly correlated tongue images, 10 groups of highly correlated pulse conditions, 28 groups of highly correlated pathogenesis, 34 groups of highly correlated herbs, and 26 groups of highly correlated western medicine diagnosis were selected. Professor ZHOU Zhongying treats thyroid cancer according to syndrome differentiation. Symptoms with more association rules included neck swelling, neck pain, cough, and dry mouth; tongue images with more association rules included dark purple tongue, dark red tongue, and fissured tongue; pulse conditions with more association rules were wiry pulse, thready pulse, small pulse, and slippery pulse; the pathogenesis with more association rules was phlegm and blood stasis, damp-heat accumulation, and impairment of both Qi and Yin; herbs with more association rules were Chaihu (Bupleuri Radix), Zeqi (Sun Euphoribiae Herb), and Tiandong (Asparagi Radix); western medicine diagnosis with more association rules included thyroid cancer, insomnia, and chronic gastritis. ConclusionThyroid cancer mostly presents as deficiency in origin and excess in manifestations. The basic pathogenesis is phlegm and blood stasis, damp-heat accumulation, and impairment of both Qi and Yin, which are closely related to liver, kidney, and spleen. Professor ZHOU Zhongying adopts both attack and supplement approaches as the general treatment principle, with a strong emphasis on regulating Qi and relieving depression, eliminating phlegm and resolving stagnation, eliminating dampness and turbidity, clearing fire and destroying poison, moistening dryness and softening hard mass, invigorating Qi and nourishing Yin, and paying attention to nourishing liver and kidney, invigorating spleen and stomach, while protecting the heart and lungs.

A data quarantine model to secure data in edge computing

<p>Edge computing provides an agile data processing platform for latency-sensitive and communication-intensive applications through a decentralized cloud and geographically distributed edge nodes. Gaining centralized control over the edge nodes can be challenging due to security issues and threats. Among several security issues, data integrity attacks can lead to inconsistent data and intrude edge data analytics. Further intensification of the attack makes it challenging to mitigate and identify the root cause. Therefore, this paper proposes a new concept of data quarantine model to mitigate data integrity attacks by quarantining intruders. The efficient security solutions in cloud, ad-hoc networks, and computer systems using quarantine have motivated adopting it in edge computing. The data acquisition edge nodes identify the intruders and quarantine all the suspected devices through dimensionality reduction. During quarantine, the proposed concept builds the reputation scores to determine the falsely identified legitimate devices and sanitize their affected data to regain data integrity. As a preliminary investigation, this work identifies an appropriate machine learning method, linear discriminant analysis (LDA), for dimensionality reduction. The LDA results in 72.83% quarantine accuracy and 0.9 seconds training time, which is efficient than other state-of-the-art methods. In future, this would be implemented and validated with ground truth data.</p>

Design and Implementation of 3D Animation Data Processing Development Platform Based on Artificial Intelligence.

Based on the whole process of computer-aided technology, a 3D animation data processing development platform based on artificial intelligence is designed and implemented. A random forest model for animation data processing and development is designed to mine the experience that can guide animation generation from the accumulated animation data. Based on the design goal and implementation principle of animation data processing and development platform, the attributes and categories of random forest model are abstracted. After standardizing a large number of historical data, the training sample set is obtained, and the random forest model is obtained after training. The parameters of the random forest model are continuously optimized by experiments, so that the learning model can better guide the dynamic animation data processing and development platform to generate animation to the satisfaction of users. The designed three-dimensional animation data processing and development platform interacts with the animation generation module, users, and system administrators. It can continuously receive the sample data of the animation generation module, automatically expand the number of training samples, analyze the status of the sample database, and put forward suggestions to the system administrator to update the learning model, so as to realize the initiative of learning. The experimental results show that the designed 3D animation data processing and development platform is effective and feasible.

NebulaStream: Data Management for the Internet of Things

The Internet of Things (IoT) presents a novel computing architecture for data management: a distributed, highly dynamic, and heterogeneous environment of massive scale. Applications for the IoT introduce new challenges for integrating the concepts of fog and cloud computing as well as sensor networks in one unified environment. In this paper, we present early approaches that address parts of the overall problem space. All approaches are incorporated into NebulaStream (NES), our novel data processing platform that addresses the heterogeneity, unreliability, and scalability challenges of the IoT and thus provides efficient data management for future applications.

ENABLING CITY DIGITAL TWINS THROUGH URBAN LIVING LABS

Abstract. The population density in urban areas is rapidly rising, leading to a constant need for new infrastructure and services for citizens. To reduce the time to implementation and optimise the monetary cost of various solutions, the plans and policies of local authorities and stakeholders would benefit from undergoing a series of virtual stress tests. To this end, prescriptive and predictive technologies are widely adopted to optimise city planning and to understand the urban processes and environment such as air pollution and transportation. Nevertheless, holistic sandboxes tightly integrated with cities are still largely lacking. The city digital twin is a promising concept that provides a tool for exploration of new solutions in a controlled environment before their deployment. The digital twin is a virtual replica of the real city, which collects data from the infrastructure, processes and services using not only the available systems, but also purposely built connected devices and sensors. In this context, the establishment of urban living labs facilitates the monitoring and understanding of urban processes and enriches the digital twin with highly-relevant data. This paper presents an urban living lab, under deployment in the district of Lozenets in Sofia, Bulgaria. It is part of a larger initiative for developing a city digital twin of Sofia to support the design, exploration, and experimentation of different solutions. The living lab is equipped with sensors for monitoring air quality, atmospheric parameters, noise pollution and pedestrian flows. In addition, a Light Detection and Ranging (LiDAR) system is realised as an edge computing facility at one of the busiest intersections of the district. Along with the equipment, the paper describes the architecture and components of the platform for data collection, storage, processing, and visualization. Finally, high-priority studies are presented, and their demographic and economic impact is discussed.

Albatross analytics a hands-on into practice: statistical and data science application

Albatross Analytics is a statistical and data science data processing platform that researchers can use in disciplines of various fields. Albatross Analytics makes it easy to implement fundamental analysis for various regressions with random model effects, including Hierarchical Generalized Linear Models (HGLMs), Double Hierarchical Generalized Linear Models (DHGLMs), Multivariate Double Hierarchical Generalized Linear Models (MDHGLMs), Survival Analysis, Frailty Models, Support Vector Machines (SVMs), and Hierarchical Likelihood Structural Equation Models (HSEMs). We provide 94 types of dataset examples.

Analysis of Ti- and Pb-based particles in the aqueous environment of Melbourne (Australia) via single particle ICP-MS

The analysis of natural and anthropogenic nanomaterials (NMs) in the environment is challenging and requires methods capable to identify and characterise structures on the nanoscale regarding particle number concentrations (PNCs), elemental composition, size, and mass distributions. In this study, we employed single particle inductively coupled plasma-mass spectrometry (SP ICP-MS) to investigate the occurrence of NMs in the Melbourne area (Australia) across 63 locations. Poisson statistics were used to discriminate between signals from nanoparticulate matter and ionic background. TiO2-based NMs were frequently detected and corresponding NM signals were calibated with an automated data processing platform. Additionally, a method utilising a larger mass bandpass was developed to screen for particulate high-mass elements. This procedure identified Pb-based NMs in various samples. The effects of different environmental matrices consisting of fresh, brackish, or seawater were mitigated with an aerosol dilution method reducing the introduction of salt into the plasma and avoiding signal drift. Signals from TiO2- and Pb-based NMs were counted, integrated, and subsequently calibrated to determine PNCs as well as mass and size distributions. PNCs, mean sizes, particulate masses, and ionic background levels were compared across different locations and environments.Graphical abstract

Human-Computer Interaction of Networked Vehicles Based on Big Data and Hybrid Intelligent Algorithm

Intelligent connected vehicle (ICV) refers to realizing the exchange and sharing of intelligent information between vehicles, roads, people, clouds, and so on by carrying advanced on-board sensors, controllers, actuators, and other devices and combining modern communication and network technology. In the age of big data, the information of everything can be transformed into digital resources, and transportation big data has become a basic resource. This paper constructs a big data platform for traffic data processing to realize the function of real-time collection, processing, and analysis of traffic data. Based on the proposed big data platform, the parallel programming framework of MapReduce and HDFS distributed storage system are used to process the real-time vehicle dynamic information in parallel, and the output result is used as the input of running genetic algorithm simulated annealing (GA-SA) for parallel calculation. At the same time, it studies the impact of various elements on users’ interactive behavior, constructs the demand framework and design model of automobile human-computer interaction, and then realizes fast and comprehensive search. The experimental results show that the human-computer interaction method of intelligent networked vehicle can find the optimal driving path, transmit it to each networked vehicle through the human-computer interaction system, realize human-computer interaction, reduce the impact of user unintentional operation on redundant motion, reduce the motion error accumulation of the system, and improve the performance of human-computer interaction system.

Deriving moderate spatial resolution leaf area index estimates from coarser spatial resolution satellite products

Leaf Area Index (LAI) is a key parameter used to characterize vegetation biophysical properties and plays an important regulatory role in terrestrial-atmospheric exchanges. Subsequently, LAI is often a critical data input to various evapotranspiration, hydrological and climatic models. While LAI data can generally be easily obtained, it is seldom available at spatio-temporal scales that can be used to guide and inform management decisions for localised applications. To this end, we propose a methodology to acquire moderate resolution LAI (LAIMR) estimates from freely available satellite-earth observation data sets and data processing platforms. Fifteen sites distributed within the KwaZulu-Natal and Mpumalanga provinces of South Africa were selected as study areas. Coarse spatial resolution MODIS LAI and vegetation index (VI) products were acquired for each of these sites to establish the LAI-VI relationship which was then used to develop machine learning-based models (MLBMs) to estimate LAIMR using VIs derived from Landsat or Sentinel-2 data. During the validation and testing phases of the study, LAI estimates were compared against the corresponding MODIS LAI product values. The results of these investigations demonstrated that MLBMs performed satisfactorily across the majority of the study sites, producing correlation coefficients ranging from 0.62 to 0.97 and 0.29–0.84 for the validation and testing phases, respectively. The poorer performance of the MLBMs when using VIs derived from Landsat or Sentinel-2 data can be largely attributed to inherent limitations associated with the proposed methodology, such as i) the lack of moderate-high spatial resolution LAI records that could be used for training and testing purposes and ii) saturation effects associated with the use of VIs. Notwithstanding these limitations, the proposed methodology has been shown to be flexible and robust and can be a useful approach to acquire LAIMR estimates with fairly reasonable accuracy in data limited circumstances.

Real-Time UAV Trash Monitoring System

This study proposes a marine trash detection system based on unmanned aerial vehicles (UAVs) and aims to replace manpower with UAVs to detect marine trash efficiently and provide information to government agencies regarding real-time trash pollution. Internet technology and computer–machine interaction were applied in this study, which involves the deployment of a marine trash detection system on a drone’s onboard computer for real-time calculations. Images of marine trash were provided to train a modified YOLO model (You Look Only Once networks). The UAV was shown to be able to fly along a predefined path and detect trash in coastal areas. The detection results were sent to a data streaming platform for data processing and analysis. The Kafka message queuing system and the Mongo database were used for data transmission and analysis. It was shown that a real-time drone map monitoring station can be built up at any place where mobile communication is accessible. While a UAV is automatically controlled by an onboard computer, it can also be controlled through a remote station. It was shown that the proposed system can perform data analysis and transmit heatmaps of coastal trash information to a remote site. From the heatmaps, government agencies can use trash categories and locations to take further action.

High speed data transmission method of substation intelligent auxiliary monitoring system

In order to better ensure the effective operation of substation intelligent auxiliary monitoring system, aiming at the problems of slow data transmission speed and poor data processing security of current monitoring system, a high-speed data transmission method of the substation intelligent auxiliary monitoring system is proposed. By constructing the data processing platform of substation intelligent auxiliary monitoring system, the data transmission algorithm of auxiliary monitoring system is optimized, reasonably select the optimal path of data transmission and conduct security encryption processing, so as to improve the high-speed data transmission efficiency of substation intelligent auxiliary monitoring system. Finally, the experiment shows that the high-speed data transmission method of the substation intelligent auxiliary monitoring system has higher practicability in the process of practical application and fully meets the research requirements.

MRMondrian: Scalable Multidimensional Anonymisation for Big Data Privacy Preservation

Scalable data processing platforms built on cloud computing becomes increasingly attractive as infrastructure for supporting big data applications. But privacy concerns are one of the major obstacles to making use of public cloud platforms. Multidimensional anonymisation, a global-recoding generalisation scheme for privacy-preserving data publishing, has been a recent focus due to its capability of balancing data obfuscation and usability. Existing multidimensional anonymisation methods suffer from scalability problems when handling big data due to the impractical serial I/O cost. Given the recursive feature of multidimensional anonymisation, parallelisation is an ideal solution to scalability issues. However, it is still a challenge to use existing distributed and parallel paradigms directly for recursive computation. In this paper, we propose a scalable approach for big data multidimensional anonymisation based on MapReduce, a state-of-the-art data processing paradigm. Our basic idea is to partition a data set recursively into smaller partitions using MapReduce until all partitions can fit in the memory of a computing node. A tree indexing structure is proposed to achieve recursive computation. Moreover, we show the applicability of our approach to differential privacy. Experimental results on real-life data demonstrate that our approach can significantly improve the scalability of multidimensional anonymisation over existing methods.

Sboing4Real: A real-time crowdsensing-based traffic management system

This work describes the architecture of the back-end engine of a real-time traffic data processing and satellite navigation system. The role of the engine is to process real-time feedback, such as speed and travel time, provided by in-vehicle devices and derive real-time reports and traffic predictions through leveraging historical data as well. We present the main building blocks and the versatile set of data sources and processing platforms that need to be combined together to form a fully-functional and scalable solution. We also present performance results focusing on meeting system requirements while keeping the need for computing resources low. The lessons and results presented are of value to additional real-time applications that rely on both recent and historical data. Finally, we discuss the application of the aforementioned solution to a successful pilot study, where the full system was deployed and processed data from 800 taxis for a period of 3 months.

Scalable recommendation systems based on finding similar items and sequences

AbstractThe rapid growth in the airline industry, which started in 2009, continued until the COVID‐19 era, with the annual number of passengers almost doubling in 10 years. This situation has led to increased competition between airline companies, whose profitability has decreased considerably. They aimed to increase their profitability by making services like seat selection, excess baggage, Wi‐Fi access optional under the name of ancillary services. To the best of our knowledge, there is no recommendation system for recommending ancillary services for airline companies. Also, to the best of our knowledge, there is no testing framework to compare recommendation algorithms considering their scalabilities and running times. In this paper, we propose a framework based on Lambda architecture for recommendation systems that run on a big data processing platform. The proposed method utilizes association rule and sequential pattern mining algorithms that are designed for big data processing platforms. To facilitate testing of the proposed method, we implement a prototype application. We conduct an experimental study on the prototype to investigate the performance of the proposed methodology using accuracy, scalability, and latency related performance metrics. The results indicate that the proposed method proves to be useful and has negligible processing overheads.