Multi-source Heterogeneous Data Research Articles

An Approach to Rapidly Evaluating Rock Mass Quality in Underground Engineering Based on Multi-source Heterogeneous Data

An Approach to Rapidly Evaluating Rock Mass Quality in Underground Engineering Based on Multi-source Heterogeneous Data

Research and Prediction of China's Air Environment

Abstract: With the rapid development of China's economy, environmental issues, particularly air pollution, have become increasingly severe, posing significant threats to human health and ecosystems. This study employed statistical learning methods to analyze air pollution composition data from thirty major Chinese cities throughout 2019-2022. The study used algorithms like regression, classification, fitting, and filtering to find changes in multi-source heterogeneous data patterns and predict how air pollutant concentrations would change over time and space. At the same time,the results indicate that the primary factors influencing the Air Quality Index (AQI) are fine particulate matter (PM2.5) and inhalable particulate matter (PM10). Additionally, this research is significant for improving public health, advancing environmental protection, fostering economic growth, strengthening laws and regulations, ensuring social stability, and promoting scientific and cultural advancements. Through this research, scientific bases can be provided for formulating policies that reduce disease, enhance government credibility, drive innovation in atmospheric research, and preserve cultural heritage.

Spatial optimization strategies for China’s hydrogen infrastructure industry chain

Promoting the development of China’s hydrogen energy industry is crucial for achieving green energy transition. However, existing research lacks systematic studies on the spatial layout of the hydrogen industry chain. This study constructed a comprehensive theoretical framework encompassing hardware infrastructure, software systems, and soft power. Using multi-source heterogeneous data, GIS analysis, and NVivo text coding methods, the current regional layout and challenges of China’s hydrogen infrastructure industry chain were systematically evaluated. The findings determined that economically developed eastern regions lead in infrastructure and soft power, while central and western regions leverage their resource and manufacturing advantages. Major challenges include regional imbalances in hardware infrastructure, uneven distribution of soft power, and misalignment between software systems and actual needs. Analysis of the “14th Five-Year Plan” of various regions elucidated deep insights into the diversity of local hydrogen energy development strategies, identifying five types of hydrogen cities: resource-advantaged, market-oriented, regionally collaborative, innovation-driven, and policy-supported. Accordingly, strategies to enhance industry chain synergy, clarify city roles, and optimize regional ecosystems were proposed. It is recommended to integrate hydrogen infrastructure with urban planning and incorporate environmental impact assessments into spatial optimization decisions. This study provides a systematic analytical framework and progressive policy recommendations for the efficient and green layout of China’s hydrogen infrastructure, offering important implications for the sustainable development of the hydrogen industry and other rapidly developing economies.

Biofuser: a multi-source data fusion platform for fusing the data of fermentation process devices.

In the past decade, the progress of traditional bioprocess optimization technique has lagged far behind the rapid development of synthetic biology, which has hindered the industrialization process of synthetic biology achievements. Recently, more and more advanced equipment and sensors have been applied for bioprocess online inspection to improve the understanding and optimization efficiency of the process. This has resulted in large amounts of process data from various sources with different communication protocols and data formats, requiring the development of techniques for integration and fusion of these heterogeneous data. Here we describe a multi-source fusion platform (Biofuser) that is designed to collect and process multi-source heterogeneous data. Biofuser integrates various data to a unique format that facilitates data visualization, further analysis, model construction, and automatic process control. Moreover, Biofuser also provides additional APIs that support machine learning or deep learning using the integrated data. We illustrate the application of Biofuser with a case study on riboflavin fermentation process development, demonstrating its ability in device faulty identification, critical process factor identification, and bioprocess prediction. Biofuser has the potential to significantly enhance the development of fermentation optimization techniques and is expected to become an important infrastructure for artificial intelligent integration into bioprocess optimization, thereby promoting the development of intelligent biomanufacturing.

Feasibility of Emergency Flood Traffic Road Damage Assessment by Integrating Remote Sensing Images and Social Media Information

In the context of global climate change, the frequency of sudden natural disasters is increasing. Assessing traffic road damage post-disaster is crucial for emergency decision-making and disaster management. Traditional ground observation methods for evaluating traffic road damage are limited by the timeliness and coverage of data updates. Relying solely on these methods does not adequately support rapid assessment and emergency management during extreme natural disasters. Social media, a major source of big data, can effectively address these limitations by providing more timely and comprehensive disaster information. Motivated by this, we utilized multi-source heterogeneous data to assess the damage to traffic roads under extreme conditions and established a new framework for evaluating traffic roads in cities prone to flood disasters caused by rainstorms. The approach involves several steps: First, the surface area affected by precipitation is extracted using a threshold method constrained by confidence intervals derived from microwave remote sensing images. Second, disaster information is collected from the Sina Weibo platform, where social media information is screened and cleaned. A quantification table for road traffic loss assessment was defined, and a social media disaster information classification model combining text convolutional neural networks and attention mechanisms (TextCNN-Attention disaster information classification) was proposed. Finally, traffic road information on social media is matched with basic geographic data, the classification of traffic road disaster risk levels is visualized, and the assessment of traffic road disaster levels is completed based on multi-source heterogeneous data. Using the “7.20” rainstorm event in Henan Province as an example, this research categorizes the disaster’s impact on traffic roads into five levels—particularly severe, severe, moderate, mild, and minimal—as derived from remote sensing image monitoring and social media information analysis. The evaluation framework for flood disaster traffic roads based on multi-source heterogeneous data provides important data support and methodological support for enhancing disaster management capabilities and systems.

Open Geospatial Engine: A Cloud-based Spatiotemporal Computing Platform

Abstract. The quantity and diversity of Earth spatiotemporal big data have significantly increased in recent years, providing the potential to comprehensively analyse complex spatiotemporal problems from new perspectives. However, integrating, managing, and applying multi-source heterogeneous Earth spatiotemporal big data remains a challenge. To address this issue, this study proposes a cloud-based spatiotemporal computing platform, the Open Geospatial Engine (OGE), for the unified organization and joint analysis of Earth spatiotemporal big data in multiple dimensions and scales. The framework of this platform comprises three modules: data management, computing engine, and service interface. The data management module adopts the GeoCube model to effectively integrate and manage multi-source heterogeneous spatiotemporal data through multi-dimensional aligned tiles. The computing engine module seamlessly maps the GeoCube model to the cloud environment by extending Spark RDD, making the GeoCube model capable of high-performance distributed computing. Following OGC specifications, the service interface module integrates and shares data, operators, and spatiotemporal analysis models through extensible APIs in an interactive development environment. Combined with a series of high-performance optimization techniques, OGE simplifies data queries and the construction of complex analytical applications by integrating these three modules. The applicability of OGE is demonstrated by case studies involving multi-dimensional queries and joint analysis of long-time-series and heterogeneous spatiotemporal data.

Research on Semantic Driven Urban Pipeline Dataspace Construction Method

Abstract. Urban pipeline data is heterogeneous in multiple sources and rich in data volume, and there are problems such as data conflict and difficult organization and management due to the heterogeneity of multiple sources when accessing the data in large-scale concurrently. To address this problem, this paper proposes a semantics-driven urban pipeline dataspace construction method, which aims to realize the efficient organization of pipeline data. Firstly, this method combines the classification and characteristics of urban pipelines, and expresses the semantic information of pipeline geographic entities from four dimensions: semantic description, spatial location, attribute characteristics and time evolution. Then, the four expression sets are embedded into the dataspace RDF model as predicates, and the associated description mechanism of pipeline geographic entities is established by means of genus classes and so on, so as to construct the pipeline dataspace RDF model. Finally, the model is stored and graphically visualized using neo4j to achieve fast retrieval of data within the pipeline dataspace. The research results show that this method provides a unified expression of pipeline entities, solves the problem of pipeline multi-source heterogeneous data conflict and organization difficulties, and improves the efficiency of multi-source heterogeneous pipeline data organization while ensuring the integrity of pipeline information to the maximum extent.

Communication Technology for Renewable Energy Access Grid System Based on Multi-source Heterogeneous Data Fusion

Abstract Incorporating renewable energy solutions into the power grid is essential for facilitating a shift in energy sources. Ensuring the effectiveness and dependability of communication technologies within smart distribution grids is vital, as these grids serve as the link between end users and the central station, thereby contributing to the grid’s stable functioning and the effective dissemination of energy. This study introduces a framework for communication that tackles the challenges associated with the integration of multi-source, heterogeneous data in smart distribution grids. The framework leverages multi-source heterogeneous data fusion and capitalizes on edge computing to efficiently process and consolidate data from diverse sensors and devices. An optimal weight allocation algorithm is proposed to mitigate data redundancy and to improve the energy efficiency of communication. The paper demonstrates, through theoretical examination and empirical tests, the proposed system’s enhanced communication capabilities, especially in terms of data transmission effectiveness and energy usage optimization. The discussion highlights the advantages of smart distribution networks over conventional communication methods and offers insights for further advancements in communication technologies within this sector.

An unsupervised learning framework for health diagnosis by incorporating multiscale data

Integrating real-time evaluations of health states with Structural Health Monitoring (SHM) data and environmental observations is crucial, particularly under large datasets. Not only is the volume of real-time data analysis substantial, but traditional algorithms like wavelet analysis often require extensive prior work and significant manual input, limiting the application of SHM and big data technologies in structural analysis. To overcome this limitation, this study introduces an unsupervised analytical framework coupling a Denoising Sparse Wavelet Network (DeSpaWN) and a dynamic-inner Principal Component Analysis (DiPCA) algorithm. The DeSpaWN is designed to extract sparse representation from multiscale sensor signals, and the features are subsequently analyzed using the DiPCA to assess health diagnoses. The DeSpaWN is specifically tailored for monitoring high-frequency time series dynamically and adaptively, enabling reconstruction and extraction of signals across various frequency domains. Additionally, the DiPCA effectively handles large-scale time-series data, providing precise evaluations by accounting for dynamic latent correlations. To validate the method, a 5-Degree of Freedom (DoF) structure model is established, equipped with five virtual sensors at each DoF and an anemometer at the top DoF. The sensor signals, obtained under different health states and subjected to artificial excitations, are input into the proposed framework. Validation results demonstrate the effectiveness in synthesizing multi-sensor data and assessing structural integrity. The framework's practical utility is showcased through a real-world case study that assesses the structural integrity of a wind turbine. By integrating real and virtual sensor data collected over a year and cross-referencing this with the fault log, the study validates the algorithm's efficacy in processing multisource heterogeneous data and its precision in structural health diagnosis. The study substantiates real-time tracking of health states for SHM-equipped structures, underscoring its potential to enhance safety and reliability in civil infrastructure.

Code and Data Repository for Fraud Detection by Integrating Multisource Heterogeneous Presence-Only Data

This repository provides data for the problem and code for the method. The main folders are data and code.

Fraud Detection by Integrating Multisource Heterogeneous Presence-Only Data

Fraud Detection by Integrating Multisource Heterogeneous Presence-Only Data

Analysis of Short-Term Heavy Rainfall-Based Urban Flood Disaster Risk Assessment Using Integrated Learning Approach

Accurate and timely risk assessment of short-term rainstorm-type flood disasters is very important for ecological environment protection and sustainable socio-economic development. Given the complexity and variability of different geographical environments and climate conditions, a single machine learning model may lead to overfitting issues in flood disaster assessment, limiting the generalization ability of such models. In order to overcome this challenge, this study proposed a short-term rainstorm flood disaster risk assessment framework under the integrated learning model, which is divided into two stages: The first stage uses microwave remote sensing images to extract flood coverage and establish disaster samples, and integrates multi-source heterogeneous data to build a flood disaster risk assessment index system. The second stage, under the constraints of Whale Optimization Algorithm (WOA), optimizes the integration of random forest (RF), support vector machine (SVM), and logistic regression (LR) base models, and then the WRSL-Short-Term Flood Risk Assessment Model is established. The experimental results show that the Area Under Curve (AUC) accuracy of the WRSL-Short-Term Flood Risk Assessment Model is 89.27%, which is 0.95%, 1.77%, 2.07%, 1.86%, and 0.47% higher than RF, SVM, LR, XGBoost, and average weight RF-SVM-LR, respectively. The accuracy evaluation metrics for accuracy, Recall, and F1 Score have improved by 5.84%, 21.50%, and 11.06%, respectively. In this paper, WRSL-Short-Term Flood Risk Assessment Model is used to carry out the risk assessment of flood and waterlogging disasters in Henan Province, and ArcGIS is used to complete the short-term rainstorm city flood and waterlogging risk map. The research results will provide a scientific assessment basis for short-term rainstorm city flood disaster risk assessment and provide technical support for regional flood control and risk management.

Multi-granularity spatiotemporal object modelling of waterborne traffic elements

The electronic navigational charts are crucial carriers for representing the multi-source heterogeneous data of Waterborne Traffic Elements (WTEs). However, their layer-based modelling method has some shortcomings in expressing the multi-granularity features, complex relationships, and dynamic evolution of elements. This paper proposes an objectification modelling method for WTEs based on the concept of multi-granularity spatiotemporal object modelling. A classification system for waterborne traffic objects is developed based on the relevance of behavior to elements; combining characteristics of waterborne traffic, a data model for waterborne traffic objects is constructed from eight aspects: spatiotemporal reference, spatiotemporal position, spatial form, basic information, attributes, behavioral ability, structure, and associative relationships. An object extraction function is also established, extracting object attributes and relationships between objects according to different element classes. Taking the Jiashan section of the Hangzhou-Shanghai Line in Zhejiang Province as the experimental subject, the multi-granularity spatiotemporal characteristics, dynamic evolution, and relationship expression of channel class objects are tested. The experimental results show that the proposed method provides the theoretical basis and data organization mode for the multi-granularity expression of WTEs.

Research on the traffic flow measurement method of intelligent connected vehicles based on series and cascade technology

At present, the major cities in our country have entered the development of smart cities, the utilization rate and prevalence rate of intelligent connected cars in smart cities have reached a new peak,to manage the real-time traffic flow of intelligent connected vehicles effectively in smart cities. This paper proposes to calculate the flow of intelligent connected vehicles based on series cascade technology. Firstly, this paper collects multi-source heterogeneous data based on multiple collectors; Secondly, a generative adversarial network is employed for data fusion, synthetic data, closely mirroring the original, is generated for handling unknown variables during model training. Next, an autoencoder model generates input data for feature representation. Lastly, CatBoost, a regression model that leverages a cascaded machine learning algorithm, is utilized to forecast the traffic flow of intelligent connected vehicles in smart cities. The results show that the method based on CASCaded machine learning has remarkable effect in predicting smart city traffic flow and alleviating urban road congestion.

Multi-modal fusion deep learning model for excavated soil heterogeneous data with efficient classification

Multimodal fusion, a cutting-edge method aiming to integrate heterogeneous data for prediction tasks, has received limited attention in geotechnical engineering. This study focuses on classifying excavated soils with large quantities rapidly and finely. By using an excavated soil information collecting system at the largest soil transferring platform in China, 3,243 groups of multi-source heterogenous data were created, including soil images, spatial series data of cone index curves, time series data of TDR waveforms, discrete net weight data, and textual descriptions of soil morphology. After data augmentation, a big dataset containing 23,122 sets with labels based on the Unified Soil Classification System, moisture content, and mineral composition was created. Multimodal deep learning models were trained using different fusion strategies: 7 early fusion, 3 intermediate fusion, and 2 late fusion cases. Performance metrics were evaluated, including loss, accuracy, F1 Score, precision, recall, specificity, NPV, FPR, and AUROC. Results showed that intermediate fusions performed best, while late fusions performed the poorest. The two-stage intermediate fusion with five modalities achieved the best results, achieving an accuracy above 0.99 on the test set. This multimodal fusion approach effectively explores the correlation between multi-source heterogeneous data of soils, leading to scientific and engineering value in geotechnics.

An outlier identification method based on parallel clustering and the Adaboost algorithm for power multi-source heterogeneous data

Abstract Influenced by the diversity of data scheduling and the multi-level differentiation of data distribution, the multi-source heterogeneous data of electric power have serious deviations in space and time scheduling, which makes it difficult to identify outliers. Therefore, a method for identifying outliers of multi-source heterogeneous data of electric power based on parallel clustering and the Adaboost algorithm is proposed. A sampling node model of heterogeneous data is constructed, and random numbers that obey Gaussian distribution are generated according to the mean value of data, and their mean square loss values are optimized to form an objective function of abnormal values of multi-source heterogeneous data of electric power. According to parallel clustering and the Adaboost algorithm, a unified format is created to obtain the load control results of abnormal values of multi-source heterogeneous data of electric power, so as to realize the identification of abnormal values of multi-source heterogeneous data of electric power. The experimental results show that at the time point of 18 s, the data identification quality of this method has reached 80, and with the increase of time, its data identification quality remains stable. It is demonstrated that the method has good recognition results.

Exploring the spatial relationship between soil quality index and soil ecosystem services driven by social-ecological factors: A peri-urban case study in central China

Urban expansion has profound impacts on soil quality in peri-urban areas. Understanding the intricate spatial relationship between soil quality index (SQI) and soil ecosystem services (SESs) in response to social-ecological factors is crucial for achieving sustainable soil utilization and management. In this study, a total of 485 soil samples were collected from different land-use types (including paddy field, dry land, orchard, and forest) in peri-urban areas of a city (Nanchang) in central China. The assessments of soil quality based on SQI and SESs were conducted using multi-source heterogeneous data. We determined the spatial relationships between SQI and six SESs using bivariate spatial autocorrelation analysis, and deciphered the role of 11 social-ecological factors in driving SQI–SES relationships across different zones. The results showed that SQI and SESs exhibited significant spatial clustering (P<0.01), despite certain variations in their distribution patterns. SQI had significant positive spatial correlations with soil organism habitat, food provision, erosion control, carbon storage, and filtering and buffering (r = 0.16–0.48, P<0.05). Two-step clustering categorized 183 villages with distinct SQI–SES relationships into four zones, i.e., soil ecological protection zone, food production core zone, soil quality enhancement zone, and soil ecological restoration zone. The impacts of various social-ecological factors on different zones were unveiled and corresponding development recommendations were presented. Results of this study facilitate the formulation of peri-urban-specific soil conservation policies and management strategies. Our work provides guidance on the improvement of soil quality and soil ecosystems, contributing to the future achievement of sustainable development goals.

Medical information management system based on multi-source heterogeneous big data

ABSTRACT This study aims to provide a medical information management system using multi-source heterogeneous big data to improve medical service quality and efficiency, with a motivation on its potential in medical insurance DRG payment. The system framework uses Back Propagation Neural Network (BPNN) technology to efficiently process and analyze multi-source medical data. Comparative experiments and parameter adjustments evaluated the system’s performance. Results show that the BPNN model achieved excellent accuracy (92.5%), recall (93%), and F1 value (92.8%) on the test data set, outperforming other models such as PSO(88%), CNN(89%), and RNN(90%). The system’s response speed was also significantly improved, with an average response time of 0.38 seconds, compared to 0.89 seconds for traditional systems. A 72-hour stability test confirmed the system’s reliability and ability to meet user needs. The proposed system demonstrates improved performance and user experience, making it a promising solution for medical information management and DRG payment applications.

Integrating Sensor Embeddings with Variant Transformer Graph Networks for Enhanced Anomaly Detection in Multi-Source Data

With the rapid development of sensor technology, the anomaly detection of multi-source time series data becomes more and more important. Traditional anomaly detection methods deal with the temporal and spatial information in the data independently, and fail to make full use of the potential of spatio-temporal information. To address this issue, this paper proposes a novel integration method that combines sensor embeddings and temporal representation networks, effectively exploiting spatio-temporal dynamics. In addition, the graph neural network is introduced to skillfully simulate the complexity of multi-source heterogeneous data. By applying a dual loss function—consisting of a reconstruction loss and a prediction loss—we further improve the accuracy of anomaly detection. This strategy not only promotes the ability to learn normal behavior patterns from historical data, but also significantly improves the predictive ability of the model, making anomaly detection more accurate. Experimental results on four multi-source sensor datasets show that our proposed method performs better than the existing models. In addition, our approach enhances the ability to interpret anomaly detection by analyzing the sensors associated with the detected anomalies.

Integrating geographic knowledge into deep learning for spatiotemporal local climate zone mapping derived thermal environment exploration across Chinese climate zones

The Local Climate Zone (LCZ) scheme representing urban structure and land use pattern is essential for urban heat island (UHI) research. Fine-grained LCZ mapping considering spatial and temporal heterogeneity can provide a more precise characterization of surface thermal properties, thereby enabling a comprehensive analysis and understanding of spatiotemporal trends in climate change research. However, data-driven deep learning-based methods have limitations in coping with the complex urban landscapes of the real-world scenarios, including the spectral similarity of different LCZ and the geospatial heterogeneity of urban LCZ categories. In this study, we constructed a geographic knowledge base for enhanced LCZ characterization with the consideration of prior surface spatial information, including a set of spectral indices and urban morphological parameters (UMPs). Then, we integrated the explicable geographic knowledge base into a learnable deep learning framework in an end-to-end manner for accurate LCZ mapping by fusing multi-source heterogeneous data with a multi-level fusion strategy. The constructed Chinese Climate Zone Time Series LCZ (CClimate-TLCZ) dataset derived from Landsat-8 data with a 30 m spatial resolution, covering 18 representative cities in China, were used to evaluate the proposed framework. The experimental results demonstrate that the proposed framework achieved optimal outcomes across 18 cities, with an average overall accuracy exceeding 94 %, which is more than 20 % higher than that obtained by the standard WUDAPT method. Furthermore, the analysis of LCZ mapping-driven land surface temperature (LST) and surface UHI (SUHI) applications shows that cities within the same climate zone have similar LST distribution patterns, while significant heterogeneity exist between different zones. The annual consistency of LST patterns within each climate zone supports the validity of the LCZ classification scheme and accurate LCZ mapping for urban thermal environment studies. From 2016 to 2022, SUHI intensity initially increases and then decreases, indicating improvements in the urban thermal environment. These findings underscore the critical role of precise LCZ mapping in urban climate resilience and sustainable urban planning.