Multi-source Heterogeneous Data Research Articles

A multi-source heterogeneous medical data enhancement framework based on lakehouse.

Obtaining high-quality data sets from raw data is a key step before data exploration and analysis. Nowadays, in the medical domain, a large amount of data is in need of quality improvement before being used to analyze the health condition of patients. There have been many researches in data extraction, data cleaning and data imputation, respectively. However, there are seldom frameworks integrating with these three techniques, making the dataset suffer in accuracy, consistency and integrity. In this paper, a multi-source heterogeneous data enhancement framework based on a lakehouse MHDP is proposed, which includes three steps of data extraction, data cleaning and data imputation. In the data extraction step, a data fusion technique is offered to handle multi-modal and multi-source heterogeneous data. In the data cleaning step, we propose HoloCleanX, which provides a convenient interactive procedure. In the data imputation step, multiple imputation (MI) and the SOTA algorithm SAITS, are applied for different situations. We evaluate our framework via three tasks: clustering, classification and strategy prediction. The experimental results prove the effectiveness of our data enhancement framework.

Building Energy Consumption Prediction and Optimization

As the proportion of building energy consumption in global energy consumption is significant and continues to rise, accurate prediction and optimization of building energy consumption are of great significance. This research aims to develop high-precision prediction models and innovative optimization strategies to address the problems of insufficient accuracy of existing prediction methods and poor optimization effects. By collecting and integrating data through multiple channels, a hybrid model architecture combining LSTM, CNN, and the attention mechanism is selected, and an automatic search and optimization method for hyperparameters is adopted. In terms of data processing, multi-source heterogeneous data fusion, data cleaning and enhancement based on spatiotemporal features, and innovative feature engineering are carried out. In the experiments, the dataset is divided considering the dynamic use of buildings and environmental changes, and indicators related to energy costs are introduced for evaluation. The research successfully constructed a model for accurately predicting building energy consumption, proposed effective optimization strategies, reduced energy consumption, and ensured comfort, providing a basis for practical applications through sensitivity analysis and robustness tests.

Parameter Adaptive Contrastive Hashing for multimedia retrieval

With the emergence of massive amounts of multi-source heterogeneous data on the Internet, quickly retrieving effective information from this extensive data has become a hot research topic. Due to the efficiency and speed of hash learning methods in multimedia retrieval, they have become a mainstream method for multimedia retrieval. However, unsupervised multimedia hash learning methods still face challenges with the difficulties of tuning due to the excessive number of hyperparameters and the lack of precise guidance on semantic similarity. To address these problems, we propose a Parameter Adaptive Contrastive Hashing (PACH) method for multimedia retrieval. The Fast Parameter Adaptive (FPA) module, combined with the powerful space exploration and dynamic optimization capabilities of reinforcement learning, designs a hot-plugging multimedia hashing method parameter adaptation module to solve for an approximate optimal combination of parameters. The Multimedia Contrastive Hashing (MCH) module comprehensively explores intra- and inter-modal semantic consistency of multimodal data, enriching the cross-modal semantic information of the hash codes. Comprehensive experiments were designed and compared with the latest hash learning methods, verifying the effectiveness and superiority of the PACH method. The code is available at https://github.com/YunfeiChenMY/PACH.

An Approach to Rapidly Evaluating Rock Mass Quality in Underground Engineering Based on Multi-source Heterogeneous Data

An Approach to Rapidly Evaluating Rock Mass Quality in Underground Engineering Based on Multi-source Heterogeneous Data

Enhanced stock price prediction with optimized ensemble modeling using multi-source heterogeneous data: Integrating LSTM attention mechanism and multidimensional gray model

The prediction of stock prices is a complex task due to the influence of various factors, high noise, and nonlinearity. This paper focuses on addressing the challenges of low prediction accuracy and poor stability, which have been a key area of interest in academic research. We proposed an optimized ensemble model that combines an LSTM-based attention mechanism and a cyclic multidimensional gray model, utilizing multi-source heterogeneous data. Our results demonstrate that the ensemble model achieves improved prediction accuracy, exhibits a good fitting effect, and outperforms individual models. The ensemble model yields smaller Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), and Root Mean Square Error (RMSE) values compared to the LSTM-based attention mechanism model and the multidimensional gray model. Furthermore, the ensemble model shows enhanced coefficient of determination (R2). Comparative analysis with alternative models such as ARIMA, GRU, CNN, and CNN-GRU reveals that the ensemble model achieves significant advancements in prediction accuracy.

Research and Prediction of China's Air Environment

Abstract: With the rapid development of China's economy, environmental issues, particularly air pollution, have become increasingly severe, posing significant threats to human health and ecosystems. This study employed statistical learning methods to analyze air pollution composition data from thirty major Chinese cities throughout 2019-2022. The study used algorithms like regression, classification, fitting, and filtering to find changes in multi-source heterogeneous data patterns and predict how air pollutant concentrations would change over time and space. At the same time,the results indicate that the primary factors influencing the Air Quality Index (AQI) are fine particulate matter (PM2.5) and inhalable particulate matter (PM10). Additionally, this research is significant for improving public health, advancing environmental protection, fostering economic growth, strengthening laws and regulations, ensuring social stability, and promoting scientific and cultural advancements. Through this research, scientific bases can be provided for formulating policies that reduce disease, enhance government credibility, drive innovation in atmospheric research, and preserve cultural heritage.

Spatial optimization strategies for China’s hydrogen infrastructure industry chain

Promoting the development of China’s hydrogen energy industry is crucial for achieving green energy transition. However, existing research lacks systematic studies on the spatial layout of the hydrogen industry chain. This study constructed a comprehensive theoretical framework encompassing hardware infrastructure, software systems, and soft power. Using multi-source heterogeneous data, GIS analysis, and NVivo text coding methods, the current regional layout and challenges of China’s hydrogen infrastructure industry chain were systematically evaluated. The findings determined that economically developed eastern regions lead in infrastructure and soft power, while central and western regions leverage their resource and manufacturing advantages. Major challenges include regional imbalances in hardware infrastructure, uneven distribution of soft power, and misalignment between software systems and actual needs. Analysis of the “14th Five-Year Plan” of various regions elucidated deep insights into the diversity of local hydrogen energy development strategies, identifying five types of hydrogen cities: resource-advantaged, market-oriented, regionally collaborative, innovation-driven, and policy-supported. Accordingly, strategies to enhance industry chain synergy, clarify city roles, and optimize regional ecosystems were proposed. It is recommended to integrate hydrogen infrastructure with urban planning and incorporate environmental impact assessments into spatial optimization decisions. This study provides a systematic analytical framework and progressive policy recommendations for the efficient and green layout of China’s hydrogen infrastructure, offering important implications for the sustainable development of the hydrogen industry and other rapidly developing economies.

Biofuser: a multi-source data fusion platform for fusing the data of fermentation process devices.

In the past decade, the progress of traditional bioprocess optimization technique has lagged far behind the rapid development of synthetic biology, which has hindered the industrialization process of synthetic biology achievements. Recently, more and more advanced equipment and sensors have been applied for bioprocess online inspection to improve the understanding and optimization efficiency of the process. This has resulted in large amounts of process data from various sources with different communication protocols and data formats, requiring the development of techniques for integration and fusion of these heterogeneous data. Here we describe a multi-source fusion platform (Biofuser) that is designed to collect and process multi-source heterogeneous data. Biofuser integrates various data to a unique format that facilitates data visualization, further analysis, model construction, and automatic process control. Moreover, Biofuser also provides additional APIs that support machine learning or deep learning using the integrated data. We illustrate the application of Biofuser with a case study on riboflavin fermentation process development, demonstrating its ability in device faulty identification, critical process factor identification, and bioprocess prediction. Biofuser has the potential to significantly enhance the development of fermentation optimization techniques and is expected to become an important infrastructure for artificial intelligent integration into bioprocess optimization, thereby promoting the development of intelligent biomanufacturing.

Feasibility of Emergency Flood Traffic Road Damage Assessment by Integrating Remote Sensing Images and Social Media Information

In the context of global climate change, the frequency of sudden natural disasters is increasing. Assessing traffic road damage post-disaster is crucial for emergency decision-making and disaster management. Traditional ground observation methods for evaluating traffic road damage are limited by the timeliness and coverage of data updates. Relying solely on these methods does not adequately support rapid assessment and emergency management during extreme natural disasters. Social media, a major source of big data, can effectively address these limitations by providing more timely and comprehensive disaster information. Motivated by this, we utilized multi-source heterogeneous data to assess the damage to traffic roads under extreme conditions and established a new framework for evaluating traffic roads in cities prone to flood disasters caused by rainstorms. The approach involves several steps: First, the surface area affected by precipitation is extracted using a threshold method constrained by confidence intervals derived from microwave remote sensing images. Second, disaster information is collected from the Sina Weibo platform, where social media information is screened and cleaned. A quantification table for road traffic loss assessment was defined, and a social media disaster information classification model combining text convolutional neural networks and attention mechanisms (TextCNN-Attention disaster information classification) was proposed. Finally, traffic road information on social media is matched with basic geographic data, the classification of traffic road disaster risk levels is visualized, and the assessment of traffic road disaster levels is completed based on multi-source heterogeneous data. Using the “7.20” rainstorm event in Henan Province as an example, this research categorizes the disaster’s impact on traffic roads into five levels—particularly severe, severe, moderate, mild, and minimal—as derived from remote sensing image monitoring and social media information analysis. The evaluation framework for flood disaster traffic roads based on multi-source heterogeneous data provides important data support and methodological support for enhancing disaster management capabilities and systems.

Open Geospatial Engine: A Cloud-based Spatiotemporal Computing Platform

Abstract. The quantity and diversity of Earth spatiotemporal big data have significantly increased in recent years, providing the potential to comprehensively analyse complex spatiotemporal problems from new perspectives. However, integrating, managing, and applying multi-source heterogeneous Earth spatiotemporal big data remains a challenge. To address this issue, this study proposes a cloud-based spatiotemporal computing platform, the Open Geospatial Engine (OGE), for the unified organization and joint analysis of Earth spatiotemporal big data in multiple dimensions and scales. The framework of this platform comprises three modules: data management, computing engine, and service interface. The data management module adopts the GeoCube model to effectively integrate and manage multi-source heterogeneous spatiotemporal data through multi-dimensional aligned tiles. The computing engine module seamlessly maps the GeoCube model to the cloud environment by extending Spark RDD, making the GeoCube model capable of high-performance distributed computing. Following OGC specifications, the service interface module integrates and shares data, operators, and spatiotemporal analysis models through extensible APIs in an interactive development environment. Combined with a series of high-performance optimization techniques, OGE simplifies data queries and the construction of complex analytical applications by integrating these three modules. The applicability of OGE is demonstrated by case studies involving multi-dimensional queries and joint analysis of long-time-series and heterogeneous spatiotemporal data.

Research on Semantic Driven Urban Pipeline Dataspace Construction Method

Abstract. Urban pipeline data is heterogeneous in multiple sources and rich in data volume, and there are problems such as data conflict and difficult organization and management due to the heterogeneity of multiple sources when accessing the data in large-scale concurrently. To address this problem, this paper proposes a semantics-driven urban pipeline dataspace construction method, which aims to realize the efficient organization of pipeline data. Firstly, this method combines the classification and characteristics of urban pipelines, and expresses the semantic information of pipeline geographic entities from four dimensions: semantic description, spatial location, attribute characteristics and time evolution. Then, the four expression sets are embedded into the dataspace RDF model as predicates, and the associated description mechanism of pipeline geographic entities is established by means of genus classes and so on, so as to construct the pipeline dataspace RDF model. Finally, the model is stored and graphically visualized using neo4j to achieve fast retrieval of data within the pipeline dataspace. The research results show that this method provides a unified expression of pipeline entities, solves the problem of pipeline multi-source heterogeneous data conflict and organization difficulties, and improves the efficiency of multi-source heterogeneous pipeline data organization while ensuring the integrity of pipeline information to the maximum extent.

Communication Technology for Renewable Energy Access Grid System Based on Multi-source Heterogeneous Data Fusion

Abstract Incorporating renewable energy solutions into the power grid is essential for facilitating a shift in energy sources. Ensuring the effectiveness and dependability of communication technologies within smart distribution grids is vital, as these grids serve as the link between end users and the central station, thereby contributing to the grid’s stable functioning and the effective dissemination of energy. This study introduces a framework for communication that tackles the challenges associated with the integration of multi-source, heterogeneous data in smart distribution grids. The framework leverages multi-source heterogeneous data fusion and capitalizes on edge computing to efficiently process and consolidate data from diverse sensors and devices. An optimal weight allocation algorithm is proposed to mitigate data redundancy and to improve the energy efficiency of communication. The paper demonstrates, through theoretical examination and empirical tests, the proposed system’s enhanced communication capabilities, especially in terms of data transmission effectiveness and energy usage optimization. The discussion highlights the advantages of smart distribution networks over conventional communication methods and offers insights for further advancements in communication technologies within this sector.

An unsupervised learning framework for health diagnosis by incorporating multiscale data

Integrating real-time evaluations of health states with Structural Health Monitoring (SHM) data and environmental observations is crucial, particularly under large datasets. Not only is the volume of real-time data analysis substantial, but traditional algorithms like wavelet analysis often require extensive prior work and significant manual input, limiting the application of SHM and big data technologies in structural analysis. To overcome this limitation, this study introduces an unsupervised analytical framework coupling a Denoising Sparse Wavelet Network (DeSpaWN) and a dynamic-inner Principal Component Analysis (DiPCA) algorithm. The DeSpaWN is designed to extract sparse representation from multiscale sensor signals, and the features are subsequently analyzed using the DiPCA to assess health diagnoses. The DeSpaWN is specifically tailored for monitoring high-frequency time series dynamically and adaptively, enabling reconstruction and extraction of signals across various frequency domains. Additionally, the DiPCA effectively handles large-scale time-series data, providing precise evaluations by accounting for dynamic latent correlations. To validate the method, a 5-Degree of Freedom (DoF) structure model is established, equipped with five virtual sensors at each DoF and an anemometer at the top DoF. The sensor signals, obtained under different health states and subjected to artificial excitations, are input into the proposed framework. Validation results demonstrate the effectiveness in synthesizing multi-sensor data and assessing structural integrity. The framework's practical utility is showcased through a real-world case study that assesses the structural integrity of a wind turbine. By integrating real and virtual sensor data collected over a year and cross-referencing this with the fault log, the study validates the algorithm's efficacy in processing multisource heterogeneous data and its precision in structural health diagnosis. The study substantiates real-time tracking of health states for SHM-equipped structures, underscoring its potential to enhance safety and reliability in civil infrastructure.

Code and Data Repository for Fraud Detection by Integrating Multisource Heterogeneous Presence-Only Data

This repository provides data for the problem and code for the method. The main folders are data and code.

Fraud Detection by Integrating Multisource Heterogeneous Presence-Only Data

Fraud Detection by Integrating Multisource Heterogeneous Presence-Only Data

Analysis of Short-Term Heavy Rainfall-Based Urban Flood Disaster Risk Assessment Using Integrated Learning Approach

Accurate and timely risk assessment of short-term rainstorm-type flood disasters is very important for ecological environment protection and sustainable socio-economic development. Given the complexity and variability of different geographical environments and climate conditions, a single machine learning model may lead to overfitting issues in flood disaster assessment, limiting the generalization ability of such models. In order to overcome this challenge, this study proposed a short-term rainstorm flood disaster risk assessment framework under the integrated learning model, which is divided into two stages: The first stage uses microwave remote sensing images to extract flood coverage and establish disaster samples, and integrates multi-source heterogeneous data to build a flood disaster risk assessment index system. The second stage, under the constraints of Whale Optimization Algorithm (WOA), optimizes the integration of random forest (RF), support vector machine (SVM), and logistic regression (LR) base models, and then the WRSL-Short-Term Flood Risk Assessment Model is established. The experimental results show that the Area Under Curve (AUC) accuracy of the WRSL-Short-Term Flood Risk Assessment Model is 89.27%, which is 0.95%, 1.77%, 2.07%, 1.86%, and 0.47% higher than RF, SVM, LR, XGBoost, and average weight RF-SVM-LR, respectively. The accuracy evaluation metrics for accuracy, Recall, and F1 Score have improved by 5.84%, 21.50%, and 11.06%, respectively. In this paper, WRSL-Short-Term Flood Risk Assessment Model is used to carry out the risk assessment of flood and waterlogging disasters in Henan Province, and ArcGIS is used to complete the short-term rainstorm city flood and waterlogging risk map. The research results will provide a scientific assessment basis for short-term rainstorm city flood disaster risk assessment and provide technical support for regional flood control and risk management.

Multi-granularity spatiotemporal object modelling of waterborne traffic elements

The electronic navigational charts are crucial carriers for representing the multi-source heterogeneous data of Waterborne Traffic Elements (WTEs). However, their layer-based modelling method has some shortcomings in expressing the multi-granularity features, complex relationships, and dynamic evolution of elements. This paper proposes an objectification modelling method for WTEs based on the concept of multi-granularity spatiotemporal object modelling. A classification system for waterborne traffic objects is developed based on the relevance of behavior to elements; combining characteristics of waterborne traffic, a data model for waterborne traffic objects is constructed from eight aspects: spatiotemporal reference, spatiotemporal position, spatial form, basic information, attributes, behavioral ability, structure, and associative relationships. An object extraction function is also established, extracting object attributes and relationships between objects according to different element classes. Taking the Jiashan section of the Hangzhou-Shanghai Line in Zhejiang Province as the experimental subject, the multi-granularity spatiotemporal characteristics, dynamic evolution, and relationship expression of channel class objects are tested. The experimental results show that the proposed method provides the theoretical basis and data organization mode for the multi-granularity expression of WTEs.

Research on the traffic flow measurement method of intelligent connected vehicles based on series and cascade technology

At present, the major cities in our country have entered the development of smart cities, the utilization rate and prevalence rate of intelligent connected cars in smart cities have reached a new peak,to manage the real-time traffic flow of intelligent connected vehicles effectively in smart cities. This paper proposes to calculate the flow of intelligent connected vehicles based on series cascade technology. Firstly, this paper collects multi-source heterogeneous data based on multiple collectors; Secondly, a generative adversarial network is employed for data fusion, synthetic data, closely mirroring the original, is generated for handling unknown variables during model training. Next, an autoencoder model generates input data for feature representation. Lastly, CatBoost, a regression model that leverages a cascaded machine learning algorithm, is utilized to forecast the traffic flow of intelligent connected vehicles in smart cities. The results show that the method based on CASCaded machine learning has remarkable effect in predicting smart city traffic flow and alleviating urban road congestion.

Multi-modal fusion deep learning model for excavated soil heterogeneous data with efficient classification

Multimodal fusion, a cutting-edge method aiming to integrate heterogeneous data for prediction tasks, has received limited attention in geotechnical engineering. This study focuses on classifying excavated soils with large quantities rapidly and finely. By using an excavated soil information collecting system at the largest soil transferring platform in China, 3,243 groups of multi-source heterogenous data were created, including soil images, spatial series data of cone index curves, time series data of TDR waveforms, discrete net weight data, and textual descriptions of soil morphology. After data augmentation, a big dataset containing 23,122 sets with labels based on the Unified Soil Classification System, moisture content, and mineral composition was created. Multimodal deep learning models were trained using different fusion strategies: 7 early fusion, 3 intermediate fusion, and 2 late fusion cases. Performance metrics were evaluated, including loss, accuracy, F1 Score, precision, recall, specificity, NPV, FPR, and AUROC. Results showed that intermediate fusions performed best, while late fusions performed the poorest. The two-stage intermediate fusion with five modalities achieved the best results, achieving an accuracy above 0.99 on the test set. This multimodal fusion approach effectively explores the correlation between multi-source heterogeneous data of soils, leading to scientific and engineering value in geotechnics.

An outlier identification method based on parallel clustering and the Adaboost algorithm for power multi-source heterogeneous data

Abstract Influenced by the diversity of data scheduling and the multi-level differentiation of data distribution, the multi-source heterogeneous data of electric power have serious deviations in space and time scheduling, which makes it difficult to identify outliers. Therefore, a method for identifying outliers of multi-source heterogeneous data of electric power based on parallel clustering and the Adaboost algorithm is proposed. A sampling node model of heterogeneous data is constructed, and random numbers that obey Gaussian distribution are generated according to the mean value of data, and their mean square loss values are optimized to form an objective function of abnormal values of multi-source heterogeneous data of electric power. According to parallel clustering and the Adaboost algorithm, a unified format is created to obtain the load control results of abnormal values of multi-source heterogeneous data of electric power, so as to realize the identification of abnormal values of multi-source heterogeneous data of electric power. The experimental results show that at the time point of 18 s, the data identification quality of this method has reached 80, and with the increase of time, its data identification quality remains stable. It is demonstrated that the method has good recognition results.