Heterogeneous Data Fusion Research Articles

Predicting Visual Outcome Following Surgery in Patients with Idiopathic Epiretinal Membrane Using A Novel Convolutional Neural Network.

To develop a deep convolutional neural network (CNN) that enables the prediction of postoperative visual outcomes following epiretinal membrane (ERM) surgery based on preoperative optical coherence tomography (OCT) images and clinical parameters to refine surgical decision-making. A total of 529 patients with idiopathic ERM who underwent standard vitrectomy with ERM peeling surgery by two surgeons between January 1, 2014, and June 1, 2020, were enrolled. The newly developed Heterogeneous Data Fusion Net (HDF-Net) was introduced to predict postoperative visual acuity (VA) outcomes (improvement ≥ 2 lines in Snellen chart or not) 12 months after surgery based on preoperative cross-sectional OCT images and clinical factors, including age, sex, and preoperative VA. The predictive accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) of the CNN model were evaluated. The developed model demonstrated an overall accuracy for visual outcome prediction of 88.68% (95% CI, 79.0%-95.7%) with an AUC of 97.8% (95% CI, 86.8%-98.0%), sensitivity of 87.0% (95% CI, 67.9%-95.5%), specificity of 92.9% (95% CI, 77.4%-98.0%), precision of 0.909, recall of 0.870, and F1 score of 0.889. The heatmaps identified the critical area for prediction as the ellipsoid zone of photoreceptors and the superficial retina which was subjected to tangential traction of the proliferative membrane. The novel HDF-Net demonstrated high accuracy in the automated prediction of visual outcomes after weighing and leveraging multiple clinical parameters, including OCT images. This approach may be helpful in establishing personalized therapeutic strategies for ERM management.

Heterogeneous sensor data fusion for multiscale, shape agnostic flaw detection in laser powder bed fusion additive manufacturing

ABSTRACT We developed and applied a novel approach for shape agnostic detection of multiscale flaws in laser powder bed fusion (LPBF) additive manufacturing using heterogenous in-situ sensor data. Flaws in LPBF range from porosity at the micro-scale (< 100 µm), layer related inconsistencies at the meso-scale (100 µm to 1 mm) and geometry-related flaws at the macroscale (> 1 mm). Existing data-driven models are primarily focused on detecting a specific type of LPBF flaw using signals from one type of sensor. Such approaches, which are trained on data from simple cuboid and cylindrical-shaped coupons, have met limited success when used for detecting multiscale flaws in complex LPBF parts. The objective of this work is to develop a heterogenous sensor data fusion approach capable of detecting multiscale flaws across different LPBF part geometries and build conditions. Accordingly, data from an infrared camera, spatter imaging camera, and optical powder bed imaging camera were acquired across separate builds with differing part geometries and orientations (Inconel 718). Spectral graph-based process signatures were extracted from this heterogeneous thermo-optical sensor data and used as inputs to simple machine learning models. The approach detected porosity, layer-level distortion, and geometry-related flaws with statistical fidelity exceeding 93% (F-score).

A Federated Platform Enabling a Systematic Collaboration Among Devices, Data and Functions for Smart Mobility

Through the vast adoption and application of emerging technologies, the intelligence and autonomy of smart mobility can be substantially elevated to address more diversified demands and supplies. Along with this trend, a systematic collaboration among three essential elements of smart mobility services, namely devices, data and functions, is being studied to comprehensively break down the intrinsic barriers that existed in current solutions, to support the integration of connectable devices, the fusion of heterogeneous data, the composability of reusable functions, and the flexibility in their cooperations. To enable such a collaboration, this paper proposes a federated platform, called Future Mobility Sensing Advisor (FMSA), which can 1) manage the three elements through standardized interfaces separately and uniformly; 2) create a fully connected knowledge graph to orchestrate the three elements efficiently and effectively; 3) support the client-server interaction in centralized and federated modes to handle service requests and edge resources with various availability and accessibilities jointly and adaptively; and 4) accommodate various mobility services to foster harmonious and sustainable mobility tenderly and invisibly. Moreover, the efficiency and effectiveness of the platform are also tested through a performance evaluation, and a pilot supported at the Great Boston Area, respectively. As a result, it shows that FMSA can 1) achieve high performance by using the two interaction modes selectively, and 2) renovate smart mobility towards sustainability through personalized services that can measure user preferences and system objectives mutually.

Research on Heterogeneous Data Fusion Technology of Substation Secondary Equipment

In intelligent substations, the MMS Lite software package is widely used to realize station control layer network communication based on IEC61850. The protocol uses the BER coding of ASN.1, so there is more redundant information, which affects the transmission efficiency. Under the background of domestic independent control in the domestic power industry, there have been increasingly urgent demands for transforming and upgrading intelligent substations based on MMS protocol. This paper proposes a heterogeneous data fusion method for the secondary equipment of substation monitoring systems, which supports the simultaneous access and data interaction of the bay-level equipment of CMS protocol and MMS protocol. The thread synchronization mechanism is studied to avoid the avalanche data loss caused by transient failure, and improve the safety and reliability of data operation and maintenance, which further improves the support for the operation monitoring, remote access, and other services of the centralized control master station.

Graph-Reinforcement-Learning-Based Task Offloading for Multiaccess Edge Computing

Network applications involve massive heterogeneous data fusion and analysis. Artificial intelligence can significantly improve the convenience and user experience, but it requires a lot of storage, bandwidth, and computing resources. Multiaccess edge computing (MEC) extends intelligence services to IoT devices through offloading approaches and joint processing, which solves the resource bottleneck. However, designing advanced collaboration technology to offload tasks to MEC servers is still challenging. Heuristic algorithms and deep reinforcement learning (DRL)-based approaches have been proposed to offload tasks and minimize application latency. However, heuristic algorithms heavily depend on accurate mathematical models for the MEC system, and DRL does not make fair use of the relationship between devices in the MEC graph. To solve this, we propose a task offloading mechanism based on graph neural network (GNN), which can directly learn on graph data with messages passing and aggregation. We propose a graph reinforcement learning-based offloading (GRLO) framework, which models MEC as an acyclic graph and the offloading policy by graph state migration. GRLO combines GNN with the actor-critic network and trains offloading decision makers without labels. To efficiently train the GRLO, we propose a method that quickly explores action space and approaches the optimal solution. The numerical results show that the GRLO has lower latency compared to baselines while having generalization ability to new environments and topologies. Moreover, we verified the effectiveness of GRLO on a prototype.

Social Media Driven Big Data Analysis for Disaster Situation Awareness: A Tutorial

Situational awareness tries to grasp the important events and circumstances in the physical world through sensing, communication, and reasoning. Tracking the evolution of changing situations is an essential part of this awareness and is crucial for providing appropriate resources and help during disasters. Social media, particularly Twitter, is playing an increasing role in this process in recent years. However, extracting intelligence from the available data involves several challenges, including (a) filtering out large amounts of irrelevant data, (b) fusion of heterogeneous data generated by the social media and other sources, and (c) working with partially geo-tagged social media data in order to deduce the needs of the affected people. Spatio-temporal analysis of the data plays a key role in understanding the situation, but is available only sparsely because only a small fraction of people post relevant text and of those very few enable location tracking. In this paper, we provide a comprehensive survey on data analytics to assess situational awareness from social media big data.

Towards sparse matrix operations: graph database approach for power grid computation

The construction of new power systems presents higher requirements for the Power Internet of Things (PIoT) technology. The “source-grid-load-storage” architecture of a new power system requires PIoT to have a stronger multi- source heterogeneous data fusion ability. Native graph databases have great advantages in dealing with multi-source heterogeneous data, which make them suitable for an increasing number of analytical computing tasks. However, only few existing graph database products have native support for matrix operation-related interfaces or functions, resulting in low efficiency when handling matrix calculations that are commonly encountered in power grids. In this paper, the matrix computation process is expressed by a strategy called graph description, which relies on the natural connection between the matrix and structure of the graph. Based on that, we implement matrix operations on graph database, including matrix multiplication, matrix decomposition, etc. Specifically, only the nodes relevant to the computation and their neighbors are concerned in the process, which prunes the influence of zero elements in the matrix and avoids useless iterations compared to the conventional matrix computation. Based on the graph description, a series of power grid computations can be implemented on graph database, which reduces redundant data import and export operations while leveraging the parallel computing capability of graph database. It promotes the efficiency of PIoT when handling multi-source heterogeneous data. An comprehensive experimental study over two different scale power system datasets compares the proposed method with Python and MATLAB baselines. The results reveal the superior performance of our proposed method in both power flow and N-1 contingency computations.

Adversarial Learning Networks for FinTech Applications Using Heterogeneous Data Sources

The dynamic property and increasing complexity are the key challenges for modeling financial technology (FinTech)-related applications such as stock markets. Over the years, a lot of inflexible predictive strategies have been proposed for predicting stock price movements that failed to achieve satisfactory results especially when a market crash occurs. To cope with this challenge, we propose a prediction framework based on an adversarial training strategy using reinforcement learning for the said FinTech application. The framework uses a heterogeneous knowledge base, including stock prices, tweets, and global indicators. We propose a modified newton-divided difference polynomial (NDDP) for missing data imputation. The informative patterns representing the intrinsic characteristics of financial markets were extracted using long short-term memory networks (LSTM). The two adversarial networks are heterogeneous data fusion representing market crash (HDFM) <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning and confrontational <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning network. Both networks are trained in an adversarial fashion to increase the effectiveness of prediction even when the financial market is volatile. The experimental results show the importance of global indicators and the proposed adversarial learning network (ALN) for improving the predictive performance in comparison with the existing state-of-the-art works.

A Novel Graph-Based Framework for Classifying Urban Functional Zones with Multisource Data and Human Mobility Patterns

Recent research has shown the advantages of incorporating multisource geospatial data into the classification of urban functional zones (UFZs), particularly remote sensing and social sensing data. However, the effects of combining datasets of varying quality have not been thoroughly analyzed. In addition, human mobility patterns from social sensing data, which capture signals of human activities, are often represented by origin-destination pairs, thus ignoring spatial relationships between UFZs embedded in mobility trajectories. To address the aforementioned issues, this study proposed a graph-based UFZ classification framework that fuses semantic features from high spatial resolution (HSR) remote sensing images, points of interest, and GPS trajectory data. The framework involves three main steps: (1) High-level scene information in HSR remote sensing imageries was extracted through deep neural networks, and multisource semantic embeddings were constructed based on physical features and social sensing features from multiple geospatial data sources; (2) UFZ mobility graph was constructed by spatially joining trajectory information with UFZs to construct topological connections between functional parcel segments; and (3) UFZ segments and multisource semantic features were transformed into nodes and embeddings in the mobility graphs, and subsequently graph-based models were adopted to identify UFZs. The proposed framework was tested on Zhuhai and Singapore datasets. Results indicated that it outperformed traditional classification methods with an overall accuracy of 76.7% and 84.5% for Zhuhai and Singapore datasets, respectively. The proposed framework contributes to literature in heterogeneous data fusion and is generalizable to other UFZ classification scenarios where human mobility patterns play a role.

Non-contact diagnosis for gearbox based on the fusion of multi-sensor heterogeneous data

Non-contact sensing technology plays an important role in the health monitoring of the gearbox. However, a single non-contact measurement is challenging to achieve the simultaneous monitoring of both structural and non-structural damages. In order to explore the fusion mechanism of multi-sensor heterogeneous measurements, acoustic and thermal characteristics of the gearbox under typical fault states are analyzed, and it is verified the fusion of infrared thermal (IRT) images and acoustic data integrates complementary fault information. In this paper, an attention-enhanced information fusion diagnosis network (AIFN-IA) is proposed for the complementary fusion of IRT images and acoustic data. Firstly, the acoustic data is converted into images by the non-hyperparameter encoding method and then fused with IRT images in data-level. Secondly, the limited shuffle attention module is designed to adaptively focus on the fault elements hidden in the complex fusion features. Finally, experimental data verify the effectiveness of the proposed AIFN-IA method in recognizing six structural and non-structural damages of the gearbox. Compared with seven state-of-the-art methods, the proposed AIFN-IA method performs best in extracting discriminating features with the highest diagnosis accuracy. Moreover, the proposed AIFN-IA method can still achieve satisfactory results under the challenges of small sample datasets and strong noise interference, which is more competitive in real industrial applications.

Towards Digital Twinning on the Web: Heterogeneous 3D Data Fusion Based on Open-Source Structure

Recent advances in Computer Science and the spread of internet connection have allowed specialists to virtualize complex environments on the web and offer further information with realistic exploration experiences. At the same time, the fruition of complex geospatial datasets (point clouds, Building Information Modelling (BIM) models, 2D and 3D models) on the web is still a challenge, because usually it involves the usage of different proprietary software solutions, and the input data need further simplification for computational effort reduction. Moreover, integrating geospatial datasets acquired in different ways with various sensors remains a challenge. An interesting question, in that respect, is how to integrate 3D information in a 3D GIS (Geographic Information System) environment and manage different scales of information in the same application. Integrating a multiscale level of information is currently the first step when it comes to digital twinning. It is needed to properly manage complex urban datasets in digital twins related to the management of the buildings (cadastral management, prevention of natural and anthropogenic hazards, structure monitoring, etc.). Therefore, the current research shows the development of a freely accessible 3D Web navigation model based on open-source technology that allows the visualization of heterogeneous complex geospatial datasets in the same virtual environment. This solution employs JavaScript libraries based on WebGL technology. The model is accessible through web browsers and does not need software installation from the user side. The case study is the new building of the University of Twente—Faculty of Geo-Information (ITC), located in Enschede (the Netherlands). The developed solution allows switching between heterogeneous datasets (point clouds, BIM, 2D and 3D models) at different scales and visualization (indoor first-person navigation, outdoor navigation, urban navigation). This solution could be employed by governmental stakeholders or the private sector to remotely visualize complex datasets on the web in a unique visualization, and take decisions only based on open-source solutions. Furthermore, this system can incorporate underground data or real-time sensor data from the IoT (Internet of Things) for digital twinning tasks.

ACU&amp;MOX-DATA: A platform for fusion analysis and visual display acupuncture multi-omics heterogeneous data

A new network platform named ACU&amp;MOX-DATA (http://cbcb.cdutcm.edu.cn/ACU&amp;MOX-DATA/) was recently set up and introduced. The purpose of this website is to integrate and analyze multi-omics data of acupuncture and moxibustion and serve as a disease, clinical, and laboratory experiment public database. Some existing functions and the future plan of the website were also discussed. This platform can help researchers to integrate analyses and visual interpretation of key scientific issues in acupuncture &amp; moxibustion under the guidance of traditional Chinese medicine theory through heterogeneous and multi-level data fusion analysis.

A Network Security Situation Awareness Method Based on GRU in Big Data Environment

Aiming at the “bottleneck” problems of the traditional network security situation awareness model, such as large equipment limitations, single data source and poor integration ability, weak level of autonomous learning and data mining, a network security situation awareness framework suitable for big data is constructed. A gate recurrent unit (GRU) model is established to effectively extract features from the situation data set through the deep learning algorithm of big data. It is a method to automatically mine and analyze the hidden relationship and change trend of network security situation, realize the high-speed acquisition and fusion of massive multi-source heterogeneous data, and perceive the network security situation from an all-round perspective. The experimental results show that this method has a good awareness effect on network threats, and has strong representation ability in the face of network threats. It can effectively perceive the network threat situation without relying on data labels, which verifies that this method can effectively improve the efficiency and accuracy of security situation awareness.

Wind Turbine Fault Detection With Multimodule Feature Extraction Network and Adaptive Strategy

Unexpected failures in wind turbines (WTs) lead to tremendous economic losses and safety hazards to wind farms. Intelligent condition monitoring and fault detection can prevent unscheduled downtime and improve reliability and efficiency. However, the variational external environments and operation states result in time-varying correlations and fluctuations of variables, which pose challenges to precise modeling and detection for WTs. This article proposes a dual-stage temperature monitoring and fault detection method, which can be applied to multiple key components in WTs. First, a novel multimodule feature extraction network (MMFENet) is designed to model the normal behavior of temperature data in the monitoring component. The proposed model decomposes the correlations into distinct patterns and devises different modules for quantification. Specifically, the static module aims to characterize the stationary correlations resulting from the inherent structure to provide reference expectations of fluctuations in the dynamic module and accelerate the model training. The dynamic module is devoted to depicting time-varying interdependence caused by variational operation states to pursue precise modeling. In addition, the proposed network also uses a heterogeneous data fusion module to use external information. Second, an adaptive detection strategy is devised for converting the prediction errors into detection results. Based on the evaluation results in the training procedure, parameter ranges during online detection can be adaptively determined to contain multiple rational values, which can enhance robustness and scalability. Experimental results using field data from real wind farms demonstrate the effectiveness and superiority of the proposed method.

Analysis of Substation Joint Safety Control System and Model Based on Multi-Source Heterogeneous Data Fusion

As the number of substations continues to increase globally and the market demand continues to rise, the current workload of maintenance and daily operation of substations in power grids cannot meet the current demand if only relying on manual work, and the design and implementation of intelligent safety control solutions for substations is imperative. Therefore, this paper proposes a joint safety control system and model analysis for substations based on multi-source heterogeneous data fusion. Firstly, a three-dimensional visualization substation efficient interactive operation platform is realized, which realizes the functions of substation scene roaming, system login, information management, equipment parameters, status viewing and operation ticket pushing; after that, a variety of intelligent hardware devices for data collection, such as multi-dimensional terminal sensors, intelligent wearable devices, intelligent pre-built positioning installation measure rod, and substation intelligent inspection robots are designed to greatly improve the substation inspection efficiency and realize real-time monitoring and data interaction in the inspection process. Finally, we propose an Attention-LSTM-based prediction model for substation multidimensional data, which can predict power equipment spatio-temporal data in the short term, and the prediction results can be combined with intelligent devices for joint diagnosis. The Attention-LSTM prediction model is well-trained in transformer oil temperature experiments, and the experimental results show that this model can provide early warning for the abnormal state of substation power equipment. In summary, this thesis describes a set of complete and practically feasible intelligent safety control methods for substations. The joint safety control system and model analysis of the substation based on multi-source heterogeneous data fusion designed in this paper is mainly oriented to the substation as an electric power workplace, which has quite a vast application prospect for energy equipment.

A multi-source heterogeneous spatial big data fusion method based on multiple similarity and voting decision

A multi-source heterogeneous spatial big data fusion method based on multiple similarity and voting decision

Landscape Ecological Risk Assessment and Planning Enlightenment of Songhua River Basin Based on Multi-Source Heterogeneous Data Fusion

In this study, the remote sensing images of the 30 km buffer zone from Zhaoyuan to Baidu of the Songhua River, which is rich in land use types and frequent in human activities, were selected as the research object to analyze land use change and driving factors. The objective of this research is to evaluate the ecological risk of watershed landscapes and provide a basis for watershed ecological environment protection and planning. On this basis, the landscape pattern index data were extracted, and a three-dimensional comprehensive index system of the natural, social and landscape pattern was constructed. In addition, based on the spatial principal component analysis (SPCA), data fusion was carried out to improve the accuracy and comprehensiveness of landscape ecological risk assessment results. The risk level of watershed landscape ecology was divided into low ecological risk, medium-low ecological risk, medium ecological risk, medium-high ecological risk, and high ecological risk by the Natural Breaks method. According to the results of the landscape ecological risk assessment and the characteristics of risk sources in each risk level area, the ecological protection and planning enlightenment suitable for each risk level area were obtained. The research content can provide ideas and evidence for environmental managers to formulate ecological risk protection strategies and reduce the impact of ecological risk threats.

Sea ice image classification based on ResFPG network and heterogeneous data fusion

ABSTRACT Sea ice detection has played an important role in climate protection and strategic deployment. Due to the autonomous learning characteristics of deep learning, it has been gradually applied to the classification of remote sensing sea ice images. At present, deep learning models are mostly used in sea ice classification of single source remote sensing data. Due to the limitations of single source data and the information loss of deep learning model in the process of feature extraction layer by layer, it is inevitable to encounter a bottleneck in sea ice detection requiring fine classification. To solve the above problems, this paper proposes a sea ice image classification method based on ResNet16-feature pyramid networks-spatial pyramid pooling-gated fusion network (ResFPG) and heterogeneous data fusion. In the feature extraction part, the method uses the improved ResNet16 to extract the multi-level feature information of sea ice in synthetic aperture radar data and optical data, reducing information loss in the feature extraction process, then mines and fuses the low-level spatial information and high-level semantic information through the improved feature pyramid networks (FPN), and then collects and fuses the output features of different scales through the spatial pyramid pooling (SPP) network. In the feature fusion part, a gated feature-level fusion strategy is designed to further improve the overall classification accuracy by adaptively adjusting the feature contribution of two heterogeneous sources of data through the gated fusion network (GFN). In order to verify the effectiveness of this method, we use two sets of heterogeneous sea ice remote sensing data located in Hudson Bay area for experiments. The experimental results show that compared with other image classification methods, the proposed method fully excavates and integrates the multi-scale and multi-level features in heterogeneous data, effectively distinguishes the feature contribution, and achieves better classification results (97.14% and 95.85%).

Multi-source Heterogeneous Data Fusion Model Based on FC-SAE

&lt;p&gt;Multi-source heterogeneous data has different degrees of data correlation or data conflict. How to fuse this data and fully mine its inherent meanings to obtain more accurate decision information is a problem that needs to be solved urgently. This paper proposes a multi-source heterogeneous data fusion model based on fully connected layers and sparse autoencoders (short for FC-SAE) to solve the above problem. This model can effectively improve the time series forecasting performance compared with the traditional time series forecasting model. The MAE value is reduced by 4.4% and the RMSE value is reduced by 3.7%. In terms of fusion strategy, the method that uses the sparse autoencoder as the fusion strategy reduces the MAE value by 1.7% and the RMSE value by 2.3% compared with the method that uses the fully connected layer as the fusion strategy.&lt;/p&gt; &lt;p&gt;&amp;nbsp;&lt;/p&gt;

Remote sensing sea ice classification based on DenseNet and heterogeneous data fusion

Most remote sensing sea ice classification methods use single-source remote sensing data, such as synthetic aperture radar (SAR) data and optical remote sensing data. SAR data contain rich sea ice texture information, but the data are relatively single, making it difficult to distinguish detailed sea ice categories. Optical data include abundant spatial-spectral information, but they are often affected by clouds, fog, and severe weather. Hence, given the limitations of single-source data, the remote sensing sea ice classification accuracy cannot be further improved. A sea ice classification method based on deep learning and multisource remote sensing data fusion is proposed utilizing an improved densely connected convolutional neural network (DenseNet) to mine and fuse the multilevel features of sea ice. According to the characteristics of SAR data and optical data, a dual-branch network structure based on an improved DenseNet is employed for feature extraction, and a squeeze-and-excitation attention mechanism is introduced to weight the fusion features to further enhance the feature weights that can effectively distinguish different types of sea ice. The fully connected network is used to perform the deep fusion of features and classify sea ice. To verify the effectiveness of the proposed method, two sets of sea ice data are utilized for sea ice classification. The experimental results show that the proposed method fully excavates and fuses the multilevel characteristics of heterogeneous data using the improved dual-branch network structure, leverages the complementary characteristics of SAR data and optical data, significantly increases the sea ice classification accuracy, and effectively improves the influence of cloud cover on the sea ice classification accuracy. Compared with the typical single-source data classification method and other heterogeneous data fusion classification methods, the proposed method achieves superior overall classification accuracy (98.49% and 98.58%).