Network Information Fusion Research Articles

BIF-Net: Boundary information fusion network for abdominal aortic aneurysm segmentation

The accurate abdominal aortic aneurysm (AAA) segmentation is significant for assisting clinicians in diagnosis and treatment planning. However, existing segmentation methods exhibit a low utilization rate for the semantic information of vessel boundaries, which is disadvantageous for segmenting AAA with significant scale variability of vessel diameter (diameter ranges from 4 mm to 85 mm). To tackle this problem, we introduce a boundary information fusion network (BIF-Net) specially designed for AAA segmentation. BIF-Net initially constructs convolutional kernel functions based on Gabor and Sobel operators, enriching the global semantic features and localization information through the Gabor and Sobel dilated convolution (GSDC) module. Additionally, BIF-Net supplements lost boundary feature information during the sampling process through the guided filtering feature supplementation (GFFS) module and the channel-spatial attention module (CSAM), enhancing the ability to capture targets with shape diversity and boundary features. Finally, we introduce a boundary feature loss function to alleviate the impact of the imbalance between positive and negative samples. The results demonstrate that BIF-Net outperforms current state-of-the-art methods across multiple evaluation metrics, achieving the highest Dice similarity coefficient (DSC) accuracies of 93.29 % and 91.01 % on the preoperative and postoperative datasets, respectively. Compared to the state-of-the-art methods, BIF-Net improves DSC by 6.86 % and 3.85 %. Due to the powerful boundary feature extraction ability, the proposed BIF-Net is a competitive AAA segmentation method exhibiting significant potential for application in diagnosis and treatment processes of AAA.

A Bearing Fault Diagnosis Method Based on Vibration Signal Extension and Time–Frequency Information Fusion Network Under Small Sample Conditions

A Bearing Fault Diagnosis Method Based on Vibration Signal Extension and Time–Frequency Information Fusion Network Under Small Sample Conditions

Fault diagnosis of photovoltaic array with multi-module fusion under hyperparameter optimization

Photovoltaic (PV) arrays’ random and intermittent output characteristics impact power system safety. To improve the performance of the PV array fault diagnosis model, a novel online fault monitoring technique is introduced. (1) Fault diagnostic model construction: Significant differences in PV arrays’ I-V and P-V curves under various fault conditions led to constructing a 3D channel feature map based on I, V, and P features. (2) Multi-source information fusion network (MSIFN): this multi-module fusion model includes a time–frequency domain fusion module (TDFM), a multi-feature shuffle expansion convolution module (MSECM), a parameter-free parallel hybrid attention enhancement module, and a multi-scale mixed pooling fusion classification module (MMPCM). (3) Multi-strategy fusion whale optimization algorithm (MSFWOA): addressing the original WOA’s deficiencies, we designed time control, parameter modification, and greedy control strategies based on lens imaging to optimize MSIFN’s hyper-parameters. Experimental results show that the MSFWOA-MSIFN model excels in PV array fault diagnosis (Paccuracy=Pprecision=Precall = 99.92 %). In three types of noise experiments with 15 dB, 25 dB, and 30 dB, the average performance index remained above 99 %. In practical experiments, the average performance indices werePaccuracy = 97.53 %, Pprecision = 97.32 %, andPrecall = 97.41 %, further demonstrating its excellent diagnostic performance. This model effectively diagnoses various faults in PV arrays, providing scientific and theoretical support for PV system operations.

Online prediction of joint mechanical properties of FSW thick AA2219-T8 based on multi-source information fusion using 1DCNN

High strength AA2219-T8, Al-Cu alloy of 18 mm plate is welding by friction stir welding (FSW) technology and used for heavy-lift launch vehicles tank. During FSW thick AA2219-T8 plate, joint tensile strength and microhardness are affected by multiple interconnected variables including welding force, temperature, and geometric quantity of the butt face. The joint mechanical properties are difficult to predict and ensure. This paper proposed a hybrid predicting approach based on multi-source information fusion, called Down-Sampling and One-Dimensional Convolutional Neural Network (DS-1DCNN). Initially, FSW experiments were conducted to collect the axial force, welding surface temperature at advancing and retreating side, gap, and mismatch signals, which were fused through DS technology. To extract temporal information and features from signals, a 1DCNN prediction model was established, with the fused multi-source signals as input and the tensile strength and Vickers hardness as output. Subsequently, the optimal model architecture and hyper-parameters of 1DCNN were determined through experiments and grid search combined with K-fold cross-validation, and an online prediction system was developed. Finally, effective validation experiments were conducted and the experimental results agreed with the DS-1DCNN prediction results with an acceptable error of approximately 5%. Parameter sensitivity analysis revealed that the number of convolutional layers and learning rate were identified as critical parameters affecting the model performance. Additionally, time analysis showed that the calculation time of the prediction system was less than 200 ms which can meet the time requirements of real-time thick-plate FSW application. The research lays a foundation on the real-time quality monitoring in the process of FSW thick-plate butt joint.

Spatio-temporal feature learning for enhancing video quality based on screen content characteristics

With the rising demands for remote desktops and online meetings, screen content videos have drawn significant attention. Different from natural videos, screen content videos often exhibit scene switches where the content abruptly changes from one frame to the next. These scene switches result in obvious distortions in compressed videos. Besides, frame freezing, where the content remains unchanged for a certain duration, is also very common in screen content videos. Existing alignment-based models struggle to effectively enhance scene switch frames and lack efficiency when dealing with frame freezing situations. Therefore, we propose a novel alignment-free method that effectively handles both scene switches and frame freezing. In our approach, we develop a spatial and temporal feature extraction module that compresses and extracts spatio-temporal information from three groups of frame inputs. This enables efficient handling of scene switches. In addition, an edge aware block is proposed for extracting edge information, which guides the model to focus on restoring the high-frequency components in frame freezing situations. The fusion module is then designed to adaptively fuse the features from three groups, considering different positions of video frames, to enhance frames during scene switch and frame freezing scenarios. Experimental results demonstrate the significant advancements achieved by the proposed edge aware with spatio-temporal information fusion network (EAST) in enhancing the quality of compressed videos, surpassing the current state-of-the-art methods.

Two-stage fusion framework driven by domain knowledge for penetration prediction of laser welding

To solve the problem of industrial application difficulties caused by insufficient depth exploration and poor interpretability of the pure data drive neural network. A two-stage fusion framework driven by domain knowledge was proposed and focused on industrial applications: laser welding penetration monitoring. First, a multi-sensor acquisition system based on plasma and molten pool signals is developed, extracting multiple features from plasma and molten pool images. Two Back Propagation (BP) neural network information fusion models are established in the first fusion stage, leveraging the characteristics of the plasma plume and molten pool front. In the second fusion stage, the Dempster-Shafer (DS) evidence theory is employed to fuse the information from the two BP neural network models. The proposed method, achieving a high accuracy of 98.7 %, is validated by comparing its average accuracy and anti-interference capability with classical Convolutional Neural Network (CNN) and BP neural network models. The physical relationship between input and output is elucidated through an established physical model, enhancing the interpretability and anti-interference capability of the model, and thereby improving the stability and accuracy of the system.

Multiscale Neighborhood Information Fusion Network for Classification of Remote Sensing LiDAR Images

Multiscale Neighborhood Information Fusion Network for Classification of Remote Sensing LiDAR Images

MFEFNet: A Multi-Scale Feature Information Extraction and Fusion Network for Multi-Scale Object Detection in UAV Aerial Images

Unmanned aerial vehicles (UAVs) are now widely used in many fields. Due to the randomness of UAV flight height and shooting angle, UAV images usually have the following characteristics: many small objects, large changes in object scale, and complex background. Therefore, object detection in UAV aerial images is a very challenging task. To address the challenges posed by these characteristics, this paper proposes a novel UAV image object detection method based on global feature aggregation and context feature extraction named the multi-scale feature information extraction and fusion network (MFEFNet). Specifically, first of all, to extract the feature information of objects more effectively from complex backgrounds, we propose an efficient spatial information extraction (SIEM) module, which combines residual connection to build long-distance feature dependencies and effectively extracts the most useful feature information by building contextual feature relations around objects. Secondly, to improve the feature fusion efficiency and reduce the burden brought by redundant feature fusion networks, we propose a global aggregation progressive feature fusion network (GAFN). This network adopts a three-level adaptive feature fusion method, which can adaptively fuse multi-scale features according to the importance of different feature layers and reduce unnecessary intermediate redundant features by utilizing the adaptive feature fusion module (AFFM). Furthermore, we use the MPDIoU loss function as the bounding-box regression loss function, which not only enhances model robustness to noise but also simplifies the calculation process and improves the final detection efficiency. Finally, the proposed MFEFNet was tested on VisDrone and UAVDT datasets, and the mAP0.5 value increased by 2.7% and 2.2%, respectively.

Adaptive information fusion network for multi‐modal personality recognition

AbstractPersonality recognition is of great significance in deepening the understanding of social relations. While personality recognition methods have made significant strides in recent years, the challenge of heterogeneity between modalities during feature fusion still needs to be solved. This paper introduces an adaptive multi‐modal information fusion network (AMIF‐Net) capable of concurrently processing video, audio, and text data. First, utilizing the AMIF‐Net encoder, we process the extracted audio and video features separately, effectively capturing long‐term data relationships. Then, adding adaptive elements in the fusion network can alleviate the problem of heterogeneity between modes. Lastly, we concatenate audio‐video and text features into a regression network to obtain Big Five personality trait scores. Furthermore, we introduce a novel loss function to address the problem of training inaccuracies, taking advantage of its unique property of exhibiting a peak at the critical mean. Our tests on the ChaLearn First Impressions V2 multi‐modal dataset show partial performance surpassing state‐of‐the‐art networks.

Social domain integrated semantic self-discovery method for recommendation

Recommender systems effectively improve the convenience for users to access interesting information from vast amounts of data resources. However, the issue of data sparsity significantly impacts recommendation performance. Heterogeneous information network (HIN), as a form of heterogeneous graph data representation, remarkably enhances recommendation performance. Despite this, current HIN-based recommendations insufficiently consider the social attributes and multidimensional interaction information of users. Additionally, the problem of expert experience dependence on extracting semantic features leads to limited generalization performance. To address these issues, we propose the social domain integrated semantic self-discovery method for recommendation (SSRec). This method constructs a social domain information fusion network to fuse multidimensional interaction information of users in the social domain and designs a multidimensional semantic knowledge mining method to extract semantic knowledge of different meta-paths. We also propose a semantic relationship self-discovery method of heterogeneous nodes that utilizes deep reinforcement learning to address the limited efficiency of traditional manual meta-paths selection. Simulation results show that the SSRec can effectively improve the recommendation performance compared with baseline methods, achieving high-precision recommendations.

Segmentation assisted Prostate Cancer Grading with Multitask Collaborative Learning

Medical image segmentation can provide doctors with more direct information on the location and size of organs or lesions, which can serve as an valuable auxiliary task for prostate cancer grading. Meanwhile, other types of diagnostic data besides images are also essential, such as patient age, Prostate-Specific Antigen (PSA), etc. Currently, there is a lack of in-depth research on how to effectively differentiate and select shared features and task-specific features in multitask learning, as well as how to balance and explore the potential correlations between different tasks. In this paper, we propose a novel Shared Feature Hybrid Gating Experts (SFHGE) architecture for collaborative main (lesion grading) and auxiliary (lesion segmentation) task learning, dynamically selecting shared and task-specific features. To efficiently utilize complementary features, we also introduce a Cross-Task Attention module (CrossTA) to capture cross-task integrated representation. Additionally, recognizing that non-image clinical information often provides crucial diagnostic insights, we further design a Heterogeneous Information Fusion Network (HIFN) to better integrate clinical data, thereby improving grading performance. Extensive experiments on the PI-CAI dataset demonstrate that our approach outperforms mainstream classification and segmentation models.

Attention-based convolutional neural network with multi-modal temporal information fusion for motor imagery EEG decoding

Convolutional neural network (CNN) has been widely applied in motor imagery (MI)-based brain computer interface (BCI) to decode electroencephalography (EEG) signals. However, due to the limited perceptual field of convolutional kernel, CNN only extracts features from local region without considering long-term dependencies for EEG decoding. Apart from long-term dependencies, multi-modal temporal information is equally important for EEG decoding because it can offer a more comprehensive understanding of the temporal dynamics of neural processes. In this paper, we propose a novel deep learning network that combines CNN with self-attention mechanism to encapsulate multi-modal temporal information and global dependencies. The network first extracts multi-modal temporal information from two distinct perspectives: average and variance. A shared self-attention module is then designed to capture global dependencies along these two feature dimensions. We further design a convolutional encoder to explore the relationship between average-pooled and variance-pooled features and fuse them into more discriminative features. Moreover, a data augmentation method called signal segmentation and recombination is proposed to improve the generalization capability of the proposed network. The experimental results on the BCI Competition IV-2a (BCIC-IV-2a) and BCI Competition IV-2b (BCIC-IV-2b) datasets show that our proposed method outperforms the state-of-the-art methods and achieves 4-class average accuracy of 85.03% on the BCIC-IV-2a dataset. The proposed method implies the effectiveness of multi-modal temporal information fusion in attention-based deep learning networks and provides a new perspective for MI-EEG decoding. The code is available at https://github.com/Ma-Xinzhi/EEG-TransNet.

DC-BVM: Dual-channel information fusion network based on voting mechanism

Emotion recognition in conversations (ERC) has been challenging due to the dynamics and complexity of emotions in conversations. Most current emotion recognition studies have focused on modeling temporal dimensions, such as context-sensitive dependencies while ignoring the spatial dimensional relationships between discourses. In this paper, we propose a dual-channel information fusion network based on a voting mechanism, in which we model modal information in two dimensions: distance dependence is constructed in the spatial dimension, and context dependence is constructed in the temporal dimension. We aim to extract more comprehensive and accurate sentiment features to recognize verbal emotions from limited data. In addition, we discard the traditional modal fusion method and propose a fusion strategy based on the voting mechanism, which significantly accelerates the model convergence speed and recognition performance. We conducted experiments on two benchmark datasets, IEMOCAP and MELD, respectively, and the experimental results demonstrate the superiority of the designed network in emotion recognition.

A multi-source domain information fusion network for rotating machinery fault diagnosis under variable operating conditions

In practical industrial scenarios, the variations of operating conditions such as load and rotational speed make mechanical systems subject to complex and variable environmental stresses, resulting in the distribution discrepancies of sample data. With the advantages of integrating the feature information and diagnosis knowledge, the transfer learning technique based on multiple source domains has become a stable and efficient solution to address the fault diagnosis challenge under variable operating conditions in the modern intelligent operation and maintenance. For the above discussions, a multi-source domain information fusion network (MDIFN) is proposed in this paper to obtain generalized knowledge with abundant feature information by combining the adversarial transfer learning technique with fine-grained information fusion of multiple source domains. First, an adversarial transfer network architecture is constructed in accordance with the complex feature transformation and the boundary equilibrium domain discrimination to implement feature learning and knowledge transfer of source and target domains. Then, a joint distribution domain adaptation mechanism is proposed to further facilitate the acquisition of domain invariant features. Finally, a class-related decision fusion (CDF) strategy is designed to realize the information fusion within the decision space. The fault diagnosis of rotating machinery under unknown operating conditions can be achieved by employing data under known multiple operating conditions for MDIFN training. The public Paderborn University (PU) bearing dataset and the actual mechanical comprehensive diagnosis simulation platform (MCDSP) bearing dataset from different testing rigs are considered to evaluate the cross-domain fault diagnosis performance of this method. The experimental results indicate that the method achieves an average accuracy of 95.97% on the PU dataset and 98.31% on the MCDSP dataset, which is superior to other state-of-the-art cross-domain fault diagnosis algorithms.

Deep convolutional neural networks information fusion and improved whale optimization algorithm based smart oral squamous cell carcinoma classification framework using histopathological images

AbstractThe most prevalent type of cancer worldwide is mouth cancer. Around 2.5% of deaths are reported annually due to oral cancer in 2023. Early diagnosis of oral squamous cell carcinoma (OSCC), a prevalent oral cavity cancer, is essential for treating and recovering patients. A few computerized techniques exist but are focused on traditional machine learning methods, such as handcrafted features. In this work, we proposed a fully automated architecture based on Self‐Attention convolutional neural network and Residual Network information fusion and optimization. In the proposed framework, the augmentation process is performed on the training and testing samples, and then two developed deep models are trained. A self‐attention MobileNet‐V2 model is developed and trained using an augmented dataset. In parallel, a Self‐Attention DarkNet‐19 model is trained on the same dataset, whereas the hyperparameters have been initialized using the whale optimization algorithm (WOA). Features are extracted from the deeper layers of both models and fused using a canonical correlation analysis (CCA) approach. The CCA approach is further optimized using an improved WOA version named Quantum WOA that removes the irrelevant features and selects only important ones. The final selected features are classified using neural networks such as wide neural networks. The experimental process is performed on the augmented dataset that includes two sets: 100× and 400×. Using both sets, the proposed method obtained an accuracy of 98.7% and 96.3%. Comparison is conducted with a few state‐of‐the‐art (SOTA) techniques and shows a significant improvement in accuracy and precision rate.

Efficient correlation information mixer for visual object tracking

Recently, Siamese tracking framework has been widely used in the visual tracking community. Siamese trackers usually use Cross-Correlation to aggregate templates and search information, so they realize the encoding of target information. However, the previous Cross-Correlation methods either ignore the object channel semantic information or ignore the object’s local information. This seriously limits the representation of embedded correlation features and reduces the performance of the trackers. In this paper, to solve these problems, we propose an effective correlation information mixer for visual target tracking. We design an information fusion network to efficiently aggregate template features and search features. In the information fusion network, we use Depthwise Cross-Correlation and Pointwise Cross-Correlation to extract the channel semantic information and local information of the object respectively, and use the correlation information mixer to fully fuse the two correlation maps to achieve the optimal target information encoding. Extensive experimental results show that our tracker achieves competitive performance compared with other state-of-the-art trackers on four benchmarks, including OTB, VOT, UAV123, and LaSOT.

Enhancing Stock Price Prediction with Deep Cross-Modal Information Fusion Network

Stock price prediction is considered a classic and challenging task, with the potential to aid traders in making more profitable trading decisions. Significant improvements in stock price prediction methods based on deep learning have been observed in recent years. However, most existing methods are reliant solely on historical stock price data for predictions, resulting in the inability to capture market dynamics beyond price indicators, thus limiting their performance to some extent. Therefore, combining social media text with historical stock price information has proposed a novel stock price prediction method, known as the Deep Cross-Modal Information Fusion Network (DCIFNet). The process is initiated by DCIFNet, which employs temporal convolution processes to encode stock prices and Twitter content. This ensures that each element has sufficient information about its surrounding components. Following this, the outcomes are inputted into a cross-modal fusion structure based on transformers to enhance the integration of crucial information from stock prices and Twitter content. Lastly, a multi-graph convolution attention network is introduced to depict the relationships between different stocks from diverse perspectives. This facilitates the more effective capturing of industry affiliations, Wikipedia references, and associated relationships among linked stocks, ultimately leading to an enhancement in stock price prediction accuracy. Trend prediction and simulated trading experiments are conducted on high-frequency trading datasets spanning nine different industries. Comparative assessments with the Multi-Attention Network for Stock Prediction (MANGSF) method, as well as ablation experiments, confirm the effectiveness of the DCIFNet approach, resulting in an accuracy rate of 0.6309, a marked improvement compared to representative methods in the field.

CIFG-Net: Cross-level information fusion and guidance network for Polyp Segmentation

Colorectal cancer is a common malignant tumor of the digestive tract. Most colorectal cancer is caused by colorectal polyp lesions. Timely detection and removal of colorectal polyps can substantially reduce the incidence of colorectal cancer. Accurate polyp segmentation can provide important polyp information that can aid in the early diagnosis and treatment of colorectal cancer. However, polyps of the same type can vary in texture, color, and even size. Furthermore, some polyps are similar in colour to the surrounding healthy tissue, which makes the boundary between the polyp and the surrounding area unclear. In order to overcome the issues of inaccurate polyp localization and unclear boundary segmentation, we propose a polyp segmentation network based on cross-level information fusion and guidance. We use a Transformer encoder to extract a more robust feature representation. In addition, to refine the processing of feature information from encoders, we propose the edge feature processing module (EFPM) and the cross-level information processing module (CIPM). EFPM is used to focus on the boundary information in polyp features. After processing each feature, EFPM can obtain clear and accurate polyp boundary features, which can mitigate unclear boundary segmentation. CIPM is used to aggregate and process multi-scale features transmitted by various encoder layers and to solve the problem of inaccurate polyp location by using multi-level features to obtain the location information of polyps. In order to better use the processed features to optimise our segmentation effect, we also propose an information guidance module (IGM) to integrate the processed features of EFPM and CIPM to obtain accurate positioning and segmentation of polyps. Through experiments on five public polyp datasets using six metrics, it was demonstrated that the proposed network has better robustness and more accurate segmentation effect. Compared with other advanced algorithms, CIFG-Net has superior performance. Code available at: https://github.com/zspnb/CIFG-Net.

RETRACTION: Construction of the Rule of Law System of Marine Ecological Environment Protection under the Background of Wireless Network Information Fusion

RETRACTION: Construction of the Rule of Law System of Marine Ecological Environment Protection under the Background of Wireless Network Information Fusion

GMM Enabled by Multimodal Information Fusion Network for Detection and Motion Planning of Robotic Liquid Pouring.

When humans perform pouring tasks, they exhibit consistent accuracy, regardless of the liquid type, container, or environmental conditions. This proficiency stems from their ability to effectively utilize both vision and hearing while also considering various factors. However, in the domain of robotic liquid pouring, the combination of multimodal information is effectively rarely leveraged to accomplish automatic control of robotic liquid pouring. To address this limitation, a multimodal information fusion network (MMFNet) is designed for estimating liquid height and pouring state. The MMFNet employs cross-attention networks and motion features to enhance visual features (VFs). Subsequently, multimodal transformers are utilized to fuse audio features with the enhanced VFs, enabling the MMFNet to estimate both liquid height and pouring state accurately. Finally, the detection results are combined with demonstration learning to make robots learn pouring motion trajectory encoded by the Gaussian mixture model (GMM). The experimental results demonstrate the effectiveness of MMFNet in significantly improving the detection accuracy of liquid height and pouring state. Furthermore, by employing the GMM enabled by MMFNet, robots can acquire robust pouring motion planning, enhancing their capabilities in performing pouring tasks.