Fusion Method Research Articles

SIFusion: Lightweight infrared and visible image fusion based on semantic injection.

The objective of image fusion is to integrate complementary features from source images to better cater to the needs of human and machine vision. However, existing image fusion algorithms predominantly focus on enhancing the visual appeal of the fused image for human perception, often neglecting their impact on subsequent high-level visual tasks, particularly the processing of semantic information. Moreover, these fusion methods that incorporate downstream tasks tend to be overly complex and computationally intensive, which is not conducive to practical applications. To address these issues, a lightweight infrared and visible light image fusion method known as SIFusion, which is based on semantic injection, is proposed in this paper. This method employs a semantic-aware branch to extract semantic feature information, and then integrates these features into the fused features through a Semantic Injection Module (SIM) to meet the semantic requirements of high-level visual tasks. Furthermore, to simplify the complexity of the fusion network, this method introduces an Edge Convolution Module (ECB) based on structural reparameterization technology to enhance the representational capacity of the encoder and decoder. Extensive experimental comparisons demonstrate that the proposed method performs excellently in terms of visual appeal and advanced semantics, providing satisfactory fusion results for subsequent high-level visual tasks even in challenging scenarios.

UPGCN: User Perception-Guided Graph Convolutional Network for Multimodal Recommendation

To tackle the challenges of cold start and data sparsity in recommendation systems, an increasing number of researchers are integrating item features, resulting in the emergence of multimodal recommendation systems. Although graph convolutional network-based approaches have achieved significant success, they still face two limitations: (1) Users have different preferences for various types of features, but existing methods often treat these preferences equally or fail to specifically address this issue. (2) They do not effectively distinguish the similarity between different modality item features, overlook the unique characteristics of each type, and fail to fully exploit their complementarity. To solve these issues, we propose the user perception-guided graph convolutional network for multimodal recommendation (UPGCN). This model consists of two main parts: the user perception-guided representation enhancement module (UPEM) and the multimodal two-step enhanced fusion method, which are designed to capture user preferences for different modalities to enhance user representation. At the same time, by distinguishing the similarity between different modalities, the model filters out noise and fully leverages their complementarity to achieve more accurate item representations. We performed comprehensive experiments on the proposed model, and the results indicate that it outperforms other baseline models in recommendation performance, strongly demonstrating its effectiveness.

Day2Dark: Pseudo-Supervised Activity Recognition Beyond Silent Daylight

AbstractThis paper strives to recognize activities in the dark, as well as in the day. We first establish that state-of-the-art activity recognizers are effective during the day, but not trustworthy in the dark. The main causes are the limited availability of labeled dark videos to learn from, as well as the distribution shift towards the lower color contrast at test-time. To compensate for the lack of labeled dark videos, we introduce a pseudo-supervised learning scheme, which utilizes easy to obtain unlabeled and task-irrelevant dark videos to improve an activity recognizer in low light. As the lower color contrast results in visual information loss, we further propose to incorporate the complementary activity information within audio, which is invariant to illumination. Since the usefulness of audio and visual features differs depending on the amount of illumination, we introduce our ‘darkness-adaptive’ audio-visual recognizer. Experiments on EPIC-Kitchens, Kinetics-Sound, and Charades demonstrate our proposals are superior to image enhancement, domain adaptation and alternative audio-visual fusion methods, and can even improve robustness to local darkness caused by occlusions. Project page: https://xiaobai1217.github.io/Day2Dark/.

YOLOX-S-TKECB: A Holstein Cow Identification Detection Algorithm

Accurate identification of individual cow identity is a prerequisite for the construction of digital farms and serves as the basis for optimized feeding, disease prevention and control, breed improvement, and product quality traceability. Currently, cow identification faces challenges such as poor recognition accuracy, large data volumes, weak model generalization ability, and low recognition speed. Therefore, this paper proposes a cow identification method based on YOLOX-S-TKECB. (1) Based on the characteristics of Holstein cows and their breeding practices, we constructed a real-time acquisition and preprocessing platform for two-dimensional Holstein cow images and built a cow identification model based on YOLOX-S-TKECB. (2) Transfer learning was introduced to improve the convergence speed and generalization ability of the cow identification model. (3) The CBAM attention mechanism module was added to enhance the model’s ability to extract features from cow torso patterns. (4) The alignment between the apriori frame and the target size was improved by optimizing the clustering algorithm and the multi-scale feature fusion method, thereby enhancing the performance of object detection at different scales. The experimental results demonstrate that, compared to the traditional YOLOX-S model, the improved model exhibits a 15.31% increase in mean average precision (mAP) and a 32-frame boost in frames per second (FPS). This validates the feasibility and effectiveness of the proposed YOLOX-S-TKECB-based cow identification algorithm, providing valuable technical support for the application of dairy cow identification in farms.

Investigation of hybrid modeling and its transferability in building load prediction used for district heating systems

In the district heating systems, the historical operation data of the buildings in those areas would be partially or entirely missing. The traditional data-driven model is hard to predict the ground truth results because the historical data is not available for model training. However, utilizing the physics-based methods for load calculation takes a long time to process and encounters low accuracy issues. This paper investigates several hybrid models that integrate the data-driven model and the physics-based models with different fusion methods. The physics-based models calculate envelope load and infiltration load, based on Fourier's law and the grand canonical ensemble theory, respectively. After undergoing load processing, features fusion, and residual connection, the best advanced hybrid models generate 21.35%, 16.35%, and 12.73% better prediction results compared with the data-driven model. Moreover, the advanced hybride models also perform strong transferability across all the data quantity groups. In terms of practical application, the advanced hybrid models could be deployed with effective generalization in limited data scenarios and robust transfer capabilities. The selected best model constructed by hybrid modeling displays the highest performance and saves the total training costs with strong transferability.

Cattle identification based on multiple feature decision layer fusion

In farming scenarios, cattle identification has become a key issue for the development of precision farming. In precision livestock farming, single-feature recognition methods are prone to misjudgment in complex scenarios involving multiple cattle obscuring each other during drinking and feeding. This paper proposes a decision-level identification method based on the multi-feature fusion of cattle faces, muzzle patterns, and ear tags. The method utilizes the SOLO algorithm to segment images and employs the FaceNet and PP-OCRv4 networks to extract features for the cattle’s faces, muzzle patterns, and ear tags. These features are compared with the Ground truth, from which the Top 3 features are extracted. The corresponding cattle IDs of these features are then processed using One-Hot encoding to serve as the final input for the decision layer, and various ensemble strategies are used to optimize the model. The results show that using the multimodal decision fusion method makes the recognition accuracy reach 95.74%, 1.4% higher than the traditional optimal unimodal recognition accuracy. The verification rate reaches 94.72%, 10.65% higher than the traditional optimal unimodal recognition verification rate. The research results demonstrate that the multi-feature fusion recognition method has significant advantages in drinking and feeding farm environments, providing an efficient and reliable solution for precise identification and management of cattle in farms and significantly improving recognition accuracy and stability.

Image Dehazing Enhancement Strategy Based on Polarization Detection of Space Targets

In view of the fact that the technology of polarization detection performs better at identifying targets through clouds and fog, the recognition ability of the space target detection system under haze conditions will be improved by applying the technology. However, due to the low ambient brightness and limited target radiation information during space target detection, the polarization information of space target is seriously lost, and the advantages of polarization detection technology in identifying targets through clouds and fog cannot be effectively exerted under the condition of haze detection. In order to solve the above problem, a dehazing enhancement strategy specifically applied to polarization images of space targets is proposed. Firstly, a hybrid multi-channel interpolation method based on regional correlation analysis is proposed to improve the calculation accuracy of polarization information during preprocessing. Secondly, an image processing method based on full polarization information inversion is proposed to obtain the degree of polarization of the image after inversion and the intensity of the image after dehazing. Finally, the image fusion method based on discrete cosine transform is used to obtain the dehazing polarization fusion enhancement image. The effectiveness of the proposed image processing strategy is verified by carrying out simulated and real space target detection experiments. Compared with other methods, by using the proposed image processing strategy, the quality of the polarization images of space targets obtained under the haze condition is significantly improved. Our research results have important practical implications for promoting the wide application of polarization detection technology in the field of space target detection.

Weighted Fusion Method of Marine Gravity Field Model Based on Water Depth Segmentation

Among the marine gravity field models derived from satellite altimetry, the Scripps Institution of Oceanography (SIO) series and Denmark Technical University (DTU) series models are the most representative and are often used to integrate global gravity field models, which were inverted by the deflection of vertical method and sea surface height method, respectively. The fusion method based on the offshore distance used in the EGM2008 model is just model stitching, which cannot realize the true fusion of the two types of marine gravity field models. In the paper, a new fusion method based on water depth segmentation is proposed, which established the Precision–Depth relationship of each model in each water depth segment in the investigated area, then constructed the FUSION model by weighted fusion based on the precision predicted from the Precision–Depth relationship at each grid in the whole region. The application in the South China Sea shows that the FUSION model built by the new fusion method has better accuracy than SIO28 and DTU17, especially in shallow water and offshore areas. Within 20 km offshore, the RMS of the FUSION model is 5.10 mGal, which is 8% and 4% better than original models, respectively. Within 100 m of shallow water, the accuracy of the FUSION model is 4.01 mGal, which is 14% and 12% higher than the original models, respectively. A further analysis shows that the fusion model is in better agreement with the seabed topography than original models. The new fusion method can blend the effective information of original models to provide a higher-precision marine gravity field.

Local element segregation-induced cellular structures and dominant dislocation planar slip enable exceptional strength-ductility synergy in an additively-manufactured CoNiV multicomponent alloy with ageing treatment

We proposed an additively manufactured equiatomic CoNiV multicomponent alloy (MCA) using a conventional laser powder bed fusion (LPBF) method, and an exceptional strength-ductility synergy of the alloy was attained through a simple post-ageing treatment. Pronounced hierarchical microstructures were achieved in our printed alloys, including heterogeneous grain structures, and intragranular cellular structures composed of interior domain with limited dislocations and cell walls led by significant vanadium local segregation. Besides the outstanding mechanical properties at room temperature of 298 K, a giga-pascal yielding strength (> 1.1 GP) and over 40% uniform elongation were attained in the aged specimen deformed at a cryogenic temperature of 77 K, predominating the mechanical properties of many alloys reported in previous works. Such exceptional performance of the aged alloy can be mainly ascribed to considerable local chemical orders (LCOs), aggravated elemental fluctuation in the alloy matrix, and intensified vanadium segregation at walls of intragranular cellular structures which can strongly interact with dislocations. As a result, a planar slip array of dislocations with an extremely high density, namely large numbers of slip bands that can sustain and transfer high strains, dominates the deformation microstructures, thus efficiently strengthening and toughening the aged alloy, especially at a low temperature like 77 K. The above post-ageing strategy is readily and low-costly employed on additively manufactured MCAs with relatively high stacking fault energy (SFE) and proved as a feasible method to produce high-performance structural materials for extreme conditions.

Multistream Adaptive Attention-Enhanced Graph Convolutional Networks for Youth Fencing Footwork Training.

The popularity of fencing and intense sports competition has burdened adolescents with excessive training, harming their immature bodies. Traditional training methods fail to provide timely movement corrections and personalized plans, leading to ineffective exercises. This paper aims to use artificial intelligence technology to reduce ineffective exercises and alleviate the training burden. We propose an action recognition algorithm based on the characteristics of adolescent athletes. This algorithm uses multimodal input data to comprehensively extract action information. Each modality is processed by the same network structure, utilizing attention mechanisms and adaptive graph structures. A multibranch feature fusion method is used to determine the final action category. We gathered the fencing footwork data set 2.0. Our model achieved 93.3% accuracy, with the highest precision at 95.8% and the highest F1-Score at 94.5% across all categories. It effectively recognized actions of adolescents with different heights and speeds, outperforming traditional methods. Our artificial intelligence-based training solution improves training efficiency and reduces the training burden on adolescents.

TIWMFLP: Two-Tier Interactive Weighted Matrix Factorization and Label Propagation Based on Similarity Matrix Fusion for Drug-Disease Association Prediction.

Accurately identifying new therapeutic uses for drugs is crucial for advancing pharmaceutical research and development. Matrix factorization is often used in association prediction due to its simplicity and high interpretability. However, existing matrix factorization models do not enable real-time interaction between molecular feature matrices and similarity matrices, nor do they consider the geometric structure of the matrices. Additionally, efficiently integrating multisource data remains a significant challenge. To address these issues, we propose a two-tier interactive weighted matrix factorization and label propagation model based on similarity matrix fusion (TIWMFLP) to assist in personalized treatment. First, we calculate the Gaussian and Laplace kernel similarities for drugs and diseases using known drug-disease associations. We then introduce a new multisource similarity fusion method, called similarity matrix fusion (SMF), to integrate these drug/disease similarities. SMF not only considers the different contributions represented by each neighbor but also incorporates drug-disease association information to enhance the contextual topological relationships and potential features of each drug/disease node in the network. Second, we innovatively developed a two-tier interactive weighted matrix factorization (TIWMF) method to process three biological networks. This method realizes for the first time the real-time interaction between the drug/disease feature matrix and its similarity matrix, allowing for a better capture of the complex relationships between drugs and diseases. Additionally, the weighted matrix of the drug/disease similarity matrix is introduced to preserve the underlying structure of the similarity matrix. Finally, the label propagation algorithm makes predictions based on the three updated biological networks. Experimental outcomes reveal that TIWMFLP consistently surpasses state-of-the-art models on four drug-disease data sets, two small molecule-miRNA data sets, and one miRNA-disease data set.

Echocardiographic mitral valve segmentation model

Segmentation of mitral valve is not only important for clinical diagnosis, but also has far-reaching impact on prevention and prognosis of the disease by experts and doctors.In this paper, the multi-channel cross fusion transformer based U-Net network model (MCCT-UNet) is proposed according to the classical U-Net architecture. First, the jump connection part of MCCT-UNet is designed by using a multi-channel cross-fusion based attention mechanism module (MCCT) instead of the original jump connection, and this module fuses the feature maps from different scales in different stages of the encoder. Second, the optimization of the feature fusion method is proposed in the decoding stage by designing the cross-compression excitation sub-module (C-SENet) to replace the simple feature splicing, and the C-SENet is used to bridge the inconsistency of the semantic hierarchy by effectively combining the deeper information in the encoding stage with the shallower information. This two modules can establish a close connection between the encoder and decoder by exploring multi-scale global contextual information to solve the semantic divide problem, thus it significantly improves the segmentation performance of the network. The experimental results show that the improvement is effective, and the MCCT-UNet model outperforms the other 9 network models. Specifically, the MCCT-UNet achieved a Dice coefficient of 0.8734, an IoU of 0.7854, and an accuracy of 0.9977, demonstrating significant improvements over the compared models.

A novel adjunctive diagnostic method for bone cancer: Osteosarcoma cell segmentation based on Twin Swin Transformer with multi-scale feature fusion

BackgroundOsteosarcoma, the most common primary bone tumor originating from osteoblasts, poses a significant challenge in medical practice, particularly among adolescents. Conventional diagnostic methods heavily rely on manual analysis of magnetic resonance imaging (MRI) scans, which often fall short in providing accurate and timely diagnosis. This underscores the critical need for advancements in medical imaging technologies to improve the detection and characterization of osteosarcoma. MethodsIn this study, we sought to address the limitations of current diagnostic approaches by leveraging Hoechst-stained images of osteosarcoma cells obtained via fluorescence microscopy. Our primary objective was to enhance the segmentation of osteosarcoma cells, a crucial step in precise diagnosis and treatment planning. Recognizing the shortcomings of existing feature extraction networks in capturing detailed cellular structures, we propose a novel approach utilizing a twin swin transformer architecture for osteosarcoma cell segmentation, with a focus on multi-scale feature fusion. ResultsThe experimental findings demonstrate the effectiveness of the proposed Twin Swin Transformer with multi-scale feature fusion in significantly improving osteosarcoma cell segmentation. Compared to conventional techniques, our method achieves superior segmentation performance, highlighting its potential utility in clinical settings. ConclusionThe development of our Twin Swin Transformer with multi-scale feature fusion method represents a significant advancement in medical imaging technology, particularly in the field of osteosarcoma diagnosis. By harnessing advanced computational techniques and leveraging high-resolution imaging data, our approach offers enhanced accuracy and efficiency in osteosarcoma cell segmentation, ultimately facilitating better patient care and clinical decision-making.

Review on VSLAM based on deep learning

Abstract. Visual simultaneous localization and mapping technology (VSLAM) provides a theoretical basis for the operation of unmanned equipment such as autonomous vehicles and sweeping robots in unfamiliar environments. Although traditional VSLAM systems have achieved great success after long-term development, it is still difficult to maintain good performance in challenging environments. Deep learning, as a newly developed technology in the field of vision in recent years, has shown outstanding advantages in image processing. Combining deep learning with VSLAM is a hot topic. Deep learning can help traditional VSLAM systems improve the lack of scale information in dynamic environments by improving the performance of traditional VSLAM in depth estimation, pose estimation, and closed loop detection. It can not only reduce the scale of the network model but also improve the accuracy of trajectory estimation. Specifically, in terms of the fusion of VSLAM method flow and deep learning, many researchers have proposed deep learning fusion methods based on visual odometry, loop detection and mapping. This work studies the trend and combination of VSLAM with deep learning algorithms, hoping to provide help for the real autonomy of future mobile robots, and finally puts forward prospects for the development of VSLAM.

Estimation of Canopy Water Content by Integrating Hyperspectral and Thermal Imagery in Winter Wheat Fields

Rapid and accurate estimation of canopy water content (CWC) is important for agricultural water management and food security. Due to the complexity of dynamic changes in water transport during plant growth, estimation of CWC using a single sensor often leads to high uncertainty in the results. Multi-sensor data fusion is one of the solutions to this problem, but suitable spectral preprocessing methods and data fusion methods still need further research. The objectives of this study were to characterize the performance of two varieties at different growth stages under five water stress conditions and screen hyperspectral sensitive spectral bands by using continuous wavelet transform (CWT) and a successive projection algorithm (SPA). Ultimately, the CWC prediction model of winter wheat hyperspectral characteristic bands and thermal imaging information fusion was created using the GRA algorithm. The results showed that canopy temperature parameters and spectral parameters responded significantly to water deficits in winter wheat. Using the CWT-SPA method, a total of 285 hyperspectral feature bands with wavelet decomposition scales ranging from one to eight were selected. The sensitive bands were mainly distributed in the following ranges: 545–561, 746–1348, 1561–1810, and 2122–2430 nm. The GRA algorithm has good multi-source data model fusion capability, and its constructed prediction model based on hyperspectral and thermal image fusion has high accuracy on the canopy water content in winter wheat (R2 = 0.930, RMSE = 5.44%, nRMSE = 7.94%). Compared to the single-feature spectral model (R2 = 0.864, RMSE = 5.92%, nRMSE = 8.63%) and thermal image CWC prediction model (R2 = 0.813, RMSE = 7.22%, nRMSE = 10.49%), the model prediction accuracy based on the GRA algorithm is increased by 7.64% and 13.69%, respectively.

High-accuracy bathymetric method fusing ICESAT-2 datasets and the two-media photogrammetry model

Improving the accuracy of nearshore bathymetric measurements is essential for understanding coastal environments, resource management, and navigation. The Ice, Cloud, and Land Elevation Satellite-2 (ICESat-2) is the first laser satellite that uses the photon-counting technique. The ICESat-2 is equipped with the Advanced Topographic Laser Altimeter System (ATLAS), which enables higher-accuracy measurements of water, ice, and land elevation on Earth. Two-media photogrammetric bathymetry is a type of nearshore bathymetric technology that uses the geometrical characteristics of light rays. With this technique, the accuracy and reliability mainly depend on eliminating systematic errors and ensuring accurate spatial photogrammetric positioning relative to the object being measured. To improve the bathymetric accuracy of two-media photogrammetry, we integrated high-accuracy elevation data from photon datasets as constraining and control parameters. The improved method effectively eliminated systematic errors in two-media photogrammetry during the established joint-block adjustment model. To improve its accuracy and reliability, we employed multispectral WorldView-2 stereo images in our experiments. Furthermore, the bathymetric results were validated and assessed using in situ and photon data. The experimental results show that the highest accuracy achieved with the bathymetric measurements in our study area was a root mean square error (RMSE) of 0.96 m and a mean absolute error of 0.57 m. Using the proposed fusion method, the bathymetric accuracy (as measured using the RMSE) was 1 m higher than that of two-media photogrammetry without the photon datasets.

Automatic Method for Detecting Deformation Cracks in Landslides Based on Multidimensional Information Fusion

As cracks are a precursor landslide deformation feature, they can provide forecasting information that is useful for the early identification of landslides and determining motion instability characteristics. However, it is difficult to solve the size effect and noise-filtering problems associated with the currently available automatic crack detection methods under complex conditions using single remote sensing data sources. This article uses multidimensional target scene images obtained by UAV photogrammetry as the data source. Firstly, under the premise of fully considering the multidimensional image characteristics of different crack types, this article accomplishes the initial identification of landslide cracks by using six algorithm models with indicators including the roughness, slope, eigenvalue rate of the point cloud and pixel gradient, gray value, and RGB value of the images. Secondly, the initial extraction results are processed through a morphological repair task using three filtering algorithms (calculating the crack orientation, length, and frequency) to address background noise. Finally, this article proposes a multi-dimensional information fusion method, the Bayesian probability of minimum risk methods, to fuse the identification results derived from different models at the decision level. The results show that the six tested algorithm models can be used to effectively extract landslide cracks, providing Area Under the Curve (AUC) values between 0.6 and 0.85. After the repairing and filtering steps, the proposed method removes complex noise and minimizes the loss of real cracks, thus increasing the accuracy of each model by 7.5–55.3%. Multidimensional data fusion methods solve issues associated with the spatial scale effect during crack identification, and the F-score of the fusion model is 0.901.

Microstructure and Mechanical Properties of Laser Powder Bed Fusion 3D-Printed Cu-10Sn Alloy

This study investigated the optimal process conditions and mechanical properties of Cu-10Sn alloys produced by the powder bed fusion (PBF) method. The optimal PBF conditions were explored by producing samples with various laser scanning speeds and laser power. It was found that under optimized conditions, samples with a density close to the theoretical density could be fabricated using PBF without any serious defects. The microstructure and mechanical properties of samples produced under optimized conditions were investigated and compared with a commercial alloy produced by the conventional method. The hardness, maximum tensile strength, and elongation of the samples were significantly higher than those of the commercially available cast alloy with the same chemical composition. Based on these results, it is expected to be possible to use the PBF technique to manufacture Cu-10Sn products with complex 3D shapes that could not be made using the conventional manufacturing method.

Wide-field scanning ghost imaging based on a local binary pattern and untrained neural network

Continuous scene imaging is an important research goal in the field of autonomous driving, and the key is to ensure the imaging quality and efficiency. In this paper, we propose a method for information fusion in wide-field scanning ghost imaging using a local binary pattern (LBP) based on deep learning. The initial physical model formed by the LBP integrated into a deep neural network, which effectively enhances the expression of image texture details. Then the collected bucket signals are used as labels for adaptive image reconstruction, enabling the acquisition of images at each scanning position without the need for training on any dataset. Moreover, by employing weighted fusion to combine the image data from each scanning position, which effectively eliminates gaps that arise from direct stitching. Both simulation and experimental results demonstrate that our approach is capable of achieving high-quality detailed imaging with fewer measurements. Additionally, we analyze the impact of the projection beam step length, finding that our method yields significantly better imaging quality with larger steps compared to other methods using smaller steps. Our research also has the application prospect in medical detection, remote sensing and other fields.

Short-term hybrid prognostics of fuel cells: A comparative and improvement study

Accurately predicting the performance degradation trend of fuel cells helps take measures in advance and prolong the stack's service life, leading to a novel hybrid forecasting approach. The first grey model prediction method based on residual exponential smoothing optimization (ES-R-GM) can capture the voltage deterioration trend. We explore the integration of two different ES techniques, specifically the double ES (ES2) and cubic ES (ES3), to investigate their effect on enhancing the predictive accuracy of the GM model. The second method of the adaptive network fuzzy inference system (ANFIS) can characterize local nonlinear behavior. We utilize the simulated annealing (SA) algorithm to optimize ANFIS results under different fuzzy rule selection strategies. The outcomes of the two prediction methods mentioned above are combined to create a hybrid prediction using the data fusion method and the moving window technique. Various hybrid methods are evaluated under general conditions and further detailed optimization. The data collected from the experimental platform confirms the suggested hybrid framework. The results show that the hybrid ES3-R-GM + ANFIS-SC method outperformed the single models in final prediction accuracy and can effectively track both global trends and local changes. Simultaneously, it takes less time to calculate than the literature. Moreover, when applied to consistent public datasets, the hybrid approach maintains its robustness and accuracy compared with other hybrid prognostic methods.