Image Information Fusion Research Articles

An image information fusion based simple diffusion network leveraging the segment anything model for guided attention on thermal images producing colorized pedestrian masks

Because of the synergistic characteristics of the data they provide, thermal imagers are becoming more and more common as secondary data-collecting modules to complement classical optical imagers. Their application is especially beneficial when ambient elements like light, smoke, or other particulates need to be handled. As a result, fusion-based detection technology, like assistive driving systems, is trying to incorporate thermal imagers. In this work, we demonstrate that it is possible to create a mask for pedestrians in thermal imaging by building a basic Denoising Diffusion Probabilistic Model and using the results from the Segment Anything Model. Moreover, we show how the anticipated mask can fuse RGB and thermal pictures into a 3-channel output. Finally, we demonstrate how the loss function optimizes the end-to-end analysis in this field. Results from both the quantitative and qualitative analyses validate our methodology.

Bidirectional Feature Fusion and Enhanced Alignment based Multimodal Semantic Segmentation for Remote Sensing Images

Image–text multimodal deep semantic segmentation leverages the fusion and alignment of image and text information and provides more prior knowledge for segmentation tasks. It is worth exploring image–text multimodal semantic segmentation for remote sensing images. In this paper, we propose a bidirectional feature fusion and enhanced alignment-based multimodal semantic segmentation model (BEMSeg) for remote sensing images. Specifically, BEMSeg first extracts image and text features by image and text encoders, respectively, and then the features are provided for fusion and alignment to obtain complementary multimodal feature representation. Secondly, a bidirectional feature fusion module is proposed, which employs self-attention and cross-attention to adaptively fuse image and text features of different modalities, thus reducing the differences between multimodal features. For multimodal feature alignment, the similarity between the image pixel features and text features is computed to obtain a pixel–text score map. Thirdly, we propose a category-based pixel-level contrastive learning on the score map to reduce the differences among the same category’s pixels and increase the differences among the different categories’ pixels, thereby enhancing the alignment effect. Additionally, a positive and negative sample selection strategy based on different images is explored during contrastive learning. Averaging pixel values across different training images for each category to set positive and negative samples compares global pixel information while also limiting sample quantity and reducing computational costs. Finally, the fused image features and aligned pixel–text score map are concatenated and fed into the decoder to predict the segmentation results. Experimental results on the ISPRS Potsdam, Vaihingen, and LoveDA datasets demonstrate that BEMSeg is superior to comparison methods on the Potsdam and Vaihingen datasets, with improvements in mIoU ranging from 0.57% to 5.59% and 0.48% to 6.15%, and compared with Transformer-based methods, BEMSeg also performs competitively on LoveDA dataset with improvements in mIoU ranging from 0.37% to 7.14%.

Bearing Fault Diagnosis Based on Image Information Fusion and Vision Transformer Transfer Learning Model

In order to improve the accuracy of bearing fault diagnosis under a small sample, variable load, and noise conditions, a new fault diagnosis method based on an image information fusion and Vision Transformer (ViT) transfer learning model is proposed in this paper. Firstly, the method applies continuous wavelet transform (CWT), Gramian angular summation field (GASF), and Gramian angular difference field (GADF) to the time series data, and generates three grayscale images. Then, the generated three grayscale images are merged into an information fusion image (IFI) using image processing techniques. Finally, the obtained IFIs are fed into the advanced ViT model and trained based on transfer learning. In order to verify the effectiveness and superiority of the proposed method, the rolling bearing dataset from Case Western Reserve University (CWRU) is used to carry out experimental studies under different working conditions. Experimental results show that the method proposed in this paper is superior to other traditional methods in terms of accuracy, and the effect of ViT model based on transfer learning (TLViT) training is better than that of the Resnet50 model based on transfer learning training (TLResnet50) under variable loads and small sample conditions. In addition, the experimental results also prove that the IFI with multiple image information has better anti-noise ability than the single information image. Therefore, the method proposed in this paper can improve the accuracy of bearing fault diagnosis under small sample, variable load and noise conditions, and provide a new method for bearing fault diagnosis.

Soybean polypeptide content detection based on fusion of spectral and image information

Soybean polypeptide is a protein hydrolysate. It has multiple benefits, such as anticancer, antioxidant, etc. Addressing the limitations of traditional methods, such as time-consuming procedures and high costs, this study proposed a novel approach for the rapid, nondestructive detection of soybean polypeptide content. This approach utilizes the fusion of spectral and image information. First, seven preprocessing methods were applied to the spectral data. The results indicated that the model accuracy was highest after applying the standard normal variate preprocessing method. Second, a combined method, consisting of synergy interval partial least squares and iteratively retains informative variables, was proposed to extract characteristic variables. A two-layer Stacking framework ensemble learning model was constructed to compare the detection effects between single spectral input and fusion of spectral and image information input. The results showed that the fusion input model achieved the best performance. The coefficient of determination (R 2) of prediction of the fusion input model reached 0.952, an improvement of 0.061 compared to single spectral input. And root mean square error of prediction (RMSEP) of the fusion input model reached 1.803 × 10−4, a reduction of 7.43 × 10−5 compared to single spectral input. These results indicate that the fusion of spectral and image information can effectively improve the model’s accuracy, offering valuable technical support for soybean breeding and quality detection.

Transfer learning–based PET/CT three-dimensional convolutional neural network fusion of image and clinical information for prediction of EGFR mutation in lung adenocarcinoma

BackgroundTo introduce a three-dimensional convolutional neural network (3D CNN) leveraging transfer learning for fusing PET/CT images and clinical data to predict EGFR mutation status in lung adenocarcinoma (LADC).MethodsRetrospective data from 516 LADC patients, encompassing preoperative PET/CT images, clinical information, and EGFR mutation status, were divided into training (n = 404) and test sets (n = 112). Several deep learning models were developed utilizing transfer learning, involving CT-only and PET-only models. A dual-stream model fusing PET and CT and a three-stream transfer learning model (TS_TL) integrating clinical data were also developed. Image preprocessing includes semi-automatic segmentation, resampling, and image cropping. Considering the impact of class imbalance, the performance of the model was evaluated using ROC curves and AUC values.ResultsTS_TL model demonstrated promising performance in predicting the EGFR mutation status, with an AUC of 0.883 (95%CI = 0.849–0.917) in the training set and 0.730 (95%CI = 0.629–0.830) in the independent test set. Particularly in advanced LADC, the model achieved an AUC of 0.871 (95%CI = 0.823–0.919) in the training set and 0.760 (95%CI = 0.638–0.881) in the test set. The model identified distinct activation areas in solid or subsolid lesions associated with wild and mutant types. Additionally, the patterns captured by the model were significantly altered by effective tyrosine kinase inhibitors treatment, leading to notable changes in predicted mutation probabilities.ConclusionPET/CT deep learning model can act as a tool for predicting EGFR mutation in LADC. Additionally, it offers clinicians insights for treatment decisions through evaluations both before and after treatment.

Research on a multi-dimensional image information fusion algorithm based on NSCT transform

Traditional inspection cameras determine targets and detect defects by capturing images of their light intensity, but in complex environments, the accuracy of inspection may decrease. Information based on polarization of light can characterize various features of a material, such as the roughness, texture, and refractive index, thus improving classification and recognition of targets. This paper uses a method based on noise template threshold matching to denoise and preprocess polarized images. It also reports on design of an image fusion algorithm, based on NSCT transform, to fuse light intensity images and polarized images. The results show that the fused image improves both subjective and objective evaluation indicators, relative to the source image, and can better preserve edge information and help to improve the accuracy of target recognition. This study provides a reference for the comprehensive application of multi-dimensional optical information in power inspection.Graphical

Development of a vehicle-mounted soil organic matter detection system based on near-infrared spectroscopy and image information fusion

In the practical application of farmland, the soil organic matter prediction model established by the traditional near-infrared (NIR) spectroscopy is affected by factors such as soil texture, which leads to a serious decline in the accuracy of the model. To improve the robustness and prediction accuracy of the model, a prediction model based on NIR spectroscopy and image fusion is proposed. A 1D-CNN organic matter prediction model (based on NIR spectroscopy) was established using eight characteristic wavelengths of extracted soil organic matter (932 nm, 999 nm, 1083 nm, 1191 nm, 1316 nm, 1356 nm, 1583 nm, and 1626 nm) as spectral information. A 2D -CNN organic matter prediction model was established using soil RGB images as information. Based on the idea of model weight fusion, 1D-CNN and 2D-CNN models are fused. When using small convolutional kernels (three-layer convolutional kernel size: 3*3, 1*1, 1*1) and 1D-CNN:2D-CNN = 6:4, the model has the highest prediction accuracy (R 2 = 0.872). The optimal fusion model was embedded into the inspection system. The final laboratory and field testing results are as follows: under laboratory conditions, the detection accuracy R 2 of the 1D CNN prediction model, 2D-CNN prediction model, and fusion model are 0.838, 0.781, and 0.869, respectively. The root mean square error is 3.005, 3.546, and 2.678, respectively. The above experimental data indicates that the R 2 of the fused model is more accurate compared to the model established with a single information. In the field test, the R 2 detection accuracy of 1D-CNN prediction model, 2D-CNN prediction model and fusion model is 0.809, 0.731 and 0.835, respectively. The root mean square errors are 3.466, 3.828 and 2.973, respectively. The results show that the fusion model improves the prediction accuracy and model robustness, and the detection system can meet the needs of soil nutrient detection in farmland.

SUTrans-NET: a hybrid transformer approach to skin lesion segmentation.

Melanoma is a malignant skin tumor that threatens human life and health. Early detection is essential for effective treatment. However, the low contrast between melanoma lesions and normal skin and the irregularity in size and shape make skin lesions difficult to detect with the naked eye in the early stages, making the task of skin lesion segmentation challenging. Traditional encoder-decoder built with U-shaped networks using convolutional neural network (CNN) networks have limitations in establishing long-term dependencies and global contextual connections, while the Transformer architecture is limited in its application to small medical datasets. To address these issues, we propose a new skin lesion segmentation network, SUTrans-NET, which combines CNN and Transformer in a parallel fashion to form a dual encoder, where both CNN and Transformer branches perform dynamic interactive fusion of image information in each layer. At the same time, we introduce our designed multi-grouping module SpatialGroupAttention (SGA) to complement the spatial and texture information of the Transformer branch, and utilize the Focus idea of YOLOV5 to construct the Patch Embedding module in the Transformer to prevent the loss of pixel accuracy. In addition, we design a decoder with full-scale information fusion capability to fully fuse shallow and deep features at different stages of the encoder. The effectiveness of our method is demonstrated on the ISIC 2016, ISIC 2017, ISIC 2018 and PH2 datasets and its advantages over existing methods are verified.

Research on the Detection Method of Organic Matter in Tea Garden Soil Based on Image Information and Hyperspectral Data Fusion.

Soil organic matter is an important component that reflects soil fertility and promotes plant growth. The soil of typical Chinese tea plantations was used as the research object in this work, and by combining soil hyperspectral data and image texture characteristics, a quantitative prediction model of soil organic matter based on machine vision and hyperspectral imaging technology was built. Three methods, standard normalized variate (SNV), multisource scattering correction (MSC), and smoothing, were first used to preprocess the spectra. After that, random frog (RF), variable combination population analysis (VCPA), and variable combination population analysis and iterative retained information variable (VCPA-IRIV) algorithms were used to extract the characteristic bands. Finally, the quantitative prediction model of nonlinear support vector regression (SVR) and linear partial least squares regression (PLSR) for soil organic matter was established by combining nine color features and five texture features of hyperspectral images. The outcomes demonstrate that, in comparison to single spectral data, fusion data may greatly increase the performance of the prediction model, with MSC + VCPA-IRIV + SVR (R2C = 0.995, R2P = 0.986, RPD = 8.155) being the optimal approach combination. This work offers excellent justification for more investigation into nondestructive methods for determining the amount of organic matter in soil.

Aligning Multi-Sequence CMR Towards Fully Automated Myocardial Pathology Segmentation.

Myocardial pathology segmentation (MyoPS) is critical for the risk stratification and treatment planning of myocardial infarction (MI). Multi-sequence cardiac magnetic resonance (MS-CMR) images can provide valuable information. For instance, balanced steady-state free precession cine sequences present clear anatomical boundaries, while late gadolinium enhancement and T2-weighted CMR sequences visualize myocardial scar and edema of MI, respectively. Existing methods usually fuse anatomical and pathological information from different CMR sequences for MyoPS, but assume that these images have been spatially aligned. However, MS-CMR images are usually unaligned due to the respiratory motions in clinical practices, which poses additional challenges for MyoPS. This work presents an automatic MyoPS framework for unaligned MS-CMR images. Specifically, we design a combined computing model for simultaneous image registration and information fusion, which aggregates multi-sequence features into a common space to extract anatomical structures (i.e., myocardium). Consequently, we can highlight the informative regions in the common space via the extracted myocardium to improve MyoPS performance, considering the spatial relationship between myocardial pathologies and myocardium. Experiments on a private MS-CMR dataset and a public dataset from the MYOPS2020 challenge show that our framework could achieve promising performance for fully automatic MyoPS.

Pyramid-attentive GAN for multimodal brain image complementation in Alzheimer’s disease classification

Multimodal medical imaging has a larger volume of data compared to unimodal medical imaging, and can reflect different biological information and tissue features of the human body, complementing the structural details and image features missing from unimodal images to achieve a more accurate and comprehensive classification and diagnosis of diseases. In Alzheimer’s disease, the PET scan imaging technique is difficult to operate and expensive to detect, and is not included in routine examinations, resulting in a lack of PET image data in the dataset. In this paper, we propose a Generative Adversarial Network based on pyramidal attention mechanism to generate PET images through pyramidal attention mechanism and standard discriminators, which can effectively solve the problem of lack of PET data, complete the multimodal data sets of MRI and PET, combine the grey matter part of MRI images with the metabolic information in PET images to achieve multimodal medical image information fusion, and achieve classification and diagnosis of fused images through neural network. The experimental results of AD:MCI:NC triple classification show that our method achieves 89.9% accuracy.

Infrared and Visible Image Fusion Based on Mask and Cross-Dynamic Fusion

Both single infrared and visible images have respective limitations. Fusion technology has been developed to conquer these restrictions. It is designed to generate a fused image with infrared information and texture details. Most traditional fusion methods use hand-designed fusion strategies, but some are too rough and have limited fusion performance. Recently, some researchers have proposed fusion methods based on deep learning, but some early fusion networks cannot adaptively fuse images due to unreasonable design. Therefore, we propose a mask and cross-dynamic fusion-based network called MCDFN. This network adaptively preserves the salient features of infrared images and the texture details of visible images through an end-to-end fusion process. Specifically, we designed a two-stage fusion network. In the first stage, we train the autoencoder network so that the encoder and decoder learn feature extraction and reconstruction capabilities. In the second stage, the autoencoder is fixed, and we employ a fusion strategy combining mask and cross-dynamic fusion to train the entire fusion network. This strategy is conducive to the adaptive fusion of image information between infrared images and visible images in multiple dimensions. On the public TNO dataset and the RoadScene dataset, we selected nine different fusion methods to compare with our proposed method. Experimental results show that our proposed fusion method achieves good results on both datasets.

Precision weed detection in wheat fields for agriculture 4.0: A survey of enabling technologies, methods, and research challenges

Weeds pose a serious threat to the safe wheat production. They are an important factor contributing to the reduction in wheat yield and quality. The current weed control methods in wheat fields have relied primarily on chemical control. The inability to determine the precise location of weeds has led to excessive usage and low utilization rate of pesticides, causing severe pollution. As agronomic operations evolve towards Agriculture 4.0, weed control technology in wheat fields is progressively becoming more precise and intelligent. Weed detection technologies and methods in wheat fields may provide the groundwork for improving the accuracy and efficiency in weeding. This study begins with a review of common weed species and distribution pattern in wheat fields and provides an in-depth analysis of current technologies and developments for weed detection. We focused on the current states of research in spectroscopy, image, imaging spectroscopy, depth information, and multi-modal information fusion for weed detection in wheat fields. We also summarized the trend of weed detection algorithms from traditional machine learning to deep learning and proposed trends of future development in wheat field weed detection. Our study has contributed to the implementation of more automatic and precise weeds management.

Chinese–Vietnamese Pseudo-Parallel Sentences Extraction Based on Image Information Fusion

Parallel sentences play a crucial role in various NLP tasks, particularly for cross-lingual tasks such as machine translation. However, due to the time-consuming and laborious nature of manual construction, many low-resource languages still suffer from a lack of large-scale parallel data. The objective of pseudo-parallel sentence extraction is to automatically identify sentence pairs in different languages that convey similar meanings. Earlier methods heavily relied on parallel data, which is unsuitable for low-resource scenarios. The current mainstream research direction is to use transfer learning or unsupervised learning based on cross-lingual word embeddings and multilingual pre-trained models; however, these methods are ineffective for languages with substantial differences. To address this issue, we propose a sentence extraction method that leverages image information fusion to extract Chinese–Vietnamese pseudo-parallel sentences from collections of bilingual texts. Our method first employs an adaptive image and text feature fusion strategy to efficiently extract the bilingual parallel sentence pair, and then, a multimodal fusion method is presented to balance the information between the image and text modalities. The experiments on multiple benchmarks show that our method achieves promising results compared to a competitive baseline by infusing additional external image information.

Multi-modal simultaneous machine translation fusion of image information

Simultaneous translation is to translate a sentence before people finish it, to understand the speaker's intention in real-time. At present, simultaneous machine translation still relies on text-to-text data resources. However, the output information from the encoder side is used for the decoder as the input data recourse in the pure text translation system. This information is only derived from the text content, and the input information is single, causing a shortage of decoding information at the decoder and the vocabulary is missed in translation. The translator will also visually capture the information of the surrounding scenes to assist himself in the translation work, based on this feature, we propose a multi-modal simultaneous machine translation of fusion image information. We extract information from the image, add the information to the decoder side of the translation system, increase the input data resource of the decoder, and help the system improve the translation quality. We use the Multi30K dataset for experimental verification. Compared with the translation system of plain text, the method we propose can translate more complete sentences, richer content, and better translation results.

Identification of Insulator Contamination Status Based on Multi-spectral Image Fusion and Radial Basis Function Neural Network

Insulator contamination monitoring is an important way to avoid insulator contamination and maintain insulation performance. In order to ensure power supply and achieve contactless detection of insulator contamination status, a method is proposed in this paper to identify insulator contamination status by adopting infrared, ultraviolet, and visible multi-spectral image information fusion. An insulator with different contamination states in a number of substations is taken as the research object in this paper. The image segmentation is performed by using the seed region growth method to extract the infrared, ultraviolet and visible features of the insulator surface, and the radial basis function neural network learning algorithm is used to classify and decompose and fuse the images according to their different local area energies. The comparison of the recognition rates using infrared and ultraviolet features with those fused shows that the method has significant advantages and provides a new method for the detection of insulator contamination status.

Large Data Oriented to Image Information Fusion Spark and Improved Fruit Fly Optimization Based on the Density Clustering Algorithm

The density-based applied spatial clustering algorithm is an algorithm based on high-density interconnected regions, which discovers class clusters of arbitrary shapes in noisy data sets and is widely used. However, it suffers from slow computation speed due to large-scale disk I/O and clustering bias due to uneven density class clusters and poor parameter search ability. To address these problems, a parallel density clustering algorithm based on an improved fruit fly optimization algorithm and Spark memory iteration is proposed. The proposed algorithm first divides the data grid using an irregular dynamic density region partitioning strategy. Then, a hybrid fruit fly particle swarm algorithm based on a genetic optimization mechanism is proposed to achieve dynamic optimization seeking for parameters in local clustering to improve the clustering effect of local clustering. Finally, the local merging of samples in irregularly bounded grid cells under each partition is achieved by designing a custom clustering merging strategy. The experiments show that the improved algorithm is generally applicable to the clustering of different shape class clusters and larger scale data and has obvious improvement in accuracy and parallel efficiency.

A Crack Defect Detection and Segmentation Method That Incorporates Attention Mechanism and Dimensional Decoupling

In this work, we propose a new crack image detection and segmentation method for addressing the issues regarding the poor detection of crack structures in certain complex background conditions, such as the light and shadow, and the easy-to-lose details in segmentation. This method can be categorized into two phases, where the first one is the coding phase. In this phase, the channel attention mechanism and crack characteristics, using the correlation channel with different scales increasing the network robustness and ability of feature extraction, have been introduced to decouple the channel dimension and space dimension. It also avoids underfitting caused by information redundancy during the jumping connection. In the second stage, i.e., the decoding stage, the spatial attention mechanism has been introduced to capture the crack edge information through the global maximum pooling and global average pooling of the high-dimensional features. Then, the correlation between the space and channel has been recovered through multiscale image information fusion to achieve accurate crack positioning. Furthermore, the Dice loss function has been employed to solve the problem of pixel imbalance between the categories. Finally, the proposed method has been tested and compared with existing methods. The experimental results illustrate that our method has a higher crack segmentation accuracy than existing methods. Furthermore, the mean intersection over the union ratio reaches 87.2% on the public dataset and 83.9% on the self-built dataset, and it has a better segmentation effect and richer details. It can solve the problem of crack image detection and segmentation under a complex background.

Estimation of Wheat Plant Height and Biomass by Combining UAV Imagery and Elevation Data

Aboveground biomass (AGB) is an important basis for wheat yield formation. It is useful to timely collect the AGB data to monitor wheat growth and to build high-yielding wheat groups. However, as traditional AGB data acquisition relies on destructive sampling, it is difficult to adapt to the modernization of agriculture, and the estimation accuracy of spectral data alone is low and cannot solve the problem of index saturation at later stages. In this study, an unmanned aerial vehicle (UAV) with an RGB camera and the real-time kinematic (RTK) was used to obtain imagery data and elevation data at the same time during the critical fertility period of wheat. The cumulative percentile and the mean value methods were then used to extract the wheat plant height (PH), and the color indices (CIS) and PH were combined to invert the AGB of wheat using parametric and non-parametric models. The results showed that the accuracy of the model improved with the addition of elevation data, and the model with the highest accuracy of multi-fertility period estimation was PLSR (PH + CIS), with R2, RMSE and NRMSE of 0.81, 1248.48 kg/ha and 21.77%, respectively. Compared to the parametric models, the non-parametric models incorporating PH and CIS greatly improved the prediction of AGB during critical fertility periods in wheat. The inclusion of elevation data therefore greatly improves the accuracy of AGB prediction in wheat compared to traditional spectral prediction models. The fusion of UAV-based elevation data and image information provides a new technical tool for multi-season wheat AGB monitoring.

VANet: a medical image fusion model based on attention mechanism to assist disease diagnosis

BackgroundToday’s biomedical imaging technology has been able to present the morphological structure or functional metabolic information of organisms at different scale levels, such as organ, tissue, cell, molecule and gene. However, different imaging modes have different application scope, advantages and disadvantages. In order to improve the role of medical image in disease diagnosis, the fusion of biomedical image information at different imaging modes and scales has become an important research direction in medical image. Traditional medical image fusion methods are all designed to measure the activity level and fusion rules. They are lack of mining the context features of different modes of image, which leads to the obstruction of improving the quality of fused images.MethodIn this paper, an attention-multiscale network medical image fusion model based on contextual features is proposed. The model selects five backbone modules in the VGG-16 network to build encoders to obtain the contextual features of medical images. It builds the attention mechanism branch to complete the fusion of global contextual features and designs the residual multiscale detail processing branch to complete the fusion of local contextual features. Finally, it completes the cascade reconstruction of features by the decoder to obtain the fused image.ResultsTen sets of images related to five diseases are selected from the AANLIB database to validate the VANet model. Structural images are derived from MR images with high resolution and functional images are derived from SPECT and PET images that are good at describing organ blood flow levels and tissue metabolism. Fusion experiments are performed on twelve fusion algorithms including the VANet model. The model selects eight metrics from different aspects to build a fusion quality evaluation system to complete the performance evaluation of the fused images. Friedman’s test and the post-hoc Nemenyi test are introduced to conduct professional statistical tests to demonstrate the superiority of VANet model.ConclusionsThe VANet model completely captures and fuses the texture details and color information of the source images. From the fusion results, the metabolism and structural information of the model are well expressed and there is no interference of color information on the structure and texture; in terms of the objective evaluation system, the metric value of the VANet model is generally higher than that of other methods.; in terms of efficiency, the time consumption of the model is acceptable; in terms of scalability, the model is not affected by the input order of source images and can be extended to tri-modal fusion.