Multi-source Image Fusion Research Articles

Chicken body temperature monitoring method in complex environment based on multi-source image fusion and deep learning

Severe diseases in chickens present substantial risks to poultry husbandry industry. Notably, alterations in body temperature serve as critical clinical indicators of these diseases. Consequently, timely and accurate monitoring of body temperature is essential for the early detection of severe health issues in chickens. This study presents a novel method for simultaneous body temperature detection of multiple chickens in caged poultry environments. A dataset of 2896 chicken head images was developed. The YOLOv8n-mvc model was created to accurately detect chicken head positions and extracted temperature data and distance information through the fusion of RGB, thermal infrared, and depth images. The chicken head temperature was calibrated using distance information. The YOLOv8n-mvc model established in this study achieved a precision of 91.6 %, recall of 92.5 %, F1 score of 92.0 %, and mAP@0.5 of 96.0 %. The model was successfully deployed on an edge computing device for validation tests, demonstrating its feasibility for chicken body temperature detection. This study provides a reference for developing a chicken health monitoring system based on body temperature.

Multi-Source Image Fusion Based on BEMD and Region Sharpness Guidance Region Overlapping Algorithm

Multi-focal image and multi-modal image fusion technology can fully take advantage of different sensors or different times, retaining the image feature information and improving the image quality. A multi-source image fusion algorithm based on bidimensional empirical mode decomposition (BEMD) and a region sharpness-guided region overlapping algorithm are studied in this article. Firstly, source images are decomposed into multi-layer bidimensional intrinsic mode functions (BIMFs) and residuals from high-frequency layer to low-frequency layer by BEMD. Gaussian bidimensional intrinsic mode functions (GBIMFs) are obtained by applying Gaussian filtering operated on BIMF and calculating the sharpness value of segmented regions using an improved weighted operator based on the Tenengrad function, which is the key to comparison selection and fusion. Then, the GBIMFs and residuals selected by sharpness comparison strategy are fused by the region overlapping method, and the stacked layers are weighted to construct the final fusion image. Finally, based on qualitative evaluation and quantitative evaluation indicators, the proposed algorithm is compared with six typical image fusion algorithms. The comparison results show that the proposed algorithm can effectively capture the feature information of images in different states and reduce the redundant information.

Multi-Source Image Fusion Based Regional Classification Method for Apple Diseases and Pests

Efficient diagnosis of apple diseases and pests is crucial to the healthy development of the apple industry. However, the existing single-source image-based classification methods have limitations due to the constraints of single-source input image information, resulting in low classification accuracy and poor stability. Therefore, a classification method for apple disease and pest areas based on multi-source image fusion is proposed in this paper. Firstly, RGB images and multispectral images are obtained using drones to construct an apple diseases and pests canopy multi-source image dataset. Secondly, a vegetation index selection method based on saliency attention is proposed, which uses a multi-label ReliefF feature selection algorithm to obtain the importance scores of vegetation indices, enabling the automatic selection of vegetation indices. Finally, an apple disease and pest area multi-label classification model named AMMFNet is constructed, which effectively combines the advantages of RGB and multispectral multi-source images, performs data-level fusion of multi-source image data, and combines channel attention mechanisms to exploit the complementary aspects between multi-source data. The experimental results demonstrated that the proposed AMMFNet achieves a significant subset accuracy of 92.92%, a sample accuracy of 85.43%, and an F1 value of 86.21% on the apple disease and pest multi-source image dataset, representing improvements of 8.93% and 10.9% compared to prediction methods using only RGB or multispectral images. The experimental results also proved that the proposed method can provide technical support for the coarse-grained positioning of diseases and pests in apple orchards and has good application potential in the apple planting industry.

Walking representation and simulation based on multi-source image fusion and multi-agent reinforcement learning for gait rehabilitation

In the formulation of strategies for walking rehabilitation, achieving precise identification of the current state and making rational predictions about the future state are crucial but often unrealized. To tackle this challenge, our study introduces a unified framework that integrates a novel 3D walking motion capture method using multi-source image fusion and a walking rehabilitation simulation approach based on multi-agent reinforcement learning. We found that, (i) the proposal achieved an accurate 3D walking motion capture and outperforms other advanced methods. Experimental evidence indicates that, compared to similar visual skeleton tracking methods, the proposed approach yields results with higher Pearson correlation (r=0.93), intra-class correlation coefficient (ICC(2,1)=0.91), and narrower confidence intervals ([0.90,0.95] for r, [0.88,0.94] for ICC(2,1)) when compared to standard results. The outcomes of the proposed approach also exhibit commendable correlation and concurrence with those obtained through the IMU-based skeleton tracking method in the assessment of gait parameters ([0.85,0.89] for r, [0.75,0.81] for ICC(2,1)); (ii) multi-agent reinforcement learning has the potential to be used to solve the simulation task of gait rehabilitation. In mimicry experiment, our proposed simulation method for gait rehabilitation not only enables the intelligent agent to converge from the initial state to the target state, but also observes evolutionary patterns similar to those observed in clinical practice through motor state resolution. This study offers valuable contributions to walking rehabilitation, enabling precise assessment and simulation-based interventions, with potential implications for clinical practice and patient outcomes.

MATI: Multimodal Adaptive Tracking Integrator for Robust Visual Object Tracking.

Visual object tracking, pivotal for applications like earth observation and environmental monitoring, encounters challenges under adverse conditions such as low light and complex backgrounds. Traditional tracking technologies often falter, especially when tracking dynamic objects like aircraft amidst rapid movements and environmental disturbances. This study introduces an innovative adaptive multimodal image object-tracking model that harnesses the capabilities of multispectral image sensors, combining infrared and visible light imagery to significantly enhance tracking accuracy and robustness. By employing the advanced vision transformer architecture and integrating token spatial filtering (TSF) and crossmodal compensation (CMC), our model dynamically adjusts to diverse tracking scenarios. Comprehensive experiments conducted on a private dataset and various public datasets demonstrate the model's superior performance under extreme conditions, affirming its adaptability to rapid environmental changes and sensor limitations. This research not only advances visual tracking technology but also offers extensive insights into multisource image fusion and adaptive tracking strategies, establishing a robust foundation for future enhancements in sensor-based tracking systems.

Parallel Hierarchical Multi-View Feature Fusion Based on Canonical Correlation Analysis for Mammogram Retrieval

Due to the diversity of image sources, content-based multi-source image fusion and retrieval have shown promising capabilities in computer vision tasks, and especially when applied in Computer-Aided Diagnosis (CAD) to automate and improve the accuracy of medical image analysis. The combination of computer vision and CAD systems has the potential to revolutionize healthcare by augmenting the expertise of clinicians, improving overall diagnostic accuracy and helping experts in the clinical decision-making process by classifying and retrieving similar annotated clinical images to a given query. In the context of multi-view mammography interpretation, the concept of multi-view feature fusion has recently been studied to improve retrieval performance while effectively guaranteeing the complementarity of both MLO and CC views. However, conventional multi-view feature fusion makes descriptors long and lacks to take into consideration the relationship between descriptors. To deal with this issue, we propose two hierarchical multi-view feature fusion methods, for multi-view mammogram retrieval, based on the Canonical Correlation Analysis (CCA), which is the most commonly used multivariate parametric test. In fact, we have adapted CCA to determine the relationship between two descriptors by processing latent correlation factors. Moreover, after extracting descriptors for each view, a comparative study of texture and shape fusion descriptors is proposed in order to identify the more discriminative features for multi-view mammogram retrieval. Then, a query-dependent distance metric preserving both visual resemblance and semantic similarity is carried out to dynamically determine the more appropriate distance measure for each query image. Extensive experiments on the challenging Curated Breast Imaging Subset of Digital Database for Screening Mammography (CBIS-DDSM) have demonstrated the effectiveness of the proposed hierarchical multi-view feature fusion for mammogram retrieval, which outperforms the performance achieved either by conventional fused information or by single view information. To improve the transparency of our paper, the source code of the proposed method and the related dataset (including readme files) are publicly accessible through the following GitHub link: https://github.com/ABDERRAHIMMAR/Multi-View-Feature-Fusion-for-Mammogram-Retrieval . This open-access resource empowers researchers and practitioners to delve deeper into our methodology, fostering collaboration and advancements in the field of computer-aided diagnosis and medical image analysis.

Automatic Pavement Crack Detection in Multisource Fusion Images Using Similarity and Difference Features

In urban pavement crack detection, the single detection platform with visible light sensors is mostly used for detection. Visible light sensors are highly susceptible to lighting conditions. In low-light environments or areas with shadows over the cracks, this may result in false positives or missed detections of pavement cracks. The paper utilizes an image acquisition system that integrates multiple detection platforms, including visible light and infrared sensors, to enhance both the efficiency and accuracy of pavement crack detection. We propose an automatic pavement crack detection approach that uses the similarities and differences between visible light images and infrared image features (MSFSD) to achieve all-weather, real-time detection of pavement cracks. MSFSD mainly includes multi-source image fusion and pixel-level crack detection. In multi-source image fusion, a generative adversarial network (C-GAN) with skip connections and dilated convolutional blocks (CRes2Net blocks) is constructed. The structural similarity index (SSIM) function and the sum of the correlations of differences (SCD) function are used as the loss function of the generator, allowing the fusion image to retain more details and edge information in the source image. In pixel-level crack detection, a crack detection model is constructed, and CRes2Net is used as the core network to obtain pixel-level labels and perform semantic segmentation. It is verified by experiments that MSFSD achieves 85.1% precision and 88.4% recall in different types of crack detection. The MIoU and detection efficiency of MSFSD are 9% and 48% higher than other methods, respectively.

In-bed human pose estimation using multi-source information fusion for health monitoring in real-world scenarios

In the field of 3D human pose estimation for in-bed health monitoring in real-world scenarios, challenging problems arise due to varying conditions of illumination and depth ambiguity. To address these issues, we followed the principle of 2D-to-3D pipeline and developed a novel 3D skeletons tracking approach based on multi-source image fusion. The utilization of thermal images allows us to estimate the 2D pose, as they are unaffected by changes in illumination and can be used even in complete darkness. Furthermore, depth images are employed to infer the pose depth, benefiting from their excellent depth resolution. During the 2D pose estimation phase, in order to exploit both spatial and temporal information, we have proposed a module for 2D pose estimation based on Graph Convolutional Network and Transformer network. To train this module, we have created a dataset using an independently developed automatic annotation method. In the subsequent 3D pose estimation phase, the typical 2D-to-3D lifting approaches entail inferring a 3D human pose from an intermediate 2D pose estimated by a deep network, but the task is an ill-problem due to the lack of depth resolution in thermal images. In our study, we overcome this challenge by inferring the pose depth through fusion with depth images, which offer superior depth resolution but are subject to limitations posed by illumination and texture conditions. We found that, (i) the developed automatic annotation method yields accurate pose annotations for training 2D pose estimation based on infrared thermal images; (ii) on the proposed dataset, our 2D pose estimation framework achieved a mean error of below 4 pixels and outperformed state-of-the-art methods; (iii) it achieved precise 3D pose estimation via fusion with depth images; and (iv) future research should consider applying the outlined approach to support the development of novel human pose models for health monitoring.

Diffusion model with disentangled modulations for sharpening multispectral and hyperspectral images

The denoising diffusion model has received increasing attention in the field of image generation in recent years, thanks to its powerful generation capability. However, diffusion models should be deeply investigated in the field of multi-source image fusion, such as remote sensing pansharpening and multispectral and hyperspectral image fusion (MHIF). In this paper, we introduce a novel supervised diffusion model with two conditional modulation modules, specifically designed for the task of multi-source image fusion. These modules mainly consist of a coarse-grained style modulation (CSM) and a fine-grained wavelet modulation (FWM), which aim to disentangle coarse-grained style information and fine-grained frequency information, respectively, thereby generating competitive fused images. Moreover, some essential strategies for the training of the given diffusion model are well discussed, e.g., the selection of training objectives. The superiority of the proposed method is verified compared with recent state-of-the-art (SOTA) techniques by extensive experiments on two multi-source image fusion benchmarks, i.e., pansharpening and MHIF. In addition, sufficient discussions and ablation studies in the experiments are involved to demonstrate the effectiveness of our approach. The code is accessible at https://github.com/294coder/Dif-PAN for reproducibility purposes.

Robust Fusion of Multi-Source Images for Accurate 3D Reconstruction of Complex Urban Scenes

Integrated reconstruction is crucial for 3D modeling urban scenes using multi-source images. However, large viewpoint and illumination variations pose challenges to existing solutions. A novel approach for accurate 3D reconstruction of complex urban scenes based on robust fusion of multi-source images is proposed. Firstly, georeferenced sparse models are reconstructed from the terrestrial and aerial images using GNSS-aided incremental SfM, respectively. Then, cross-platform match pairs are selected based on point-on-image observability. The terrestrial and aerial images are robustly matched based on the selected match pairs to generate cross-platform tie points. Thirdly, the tie points are triangulated to derive cross-platform 3D correspondences. The 3D correspondences are refined using a novel outlier detection method. Finally, the terrestrial and aerial sparse models are merged based on the refined correspondences, and the integrated model is globally optimized to obtain an accurate reconstruction of the scene. The proposed methodology is evaluated on five benchmark datasets, and extensive experiments are performed. The proposed pipeline is compared with a state-of-the-art methodology and three widely used software packages. Experimental results demonstrate that the proposed methodology outperforms the other pipelines in terms of robustness and accuracy.

Thermal-textured BIM generation for building energy audit with UAV image fusion and histogram-based enhancement

Mapping Unmanned Aerial Vehicle (UAV)-based thermal images of building façades onto a Building Information Model (BIM) can greatly support building energy audit. However, accurately registering these images into BIM is challenging due to thermal image’s unique characteristics of low texture and high distortion. To address these issues, this paper proposes a new multi-source image fusion framework for registering UAV thermal images to BIM. First, a low-cost target is designed and produced using 3D printing for calibrating UAV thermal camera. The rectified thermal images are then fused with RGB images through Homography transformation, enabling thermal images to be registered accurately with BIM. Additionally, a histogram-based approach is proposed to correct the thermally inconsistent phenomenon and enhance the contrast to produce high-contrast thermal-textured BIM. Field tests confirmed a mean image fusion error of around 5.713 pixels. A case study on a masonry building demonstrated the feasibility and effectiveness of the proposed framework.

A survey of multi-source image fusion

A survey of multi-source image fusion

Enhanced crop classification through integrated optical and SAR data: a deep learning approach for multi-source image fusion

ABSTRACT Agricultural crop mapping has advanced over the last decades due to improved approaches and the increased availability of image datasets at various spatial and temporal resolutions. Considering the spatial and temporal dynamics of different crops during a growing season, multi-temporal classification frameworks are well-suited for mapping crops at large scales. Addressing the challenges posed by imbalanced class distribution, our approach combines the strengths of different deep learning models in an ensemble learning framework, enabling more accurate and robust classification by capitalizing on their complementary capabilities. This research aims to enhance the crop classification of maize, soybean, and wheat in Bei’an County, Northeast China, by developing a novel deep learning architecture that combines a three-dimensional convolutional neural network (3D-CNN) with a variant of convolutional recurrent neural networks (ConvRNN). The proposed method integrates multi-temporal Sentinel-1 polarimetric features with Sentinel-2 surface reflectance data for multi-source fusion and achieves an overall accuracy of 91.7%, a Kappa coefficient of 85.7%, and F1 scores of 93.7%, 92.2%, and 90.9% for maize, soybean, and wheat, respectively. Our proposed model is also compared with alternative data augmentation techniques, maintaining the highest mean F1 score (87.7%). The best performer was weakly supervised with ten per cent of ground truth data collected in Bei’an in 2017 and used to produce an annual crop map for measuring the model’s generalizability. The model learning reliability of the proposed method is interpreted through the visualization of model soft outputs and saliency maps.

Research on Multi-Source Image Fusion Technology in the Digital Reconstruction of Classical Garden and Ancient Buildings

Research on Multi-Source Image Fusion Technology in the Digital Reconstruction of Classical Garden and Ancient Buildings

Former-CR: A Transformer-Based Thick Cloud Removal Method with Optical and SAR Imagery

In the field of remote sensing, cloud and cloud shadow will result in optical remote sensing image contamination, particularly high cloud cover, which will result in the complete loss of certain ground object information. The presence of thick cloud severely limits the use of optical images in production and scientific research, so it is critical to conduct further research into removing the thick cloud occlusion in optical images to improve the utilization rate of optical images. The state-of-the-art cloud removal methods proposed are largely based on convolutional neural network (CNN). However, due to CNN’s inability to gather global content information, those cloud removal approaches cannot be improved further. Inspired by the transformer and multisource image fusion cloud removal method, we propose a transformer-based cloud removal method (Former-CR), which directly reconstructs cloudless images from SAR images and cloudy optical images. The transformer-based model can efficiently extract and fuse global and local context information in SAR and optical images, generating high-quality cloudless images with higher global consistency. In order to enhance the global structure, local details, and visual effect of the reconstructed image, we design a new loss function to guide the image reconstruction. A comparison with several SAR-based cloud removal methods through qualitative and quantitative experimental evaluation on the SEN12MS-CR dataset demonstrates that our proposed method is effective and superior.

Real-time panoramic map modeling method based on multisource image fusion and three-dimensional rendering

Under the complex environment, in the process of sensing and understanding the surrounding scene with an unmanned system, the traditional single-source panorama provides limited information and is prone to interference from lighting, it is difficult to capture hidden objects, and the user’s three-dimensional (3D) experience is poor. To solve this problem, we propose a real-time panoramic map modeling method based on multisource image fusion and 3D rendering with intensity, hue, saturation transform and wavelet technology, open graphics library. Five general image sharpness evaluation methods are used to analyze the single-source and multisource close-range and distant images during the day and night. The results show that in the daytime, the sharpness of the multisource image is significantly improved compared with the single-source visible light image, especially the energy gradient evaluation can reach 3.4 times. Similarly, compared with the single-source infrared image, the multisource image at night also improve significantly, the energy gradient evaluation can reach 1.4 times. This conclusion is verified by the subsequent experiments of several indoor scenes, the 3D rendering processing technology of panoramic map is realized, which improves user experience of 3D stereoscopic vision, and will provide multisource image fusion and 3D rendering technical support for unmanned real-time panoramic map modeling.

A novel pig-body multi-feature representation method based on multi-source image fusion

The detection of pig-body health has been a hot topic since it is closely related to the safety of agricultural products quality in recent years. In this paper, shape and temperature feature are viewed as health representation, and the multi-source image fusion method should be used to obtain pig-body multi-feature. However, traditional infrared and visible fusion algorithm could not represent more thermal radiation and texture information of fused images. To more usefully represent pig-body mulit-feature, a novel multi-source image fusion method is proposed in view of Generative Adversarial Network (GAN), named as MGANFuse. Firstly, multi-source images are fused by modified GAN fusion framework. Then, based on fused images, pig-body shape feature is represented by automatic threshold of Otsu segmentation and morphology method. Finally, maximum temperature feature is obtained based on pig-body shape representation. Experimental results reveal that representation method in view of proposed fusion model is capable of realizing 1.551–3.876% higher average accuracy rate than more recently published algorithms. Moreover, it lays the foundation for effective representation of pig-body multi-feature.

MsIFT: Multi-Source Image Fusion Transformer

Multi-source image fusion is very important for improving image representation ability since its essence relies on the complementarity between multi-source information. However, feature-level image fusion methods based on the convolution neural network are impacted by the spatial misalignment between image pairs, which leads to the semantic bias in merging features and destroys the representation ability of the region-of-interests. In this paper, a novel multi-source image fusion transformer (MsIFT) is proposed. Due to the inherent global attention mechanism of the transformer, the MsIFT has non-local fusion receptive fields, and it is more robust to spatial misalignment. Furthermore, multiple classification-based downstream tasks (e.g., pixel-wise classification, image-wise classification and semantic segmentation) are unified in the proposed MsIFT framework, and the fusion module architecture is shared by different tasks. The MsIFT achieved state-of-the-art performances on the image-wise classification dataset VAIS, semantic segmentation dataset SpaceNet 6 and pixel-wise classification dataset GRSS-DFC-2013. The code and trained model are being released upon the publication of the work.

Review of Remote Sensing Applications in Grassland Monitoring

The application of remote sensing technology in grassland monitoring and management has been ongoing for decades. Compared with traditional ground measurements, remote sensing technology has the overall advantage of convenience, efficiency, and cost effectiveness, especially over large areas. This paper provides a comprehensive review of the latest remote sensing estimation methods for some critical grassland parameters, including above-ground biomass, primary productivity, fractional vegetation cover, and leaf area index. Then, the applications of remote sensing monitoring are also reviewed from the perspective of their use of these parameters and other remote sensing data. In detail, grassland degradation and grassland use monitoring are evaluated. In addition, disaster monitoring and carbon cycle monitoring are also included. Overall, most studies have used empirical models and statistical regression models, while the number of machine learning approaches has an increasing trend. In addition, some specialized methods, such as the light use efficiency approaches for primary productivity and the mixed pixel decomposition methods for vegetation coverage, have been widely used and improved. However, all the above methods have certain limitations. For future work, it is recommended that most applications should adopt the advanced estimation methods rather than simple statistical regression models. In particular, the potential of deep learning in processing high-dimensional data and fitting non-linear relationships should be further explored. Meanwhile, it is also important to explore the potential of some new vegetation indices based on the spectral characteristics of the specific grassland under study. Finally, the fusion of multi-source images should also be considered to address the deficiencies in information and resolution of remote sensing images acquired by a single sensor or satellite.

A multi-autoencoder fusion network guided by perceptual distillation

In this study, a novel distillation paradigm named perceptual distillation is proposed to guide the training of image fusion networks without ground truths. In the paradigm, the student network which we called main autoencoder takes in source images and produces a fused image, and the teacher network is a well-trained network exploited to compute teacher representations of images. Knowledge in the teacher representations of source images is distilled and transferred to our student main autoencoder with the help of the perceptual saliency scheme. The scheme also derives a pixel level scheme of pixel compensation, which combines with source image to enhance the pixel intensity of the fused image. Moreover, a multi-autoencoder architecture is developed by assembling two auxiliary decoders behind the main autoencoder. The architecture is trained with self-supervision to consolidate fusion training against the limitation of teacher network. Qualitative and quantitative experiments demonstrate that the proposed network achieves the state-of-the-art performance on multi-source image fusion compared with the existing fusion methods.