Multi-sensor Image Fusion Research Articles

Integrating temporal-aggregated satellite image with multi-sensor image fusion for seasonal land-cover mapping of Shilansha watershed, rift valley basin of Ethiopia

Accurate land-cover mapping in regions with frequent cloud-cover and rapidly changing agricultural land cover by crop growth cycles cannot be guaranteed by use of single sensor images, or an image from a single-acquisition-date. This study addressed these challenges by applying temporal-aggregation of single sensor image features that is integrated with multi-sensor image fusion. Results of land-cover classification target fallow, growing, and harvest/post-harvest agricultural seasons. Satellite based features used were frequency bands of Sentinel-1 (S1) and Sentinel-2 (S2), including vegetation indices (VIs) and biophysical variables (BPVs). Temporal aggregation improved classification accuracy. The single-acquisition-date S2 image, overall accuracy (OA) ranged from 0.81 to 0.85, increased to 0.86 to 0.87 after temporal-aggregation. Meanwhile, for single-acquisitions of S1, OA ranged from 0.44 to 0.79 increased to 0.6 to 0.86 across respective seasons. Fusing temporally aggregated S1 and S2 image features including VIs and BPVs increased OA up to 0.90. Selecting 11, 8, and 10 out of 18 optimum numbers of features for fallow, growing, and harvest/post-harvest seasons respectively improved OA by 3%, 2%, and 1.86%. PCA fusion of the temporally aggregated best performing feature set enhanced harvest/post-harvest season, fallow, and growing seasons with OA of 0.98, 0.96 and 0.94 respectively. Accuracy was enhanced when selecting different best performing feature set for the three seasons. The study enhanced knowledge of advanced remote sensing for agricultural land cover mapping, with practical implications of land monitoring and management.

Development of Multi Modal Image Fusion Techniques in Night Vision Technology Using Deep Learning

This study is aimed at exploring new development in nighttime image processing based on nighttime photography/night vision. We use various learning methodologies based on neural networks to develop multisensor data processing techniques. The ability of Convolutional Neural Networks (CNN) to enhance image quality and clarity under conditions of low light or at night is the focus of our research. To do this A large database has been created, containing 3200 images both taken from the internet as well as our own. This data is preprocessed, normalized and features expanded using some common methods of data augmentation in order to optimize model performance. We compared our proposed CNN model with existing literature through a series of experiments. Performance was measured using Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Feature Similarity Index (FSIM). The Results are now in. They showed that our model can improve various metrics with higher scores. And they show the real-time monitoring capabilities in urban streets for our model. This research brings multisensor image fusion based on night-adjacent techniques forward from night vision technology and provides insight into practical and potential future directions for using deep learning techniques to improve safety and security in many different environments.

Bounded PCA based Multi Sensor Image Fusion Employing Curvelet Transform Coefficients

The fusion of thermal and visible images acts as an important device for target detection. The quality of the spectral content of the fused image improves with wavelet-based image fusion. However, compared to PCA-based fusion, most wavelet-based methods provide results with a lower spatial resolution. The outcome gets better when the two approaches are combined, but they may still be refined. Compared to wavelets, the curvelet transforms more accurately depict the edges in the image. Enhancing the edges is a smart way to improve spatial resolution and the edges are crucial for interpreting the images. The fusion technique that utilizes curvelets enables the provision of additional data in both spectral and spatial areas concurrently. In this paper, we employ an amalgamation of Curvelet Transform and a Bounded PCA (CTBPCA) method to fuse thermal and visible images. To evidence the enhanced efficiency of our proposed technique, multiple evaluation metrics and comparisons with existing image merging methods are employed. Our approach outperforms others in both qualitative and quantitative analysis, except for runtime performance. Future Enhancement-The study will be based on using the fused image for target recognition. Future work should also focus on this method’s continued improvement and optimization for real-time video processing.

TGF: Multiscale transformer graph attention network for multi-sensor image fusion

Multisensor image fusion is a challenging task that aims to produce a composite image by fusing visible (VI) and infrared (IR) images. Deep neural networks have shown impressive performance for VI and IR image fusion however, majority of them overlook the internal patch-recurrence property of source images, which limits their ability to learn diverse features. To address this issue, we propose a novel fusion framework based on vision transformer and graph attention, utilizing local patch repetition to enhance the feature representation and texture recovery. In particular, the proposed transformer blocks can learn high-frequency domain-specific information of the source images. The graph attention mechanism provides additional guidance for the features utilizing the similarity and symmetry information across patches. Furthermore, the proposed graph attention fusion block (GAFB) improves the selectivity and effectiveness of feature learning. The GAFB can identify significant corresponding local and global details of source images. The complementary information containing long-range and local symmetric details across domains is combined while preserving the appropriate apparent intensity to generate the fused image. Through extensive evaluations on benchmark datasets, our proposed technique demonstrates superior performance. Significantly, our approach achieves SSIM scores of 0.7552 on the TNO dataset and 0.7673 on the roadscene dataset, surpassing the state-of-the-art techniques used for evaluation.

Research on Image Fusion in UAV Visual Navigation

The accuracy of multi-sensor image fusion is the key to improve the accuracy of UAV visual navigation. In this paper, a two-step registration method combining region registration and feature registration is proposed. First, rough image registration is carried out based on improved Hough transform and Fourier-Mellin transform, and image precise registration is implemented based on Harris corner feature and cross-correlation function. Finally, wavelet transform is applied to image fusion and image edge enhancement to realize the coarse to accurate registration of SAR images.

Infrared and Visible Image Fusion via Feature-Oriented Dual-Module Complementary

With the industrial demand caused by multi-sensor image fusion, infrared and visible image fusion (IVIF) technology is flourishing. In recent years, scale decomposition methods have led the trend for feature extraction. Such methods, however, have low time efficiency. To address this issue, this paper proposes a simple yet effective IVIF approach via a feature-oriented dual-module complementary. Specifically, we analyze five classical operators comprehensively and construct the spatial gradient capture module (SGCM) and infrared brightness supplement module (IBSM). In the SGCM, three kinds of feature maps are obtained, respectively, by introducing principal component analysis, saliency, and proposing contrast estimation operators considered the relative differences of contrast information covered by the input images. These maps are later reconstructed through pyramidal transformation to obtain the predicted image. The IBSM is then proposed to refine the missing infrared thermal information in the predicted image. Among them, we improve the measurement operators applied to the exposure modalities, namely, the gradient of the grayscale images (2D gradient) and well-exposedness. The former is responsible for extracting fine details, and the latter is meant for locating brightness regions. Experiments performed on public datasets demonstrate that the proposed method outperforms nine state-of-the-art methods in terms of subjective visual and objective indicators.

Neural network-based multisensor image fusion algorithm

Aiming at the problems of high mean square error and low fusion efficiency of existing fusion algorithms, a neural network-based multi-sensor image fusion algorithm is proposed. The fusion algorithm based on depth-separable convolution neural network (CNN) is determined by analyzing the quality evaluation and fusion methods of multi-sensor images, and summarizing the fusion rules. It is found that the integrity of image information acquisition is 97%, the mean square error is 4, and the fusion time is 2 s. Therefore, the algorithm has a good image fusion effect.

Application of Guided Image Filter and NSST for Contrast Enhanced Multi Sensor Image Fusion

Application of Guided Image Filter and NSST for Contrast Enhanced Multi Sensor Image Fusion

Multi-Sensor Image Fusion Method for Defect Detection in Powder Bed Fusion.

Multi-sensor defect detection technology is a research hotspot for monitoring the powder bed fusion (PBF) processes, of which the quality of the captured defect images and the detection capability is the vital issue. Thus, in this study, we utilize visible information as well as infrared imaging to detect the defects in PBF parts that conventional optical inspection technologies cannot easily detect. A multi-source image acquisition system was designed to simultaneously acquire brightness intensity and infrared intensity. Then, a multi-sensor image fusion method based on finite discrete shearlet transform (FDST), multi-scale sequential toggle operator (MSSTO), and an improved pulse-coupled neural networks (PCNN) framework were proposed to fuse information in the visible and infrared spectra to detect defects in challenging conditions. The image fusion performance of the proposed method was evaluated with different indices and compared with other fusion algorithms. The experimental results show that the proposed method achieves satisfactory performance in terms of the averaged information entropy, average gradient, spatial frequency, standard deviation, peak signal-to-noise ratio, and structural similarity, which are 7.979, 0.0405, 29.836, 76.454, 20.078 and 0.748, respectively. Furthermore, the comparison experiments indicate that the proposed method can effectively improve image contrast and richness, enhance the display of image edge contour and texture information, and also retain and fuse the main information in the source image. The research provides a potential solution for defect information fusion and characterization analysis in multi-sensor detection systems in the PBF process.

Development of a multi-sensor system for defects detection in additive manufacturing

Defects detection technology is essential for monitoring and hence maintaining the product quality of additive manufacturing (AM) processes; however, traditional detection methods based on single sensor have great limitations such as low accuracy and scarce information. In this study, a multi-sensor defect detection system (MSDDS) was proposed and developed for defect detection with the fusion of visible, infrared, and polarization detection information. The assessment criteria for imaging quality of the MSDDS have been optimized and evaluated. Meanwhile, the feasibility of processing and assembly of each sensor module has been demonstrated with tolerance sensitivity and the Monte Carlo analysis. Moreover, multi-sensor image fusion processing, super-resolution reconstruction, and feature extraction of defects are applied. Simulation and experimental studies indicate that the developed MSDDS can obtain high contrast and clear key information, and high-quality detected images of AM defects such as cracking, scratches, and porosity can be effectively extracted. The research provides a helpful and potential solution for defect detection and processing parameter optimization in AM processes such as Selective Laser Melting.

A Study of Image Fusion and Techniques for Denoising In Different Transformations

Image fusion is the process of combining relevant data from a collection of data images into a single image. Image fusion has emerged as a new and attractive investigation zone as multi-sensor information has been more widely available in sectors such as distant detecting, clinical imaging, machine vision, and military applications. In remote detection, image fusion aims to create new images with both low spatial goal multispectral information (shading data) and high spatial goal panchromatic information (subtleties). Different programming calculations can be used to create an entangled image with a greater spatial aim; nevertheless, the bulk of image preparation calculations are timeconsuming due to the large number of figures involved. It is appealing to use a quick reconfigurable equipment framework, such as a Field Programmable Gate Array, to handle difficult calculating calculations and execute similar operations with swift qualities (FPGA). The use of multisensor image fusion on FPGA, on the other hand, appears to be a promising area of research. As a result, the primary goal of this study is to create and implement a rapid discrete wavelet transform (DWT) based multisensory image fusion via equipment programming co-reenactment.

Wireless Sensor System of UAV Infrared Image and Visible Light Image Registration Fusion

The application of multisource sensors to drones requires high-quality images to ensure it. In simultaneous interpreting two or more multisensor images based on the same scene or target, the image obtained by the UAV sensor is limited by the imaging time and the shooting angle. The images obtained may not be aligned in the spatial position, thus affecting the fusion effect. Therefore, different sensor images must be registered before image fusion. During the shooting process of the drone imaging sensor, imaging angle, and environmental conditions, the obtained various sensor images will have rotation, translation, and other deformations in the spatial position so that they do not reach the spatial position. Therefore, it is impossible to directly perform image fusion directly. Therefore, before the multisensor image fusion, the image registration process must be completed to ensure that the two images are aligned in space. This paper analyzes the principles; based on the principles of the Powell search algorithm and improved walking algorithm, an algorithm combining Powell and improved walking algorithm is proposed. This paper also studies several traditional image neutrosophic fusions. The algorithm combines the fusion optimization algorithm proposed in this paper greatly reduces the calculation speed and improves the performance of the optimization algorithm and success rate.

An Image Fusion Algorithm based on Nonsampled Contourlet Transform

In this paper, a multisensor image fusion method based on non-sampling Contourlet transform is proposed. The transform has translation invariance, is suitable for expressing images with rich detail information and direction information, and can avoid the ringing effect introduced by general methods to the fused images. The image is decomposed by the transform to obtain high and low frequency components. The high-frequency component uses PCNN as the fusion rule, and the low-frequency component uses the golden section method to search the optimal low-frequency fusion weight, so as to fuse the low-frequency subband coefficients of multi-sensor images adaptively. Comparative experimental results show that the proposed method can achieve better fusion effect.

Image Recognition Method of Agricultural Pests Based on Multisensor Image Fusion Technology

With the rise and development of precision agriculture and smart agriculture concepts, traditional agricultural pest detection and identification methods have become increasingly unable to meet current agricultural production requirements due to their slow recognition speed, low recognition accuracy, and strong subjectivity need. This article aims to combine multifeature fusion technology with sensors to apply to crop pest detection and build crop pest detection services based on image recognition. In terms of image recognition, the use of image denoising methods based on median filtering, image preprocessing methods based on the maximum between-class error method (Otsu), image segmentation methods based on super green features, and feature extraction methods based on multiparameter features and based on the one-to-one elimination strategy and the M-SVM multiclass recognition algorithm fused with the kernel function, it realizes the identification and detection of six soybean leaf borers. The system uses the ARM920T series S3C2440 chip as the central processing unit. Through the temperature and humidity sensor and infrared, the multisensor module composed of sensors collects real-time information on the agricultural greenhouse. After normalizing the information, the central processing unit performs judgment processing and information fusion. And through experimental data, it is finally verified that the image recognition method used in this paper improves the recognition rate and effectiveness by nearly 65% in the detection of soybean leaf moth pests.

Outdoor real-time RGBD sensor fusion of stereo camera and sparse lidar

Autonomous driving technology relies on high-resolution, high-performance vision sensing technology. As the two most popular sensors in the industry, Lidar and stereo cameras play an important role in sensing, detection, control and planning decisions. At this stage, the Lidar can obtain more accurate depth map information, but with the increase of resolution, its cost increases sharply and cannot be applied in large scale in the market. At the same time, the depth detection scheme based on computer vision has high resolution, but the data accuracy is very low. In order to solve the practical application defects of the above two sensors, this paper proposes a new sensor fusion method for stereo camera and low-resolution Lidar, which has high resolution, high performance and low cost. This paper adopts new sensor design, multi-sensor calibration, classification and selection of Lidar pointcloud features, large-scale and efficient stereo image matching and depth map calculation, and method of filling missing information based on pointcloud segmentation. To verify the effectiveness of the method in this paper, this paper used high-resolution Lidar as ground truth for comparison. The results show that the fusion method can achieve an average improved accuracy of 30% within a close range of 30 meters, while achieving 98% resolution. In addition, this paper presents a scheme for visualizing multi-sensor image fusion, and encapsulates five modules: multi-sensor calibration, large-scale stereo depth calculation, low-resolution Lidar simulation, sensor data fusion, and fusion image and error visualization, in order to facilitate future secondary development.

Detection of surface defects and subsurface defects of polished optics with multisensor image fusion

Surface defects (SDs) and subsurface defects (SSDs) are the key factors decreasing the laser damage threshold of optics. Due to the spatially stacked structure, accurately detecting and distinguishing them has become a major challenge. Herein a detection method for SDs and SSDs with multisensor image fusion is proposed. The optics is illuminated by a laser under dark field condition, and the defects are excited to generate scattering and fluorescence lights, which are received by two image sensors in a wide-field microscope. With the modified algorithms of image registration and feature-level fusion, different types of defects are identified and extracted from the scattering and fluorescence images. Experiments show that two imaging modes can be realized simultaneously by multisensor image fusion, and HF etching verifies that SDs and SSDs of polished optics can be accurately distinguished. This method provides a more targeted reference for the evaluation and control of the defects of optics, and exhibits potential in the application of material surface research.

Satellite Multi-Sensor Data Fusion for Soil Clay Mapping Based on the Spectral Index and Spectral Bands Approaches

Integrating satellite data at different resolutions (i.e., spatial, spectral, and temporal) can be a helpful technique for acquiring soil information from a synoptic point of view. This study aimed to evaluate the advantage of using satellite mono- and multi-sensor image fusion based on either spectral indices or entire spectra to predict the topsoil clay content. To this end, multispectral satellite images acquired by various sensors (i.e., Landsat-5 Thematic Mapper (TM), Landsat-8 Operational Land Imager (OLI), Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER), and Sentinel2-MultiSpectral Instrument (S2-MSI)) have been used to assess their potential in identifying bare soil pixels over an area in northeastern Tunisia, the Lebna and Chiba catchments. A spectral index image and a spectral bands image are generated for each satellite sensor (i.e., TM, OLI, ASTER, and S2-MSI). Then, two multi-sensor satellite image fusions are generated, one from the spectral index images and the other from spectral bands. The resulting spectral index and spectral band images based on mono-and multi-sensor satellites are compared through their spectral patterns and ability to predict the topsoil clay content using the Multilayer Perceptron with backpropagation learning algorithm (MLP-BP) method. The results suggest that for clay content prediction: (i) the spectral bands’ images outperformed the spectral index images regardless of the used satellite sensor; (ii) the fused images derived from the spectral index or bands provided the best performances, with a 10% increase in the prediction accuracy; and (iii) the bare soil images obtained by the fusion of many multispectral sensor satellite images can be more beneficial than using mono-sensor images. Soil maps elaborated via satellite multi-sensor data fusion might become a valuable tool for soil survey, land planning, management, and precision agriculture.

Fusion and classification of multi-temporal SAR and optical imagery using convolutional neural network

ABSTRACT Remote sensing image classification is difficult, especially for agricultural crops with identical phenological growth periods. In this context, multi-sensor image fusion allows a comprehensive representation of biophysical and structural information. Recently, Convolutional Neural Network (CNN)-based methods are used for several applications due to their spatial-spectral interpretability. Hence, this study explores the potential of fused multi-temporal Sentinel 1 (S1) and Sentinel 2 (S2) images for Land Use/Land Cover classification over an agricultural area in India. For classification, Bayesian optimised 2D CNN-based DL and pixel-based SVM classifiers were used. For fusion, a CNN-based siamese network with Ratio-of-Laplacian pyramid method was used for the images acquired over the entire winter cropping period. This fusion strategy leads to better interpretability of results and also found that 2D CNN-based DL classifier performed well in terms of classification accuracy for both single-month (95.14% and 96.11%) as well as multi-temporal (99.87% and 99.91%) fusion in comparison to the SVM with classification accuracy for single-month (80.02% and 81.36%) and multi-temporal fusion (95.69% and 95.84%). Results indicate better performance by Vertical-Vertical polarised fused images than Vertical-Horizontal polarised fused images. Thus, implying the need to analyse classified images obtained by DL classifiers along with the classification accuracy.

Real-Time One-Stream Semantic-Guided Refinement Network for RGB-Thermal Salient Object Detection

Salient Object Detection (SOD) has been widely used in practical applications such as multi-sensor image fusion, remote sensing, and defect detection. Recently, SOD from RGB and Thermal (T) has been rapidly developed due to its robustness to extreme situations like low illumination and occlusion. However, existing methods all utilize a dual-stream encoder, which significantly increases the computation burdens and hinders real-world deployment. To this end, we propose a real-time One-stream Semantic-guided Refinement Network (OSRNet) for RGB-T SOD. Specifically, we firstly fuse the RGB and T via concatenation, addition, and multiplication operations to dig the complementary information between each modality. The efficient early fusion not only facilitates the information exchange between each modality but also avoids the cumbersome dual-stream encoder structure. Then, the light-weight decoder is proposed, making the high-level semantic information filter the low-level noisy features and gradually refine the final prediction. Also, we apply deep supervision to make the training procedure more stable and fast. Due to the early fusion strategy, OSRNet can run at a real-time speed (53-60<i>fps</i>) on a single GPU. Extensive quantitative and qualitative experiments show our network outperforms eleven state-of-the-art methods in terms of seven evaluation metrics. Our codes have been released at: https://github.com/huofushuo/OSRNet.

DFPGAN: Dual fusion path generative adversarial network for infrared and visible image fusion

Infrared and visible image fusion is an essential task for multi-sensor image fusion. Generative adversarial networks (GAN) have achieved remarkable performance in the fusion of infrared and visible image. Existing GAN based fusion methods merely using infrared and visible image as input for the fusion, while we found that differential images obtained by subtraction between two image sources could provide contrast information for the fusion. To this end, a novel dual fusion path generative adversarial network (DFPGAN) is proposed in this paper for infrared and visible image fusion. We divided the generator of generative adversarial network into two fusion paths namely infrared–visible path and differential path. The input of infrared–visible path concatenated two image sources to make infrared intensity and texture details keep balance fusion in this path. The input of differential path concatenated differential images obtained by subtraction between two image sources to make contrast information fusion in this path. The features extracted by two fusion paths are concatenated at the end of the generator to generate fused images with contrast effect and balanced information distribution. Meanwhile, we have implemented dual self-attention feature refine module (DSAM) on two fusion paths to refine feature maps in two fusion paths. We adopted switchable normalization layer (SN) substitute for batch normalization layer (BN) in the generator and discriminator to avoid fusion artifact. Furthermore, a mixed content loss is integrated in the generator loss functions to guide the generated image keep balanced information distribution and preserving contrast simultaneously. The adversarial training employed dual adversarial architecture to balance the distribution of infrared intensity and texture details. To verifying the improvement effect of fusion image on target detection, we introduce the Scaled-YOLOv4 target detection framework as evaluation framework, and use the proposed network to fuse RGB images and infrared images for target detection. The results of qualitative and quantitative experiments conducted on public datasets demonstrated the superiority of proposed network over other state-of-the-art methods and could generate fused images with distinctly contrast.