Visible Image Fusion Method Research Articles

A novel multisource pig-body multifeature fusion method based on Gabor features

The multi-source image fusion has been a hot topic during recent years because of its higher segmentation accuracy rate. However, the traditional multi-source image fusion methods could not obtain better contrast and more details of the fused image. To better detect the pig-body feature, a novel infrared and visible image fusion method for pig-body segmentation and temperature detection is proposed in non-subsampled contourlet transform (NSCT) domain, named as NSCT-GF. Firstly, the visible and infrared images were decomposed into a series of multi-scale and multi-directional sub-bands using NSCT. Then, to better represent the fine-scale of texture information, the Gabor energy map was extracted by Gabor filter with even-symmetry, and the low-frequency coefficients were fused by the maximum of Ordinal encoding. Then, to preserve the more coarse-scale and edge detail information, Gabor filter with odd-symmetry was employed to fuse high-frequency NSCT sub-bands and the fused coefficients were reconstructed into a final fusion image by inverse NSCT. Next, the pig-body shape was obtained by Ostu automatic threshold segmentation and optimized by morphological processing. Finally, the pig-body temperature was extracted based on shape segmentation. Experimental results showed that the proposed segmentation method was capable of achieving 1.84–3.89% higher average segmentation accuracy rate than the prevailing conventional methods and also improved efficiency in terms of time consumption. It lays a foundation for accurately measuring the temperature of pig-body.

Infrared and Visible Image Fusion Based on Spatial Convolution Sparse representation

In the traditional sparse representation based infrared and visible image fusion method, the detail information is not able to be effectively extracted, resulting in the decrease of infrared target intensity and the blurry of visible background information. In order to solve the above issue, a new image fusion method based on spatial convolution sparse representation is proposed. Firstly, a spatial convolution sparse representation is used to perform two-scale decomposition of infrared and visible images by introducing a gradient regularization, and the detail and intensity information are extracted effectively from the source images. Then, the weighted average rule and the maximum selection rule are used to fuse the base layer and detail layer images, respectively. Finally, the fusion image is constructed. Experimental results illustrate that the proposed method is superior to the traditional fusion method based on sparse representation.

Unsupervised densely attention network for infrared and visible image fusion

Integrating the information of infrared and visible images without human supervision is a long-standing problem. A key technical challenge in this domain is how to extract features from heterogeneous data-sources and fuse them appropriately. Prior deep learning works either extract the middle layers information or use costly training step to improve fusion performance, which limited their performances in cluttered scenes and real-time applications. In this paper, we introduce a novel and pragmatic unsupervised infrared and visible image fusion method based on a pre-trained deep network, which employs a densely connection structure and incorporates the attention mechanism to achieve high fusion performance. Furthermore, we propose to use the cross-dimensional weighting and aggregation to compute the attention map for infrared and visible image fusion. The attention map enables more efficient feature extraction and captures more structure information from source images. We evaluate our method and compare it with ten typical state-of-the-art fusion methods. Extensive experimental results demonstrate that our method achieves state-of-the-art fusion performance in both subjective and objective evaluation.

Infrared and Visible Image Fusion with Hybrid Image Filtering

Image fusion is an important technique aiming to generate a composite image from multiple images of the same scene. Infrared and visible images can provide the same scene information from different aspects, which is useful for target recognition. But the existing fusion methods cannot well preserve the thermal radiation and appearance information simultaneously. Thus, we propose an infrared and visible image fusion method by hybrid image filtering. We represent the fusion problem with a divide and conquer strategy. A Gaussian filter is used to decompose the source images into base layers and detail layers. An improved co-occurrence filter fuses the detail layers for preserving the thermal radiation of the source images. A guided filter fuses the base layers for retaining the background appearance information of the source images. Superposition of the fused base layer and fused detail layer generates the final fusion image. Subjective visual and objective quantitative evaluations comparing with other fusion algorithms demonstrate the better performance of the proposed method.

A novel multi-source image fusion method for pig-body multi-feature detection in NSCT domain

The multi-source image fusion has been a hot topic during the recent years because of its higher detection rate. To improve the accuracy of pig-body multi-feature detection, a multi-source image fusion method was adopted in this field. However, the traditional multi-source image fusion methods could not obtain better contrast and more details of the fused image. To better detect shape and temperature feature of pig-body, a novel infrared and visible image fusion method was proposed in non-subsampled contourlet transform (NSCT) domain and named NSCT-GF-IAG. Through this technique, the visible and infrared images were first decomposed into a series of multi-scale and multi-directional sub-bands using NSCT. Then, to better represent the fine-scale of texture information and coarse-scale detail information, Gabor filter with even-symmetry and improved average gradient (IAG) were employed to fuse low-frequency and high-frequency sub-bands, respectively. Next, the fused coefficients were reconstructed into a final fusion image by inverse NSCT. Finally, the shape feature of pig-body was obtained by automatic threshold segmentation and optimized by morphological processing. Moreover, the highest temperature was extracted based on shape segmentation of pig-body. Experimental results showed that the proposed fusion method for detecting multi-feature was capable of achieving 2.175–5.129% higher average segmentation rate than the prevailing conventional methods. Besides this, the proposed method also improved efficiency in terms of time consumption.

A Heterogeneous Image Fusion Method Based on DCT and Anisotropic Diffusion for UAVs in Future 5G IoT Scenarios

Unmanned aerial vehicles, with their inherent fine attributes, such as flexibility, mobility, and autonomy, play an increasingly important role in the Internet of Things (IoT). Airborne infrared and visible image fusion, which constitutes an important data basis for the perception layer of IoT, has been widely used in various fields such as electric power inspection, military reconnaissance, emergency rescue, and traffic management. However, traditional infrared and visible image fusion methods suffer from weak detail resolution. In order to better preserve useful information from source images and produce a more informative image for human observation or unmanned aerial vehicle vision tasks, a novel fusion method based on discrete cosine transform (DCT) and anisotropic diffusion is proposed. First, the infrared and visible images are denoised by using DCT. Second, anisotropic diffusion is applied to the denoised infrared and visible images to obtain the detail and base layers. Third, the base layers are fused by using weighted averaging, and the detail layers are fused by using the Karhunen–Loeve transform, respectively. Finally, the fused image is reconstructed through the linear superposition of the base layer and detail layer. Compared with six other typical fusion methods, the proposed approach shows better fusion performance in both objective and subjective evaluations.

AttentionFGAN: Infrared and Visible Image Fusion Using Attention-Based Generative Adversarial Networks

Infrared and visible image fusion aims to describe the same scene from different aspects by combining complementary information of multi-modality images. The existing Generative adversarial networks (GAN) based infrared and visible image fusion methods cannot perceive the most discriminative regions, and hence fail to highlight the typical parts existing in infrared and visible images. To this end, we integrate multi-scale attention mechanism into both generator and discriminator of GAN to fuse infrared and visible images (AttentionFGAN). The multi-scale attention mechanism aims to not only capture comprehensive spatial information to help generator focus on the foreground target information of infrared image and background detail information of visible image, but also constrain the discriminators focus more on the attention regions rather than the whole input image. The generator of AttentionFGAN consists of two multi-scale attention networks and an image fusion network. Two multi-scale attention networks capture the attention maps of infrared and visible images respectively, so that the fusion network can reconstruct the fused image by paying more attention to the typical regions of source images. Besides, two discriminators are adopted to force the fused result keep more intensity and texture information from infrared and visible image respectively. Moreover, to keep more information of attention region from source images, an attention loss function is designed. Finally, the ablation experiments illustrate the effectiveness of the key parts of our method, and extensive qualitative and quantitative experiments on three public datasets demonstrate the advantages and effectiveness of AttentionFGAN compared with the other state-of-the-art methods.

Infrared and visible image fusion based on oversampled graph filter banks

The infrared image (RI) and visible image (VI) fusion method merges complementary information from the infrared and visible imaging sensors to provide an effective way for understanding the scene. The graph filter bank-based graph wavelet transform possesses the advantages of the classic wavelet filter bank and graph representation of a signal. Therefore, we propose an RI and VI fusion method based on oversampled graph filter banks. Specifically, we consider the source images as signals on the regular graph and decompose them into the multiscale representations with M-channel oversampled graph filter banks. Then, the fusion rule for the low-frequency subband is constructed using the modified local coefficient of variation and the bilateral filter. The fusion maps of detail subbands are formed using the standard deviation-based local properties. Finally, the fusion image is obtained by applying the inverse transform on the fusion subband coefficients. The experimental results on benchmark images show the potential of the proposed method in the image fusion applications.

Infrared and visible image fusion and denoising via [formula omitted] norm minimization

Most traditional infrared and visible image fusion methods often ignore noise in acquisition or transmission and their performance inevitably decreases in practical applications. To address this problem, a new and effective variational model is proposed for simultaneous image fusion and denoising. In an ℓ2−ℓp norm minimization setting with p=0 and p=1 respectively, the hybrid ℓ2 norm fidelity term is built to preserve image intensity and details from both infrared and visible images. And the nonconvex ℓ0 norm and convex ℓ1 norm sparsity constraints are applied to reduce noise while preserving important image fine features. Furthermore, a computationally efficient numerical algorithm based on half-quadratic splitting iteration is used to solve the complex optimization problem. Experimental results demonstrate that the proposed method can achieve a superior performance compared with existing fusion methods in both subjective and objective assessments.

Fractional-order total variation for improving image fusion based on saliency map

The fusion of infrared and visible images is difficult because of their different modalities. Current fusion methods are difficult to maintain both complementary information and good visual effects, such as methods of region discrimination based on visual saliency and methods based on total variation (TV). Among them, methods of region discrimination based on visual saliency for fusion have better complementary information but poor visual consistency, while methods based on total variation for fusion have good visual consistency, but there is no proper regularization to ensure sufficient selection and fusion of complementary information. In this paper, an improved infrared and visible image fusion method via visual saliency and fractional-order total variation is proposed. First, the infrared and visible images are fused through the saliency map to obtain a fused image, and then the fused image and an selected original image are fused by the fractional-order total variation to obtain the final fused image. In this paper, visual saliency map-based fusion makes the fused image contain as much complementary information as possible from the source image, while fractional-order total variation-based fusion makes the fused image have better visual effects. Compared with the state-of-the-art image fusion algorithm, the experimental results show that the proposed method is more competitive in retaining image texture details and having visual effects.

Infrared and Visible Image Fusion with a Generative Adversarial Network and a Residual Network

Infrared and visible image fusion can obtain combined images with salient hidden objectives and abundant visible details simultaneously. In this paper, we propose a novel method for infrared and visible image fusion with a deep learning framework based on a generative adversarial network (GAN) and a residual network (ResNet). The fusion is accomplished with an adversarial game and directed by the unique loss functions. The generator with residual blocks and skip connections can extract deep features of source image pairs and generate an elementary fused image with infrared thermal radiation information and visible texture information, and more details in visible images are added to the final images through the discriminator. It is unnecessary to design the activity level measurements and fusion rules manually, which are now implemented automatically. Also, there are no complicated multi-scale transforms in this method, so the computational cost and complexity can be reduced. Experiment results demonstrate that the proposed method eventually gets desirable images, achieving better performance in objective assessment and visual quality compared with nine representative infrared and visible image fusion methods.

基于潜在低秩表示与复合滤波的红外与弱可见光增强图像融合方法

基于潜在低秩表示与复合滤波的红外与弱可见光增强图像融合方法

Fusion of Infrared-Visible Images in UE-IoT for Fault Point Detection Based on GAN

In the development of modern intelligent Ubiquitous Electric Internet of Things (UE-IoT), infrared thermal imaging always plays an important role in automated early-warning detection of developing failures of critical assets such as transformers, disconnects and capacity banks in electrical power substation non-intrusively. However, the low resolution and contrast of infrared images hinder the subsequent analysis and recognition of fault points. In contrast, visible images present abundant texture details of the equipment without thermal information. In order to assist the detection of fault points, this paper proposes a Generative adversarial networks (GAN) based infrared and visible image fusion method to produce a composite image with enhanced edges and better quality. The edge loss function is added to represent the perceptual edges. In the discriminator, the proposed method improves the texture similarity between fusion image and visible image by minimizing the Wasserstein distance in VGG (Visual Geometry Group network) feature space. The experimental results show that the fault regions become more salient and the details are enhanced. In this way, it can facilitate the detection of fault points both reliably and accurately.

Infrared and Visible Image Fusion Using a Deep Unsupervised Framework With Perceptual Loss

The fusion of infrared and visible images can utilize the indication characteristics and the textural details of source images to realize the all-weather detection. The deep learning (DL) based fusion solutions can reduce the computational cost and complexity compared with traditional methods since there is no need to design complex feature extraction methods and fusion rules. There are no standard reference images and the publicly available infrared and visible image pairs are scarce. Most supervised DL-based solutions have to take pre-training on other labeled large datasets which may not behave well when testing. The few unsupervised fusion methods can hardly obtain ideal images with good visual impression. In this paper, an infrared and visible image fusion method based on unsupervised convolutional neural network is proposed. When designing the network structure, densely connected convolutional network (DenseNet) is used as the sub-network for feature extraction and reconstruction to ensure that more information of source images can be retained in the fusion images. As to loss function, the perceptual loss is creatively introduced and combined with the structure similarity loss to constrain the updating of weight parameters during the back propagation. The perceptual loss designed helps to improve the visual information fidelity (VIF) of the fusion image effectively. Experimental results show that this method can obtain fusion images with prominent targets and obvious details. Compared with other 7 traditional and deep learning methods, the fusion results of this method are better on objective evaluation and visual observation when taken together.

基于Tikhonov正则化和细节重建的红外与可见光图像融合方法

基于Tikhonov正则化和细节重建的红外与可见光图像融合方法

Infrared and visible image fusion based on intuitionistic fuzzy sets

Infrared and visible image fusion plays a vital role in target tracking, surveillance, military, etc. In this paper, we put forward a new infrared and visible image fusion method employing non-subsampled contourlet transform (NSCT) with intuitionistic fuzzy sets. We use NSCT to decompose the pre-registered test images into low frequency and high frequency sub-bands. Furthermore, we introduce a new rule based on intuitionistic fuzzy sets for high frequency sub-bands fusion, and for low frequency sub-bands, the energy-based fusion rule is used. The fused image is obtained by inverse NSCT. Experiment results show that the proposed fusion method outperforms the advanced fusion methods in terms of objective assessment and visual quality.

Infrared and visible image fusion method based on rolling guidance filter and NSST

The rolling guidance filtering (RGF) has a good characteristic which can smooth texture and preserve the edges, and non-subsampled shearlet transform (NSST) has the features of translation invariance and direction selection based on which a new infrared and visible image fusion method is proposed. Firstly, the rolling guidance filter is used to decompose infrared and visible images into the base and detail layers. Then, the NSST is utilized on the base layer to get the high-frequency coefficients and low-frequency coefficients. The fusion of low-frequency coefficients uses visual saliency map as a fusion rule, and the coefficients of the high-frequency subbands use gradient domain guided filtering (GDGF) and improved Laplacian sum to fuse coefficients. Finally, the fusion of the detail layers combines phase congruency and gradient domain guided filtering as the fusion rule. As a result, the proposed method can not only extract the infrared targets, but also fully preserves the background information of the visible images. Experimental results indicate that our method can achieve a superior performance compared with other fusion methods in both subjective and objective assessments.

Infrared and Visible Image Fusion Method by Using Hybrid Representation Learning

For remote sensing image fusion, infrared and visible images have very different brightness due to their disparate imaging mechanisms, the result of which is that nontarget regions in the infrared image often affect the fusion of details in the visible image. This letter proposes a novel infrared and visible image fusion method basing hybrid representation learning by combining dictionary-learning-based joint sparse representation (JSR) and nonnegative sparse representation (NNSR). In the proposed method, different fusion strategies are adopted, respectively, for the mean image, which represents the primary energy information, and for the deaveraged image, which contains important detail features. Since the deaveraged image contains a large amount of high-frequency details information of the source image, JSR is utilized to sparsely and accurately extract the common and innovation features of the deaveraged image, thus, accurately merging high-frequency details in the deaveraged image. Then, the mean image represents low-frequency and overview features of the source image, according to NNSR, mean image is classified well-directed to different feature regions and then fused, respectively. Such proposed method, on the one hand, can eliminate the impact on fusion result suffering from very different brightness causing by different imaging mechanism between infrared and visible image; on the other hand, it can improve the readability and accuracy of the result fusion image. Experimental result shows that, compared with the classical and state-of-the-art fusion methods, the proposed method not only can accurately integrate the infrared target but also has rich background details of the visible image, and the fusion effect is superior.

A New Deep Learning Based Multi-Spectral Image Fusion Method.

In this paper, we present a new effective infrared (IR) and visible (VIS) image fusion method by using a deep neural network. In our method, a Siamese convolutional neural network (CNN) is applied to automatically generate a weight map which represents the saliency of each pixel for a pair of source images. A CNN plays a role in automatic encoding an image into a feature domain for classification. By applying the proposed method, the key problems in image fusion, which are the activity level measurement and fusion rule design, can be figured out in one shot. The fusion is carried out through the multi-scale image decomposition based on wavelet transform, and the reconstruction result is more perceptual to a human visual system. In addition, the visual qualitative effectiveness of the proposed fusion method is evaluated by comparing pedestrian detection results with other methods, by using the YOLOv3 object detector using a public benchmark dataset. The experimental results show that our proposed method showed competitive results in terms of both quantitative assessment and visual quality.

Terahertz / Visible Dual-band Image Fusion Based on Hybrid Principal Component Analysis

This study puts forward a kind of passive terahertz image and visible image fusion method. Our method takes two steps of image fusion. First step involves a pre-processing of the passive terahertz image and image fusion using principal component analysis. Registered terahertz image is segmented and pseudo-color encoded. Taking the characteristics of terahertz image and visible image into account, principal component analysis is used to fuse the dualband images. HIS method is used for the second step of image fusion. The experiment results show that this algorithm can effectively fuse passive terahertz and visible image and is helpful to detect and locate dangerous objects concealed under the clothes of the target subjects to strengthen the practicability of terahertz imaging system for security screening.