Visible Color Image Research Articles

Cross-domain colorization of unpaired infrared images through contrastive learning guided by color feature selection attention

In challenging lighting conditions, infrared detectors have become vital tools for enhancing visual perception, overcoming the limitations of visible cameras. However, inherent imaging principles and manufacturing constraints confine infrared imaging systems to grayscale, significantly impacting their utility. In comparison to visible imagery, infrared images lack detailed semantic information, color representation, and suffer from reduced contrast. While existing infrared image colorization techniques have made significant progress in improving color quality, challenges such as erroneous semantic color prediction and blurred depiction of fine details persist. Acquiring paired color images corresponding to real-world infrared scenarios poses substantial difficulties, exacerbating challenges in cross-domain colorization of infrared images. To address these critical issues, this paper introduces an innovative approach utilizing contrastive learning for unsupervised cross-domain mapping between unpaired infrared and visible color images. Additionally, we introduce a color feature selection attention module guiding rational infrared image coloring. The proposed method employs the Residual Fusion Attention Network (RFANet) as a generator, enhancing the encoder’s ability to represent color and structural features. Furthermore, to ensure structural content consistency and enhance overall color style matching accuracy, we design a comprehensive joint global loss function integrating both detailed content and color style. Experimental evaluations on publicly available datasets demonstrate the superior performance of the proposed unsupervised cross-domain colorization method for infrared images compared to previous approaches.

EV-Fusion: A Novel Infrared and Low-Light Color Visible Image Fusion Network Integrating Unsupervised Visible Image Enhancement

EV-Fusion: A Novel Infrared and Low-Light Color Visible Image Fusion Network Integrating Unsupervised Visible Image Enhancement

Nighttime Thermal Infrared Image Translation Integrating Visible Images

Nighttime Thermal InfraRed (NTIR) image colorization, also known as the translation of NTIR images into Daytime Color Visible (DCV) images, can facilitate human and intelligent system perception of nighttime scenes under weak lighting conditions. End-to-end neural networks have been used to learn the mapping relationship between temperature and color domains, and translate NTIR images with one channel into DCV images with three channels. However, this mapping relationship is an ill-posed problem with multiple solutions without constraints, resulting in blurred edges, color disorder, and semantic errors. To solve this problem, an NTIR2DCV method that includes two steps is proposed: firstly, fuse Nighttime Color Visible (NCV) images with NTIR images based on an Illumination-Aware, Multilevel Decomposition Latent Low-Rank Representation (IA-MDLatLRR) method, which considers the differences in illumination conditions during image fusion and adjusts the fusion strategy of MDLatLRR accordingly to suppress the adverse effects of nighttime lights; secondly, translate the Nighttime Fused (NF) image to DCV image based on HyperDimensional Computing Generative Adversarial Network (HDC-GAN), which ensures feature-level semantic consistency between the source image (NF image) and the translated image (DCV image) without creating semantic label maps. Extensive comparative experiments and the evaluation metrics values show that the proposed algorithms perform better than other State-Of-The-Art (SOTA) image fusion and translation methods, such as FID and KID, which decreased by 14.1 and 18.9, respectively.

MUGAN: Thermal Infrared Image Colorization Using Mixed-Skipping UNet and Generative Adversarial Network

Common cameras cannot capture high quality image in the night or extreme weather conditions that without enough light, while thermal infrared(TIR) cameras are not limited in this situation. Hence, TIR imaging technique is widely used in military, surveillance, nighttime traffic and other scenarios. However, TIR images are monochromatic, and the majority of details of such images are lost, which are difficult for human or computer system to analyze. Translating TIR images into visible images is beneficial to subsequent observation or further processing. Though there are some advances to realize the transformation from TIR images to color visible images, edge distortion and semantic confusion remain to be solved. Therefore, we propose a Mixed-Skipping UNet(MS-UNet) based image colorization model joint Generative Adversarial Network, which is denoted by MUGAN. Firstly, the dense skip connections of UNet++ and full-scale skip connections of UNet 3+ are combined to form the MS-UNet, which is regarded as the generator. In addition, we design a feature extraction module in MS-UNet to effectively capture the multi-scale features in source image. Then, a novel attention mechanism module is designed for decoding stage which can help decoder of MS-UNet to focus on the important information. Moreover, we explore the effect of different loss functions in TIR image colorization task, and the loss function with excellent performance is selected to further optimize the training process of the model. Extensive experiments prove the superiority of our method in the task of TIR image colorization. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/HangyingLiao/MUGAN</uri> .

An Unpaired Thermal Infrared Image Translation Method Using GMA-CycleGAN

Automatically translating chromaticity-free thermal infrared (TIR) images into realistic color visible (CV) images is of great significance for autonomous vehicles, emergency rescue, robot navigation, nighttime video surveillance, and many other fields. Most recent designs use end-to-end neural networks to translate TIR directly to CV; however, compared to these networks, TIR has low contrast and an unclear texture for CV translation. Thus, directly translating the TIR temperature value of only one channel to the RGB color value of three channels without adding additional constraints or semantic information does not handle the one-to-three mapping problem between different domains in a good way, causing the translated CV images not only to have blurred edges but also color confusion. As for the methodology of the work, considering that in the translation from TIR to CV the most important process is to map information from the temperature domain into the color domain, an improved CycleGAN (GMA-CycleGAN) is proposed in this work in order to translate TIR images to grayscale visible (GV) images. Although the two domains have different properties, the numerical mapping is one-to-one, which reduces the color confusion caused by one-to-three mapping when translating TIR to CV. Then, a GV-CV translation network is applied to obtain CV images. Since the process of decomposing GV images into CV images is carried out in the same domain, edge blurring can be avoided. To enhance the boundary gradient between the object (pedestrian and vehicle) and the background, a mask attention module based on the TIR temperature mask and the CV semantic mask is designed without increasing the network parameters, and it is added to the feature encoding and decoding convolution layers of the CycleGAN generator. Moreover, a perceptual loss term is applied to the original CycleGAN loss function to bring the translated images closer to the real images regarding the space feature. In order to verify the effectiveness of the proposed method, the FLIR dataset is used for experiments, and the obtained results show that, compared to the state-of-the-art model, the subjective quality of the translated CV images obtained by the proposed method is better, as the objective evaluation metric FID (Fréchet inception distance) is reduced by 2.42 and the PSNR (peak signal-to-noise ratio) is improved by 1.43.

MrFDDGAN: Multireceptive Field Feature Transfer and Dual Discriminator-Driven Generative Adversarial Network for Infrared and Color Visible Image Fusion

Most of the previous infrared and visible image fusion methods based on deep learning were on the basis of grayscale images and used the single convolution kernel receptive field to extract deep features, which would inevitably cause information loss in the process of feature transfer. Accordingly, this paper proposes a new generative adversarial network based on multi-receptive field feature transfer and dual discriminators, which is named MrFDDGAN. It is applied to infrared and color visible image fusion. Firstly, three different receptive fields are used to extract multi-scale and multi-level deep features of multi-modal images on three channels, so as to obtain significant features of source images from multi-view and multi-perspective in a more comprehensive way. Secondly, a feature interaction module is introduced in the encoder to realize the information interaction and information pre-fusion among the three feature channels. Thirdly, the multi-level features fused in the encoder are cascaded with the deep features in the decoder to enhance feature transfer and feature reuse. Finally, this experiment adopts a dual-discriminator adversarial network structure to keep the balance of infrared image intensity and visible image texture preserved by fusion results. And qualitative and quantitative experimental analyses are carried out on two public datasets of grayscale infrared as well as visible images and one dataset of infrared and color visible images. The experimental results prove that the proposed MrFDDGAN algorithm has a better subjective visual effect and objective performance than the existing state-of-the-art fusion methods.

Image Separation With Side Information: A Connected Auto-Encoders Based Approach

X-radiography (X-ray imaging) is a widely used imaging technique in art investigation. It can provide information about the condition of a painting as well as insights into an artist's techniques and working methods, often revealing hidden information invisible to the naked eye. X-radiograpy of double-sided paintings results in a mixed X-ray image and this paper deals with the problem of separating this mixed image. Using the visible color images (RGB images) from each side of the painting, we propose a new Neural Network architecture, based upon 'connected' auto-encoders, designed to separate the mixed X-ray image into two simulated X-ray images corresponding to each side. This connected auto-encoders architecture is such that the encoders are based on convolutional learned iterative shrinkage thresholding algorithms (CLISTA) designed using algorithm unrolling techniques, whereas the decoders consist of simple linear convolutional layers; the encoders extract sparse codes from the visible image of the front and rear paintings and mixed X-ray image, whereas the decoders reproduce both the original RGB images and the mixed X-ray image. The learning algorithm operates in a totally self-supervised fashion without requiring a sample set that contains both the mixed X-ray images and the separated ones. The methodology was tested on images from the double-sided wing panels of the Ghent Altarpiece, painted in 1432 by the brothers Hubert and Jan van Eyck. These tests show that the proposed approach outperforms other state-of-the-art X-ray image separation methods for art investigation applications.

Detection of Small Moving Targets in Cluttered Infra-red Imagery

Deep convolutional neural networks can achieve remarkable results for detecting and recognizing large and medium sized objects in visible band color images. However, the ability to detect small objects has yet to achieve the same level performance. Our focus is on applications that require the accurate detection and localization of small moving objects that are distant from the sensor. State of the art object detection and classification networks (such YOLOv3 &#x005B;1&#x005D; and Mask R-CNN &#x005B;2&#x005D;) are not well suited for finding small moving objects in infra-red imagery, nor do they handle temporal information in video sequences. To overcome the limitations of these methods, we propose the Moving Target Indicator Network (MTINet), a new model specifically designed for detecting small moving targets in clutter. Several versions of the MTINet are presented with different refinements (such as spatial pyramid blocks and improved attention mechanism) to increase the probability of detection and reduce false alarm rates. The MTINet is trained to maximize the <i>target to clutter ratio</i> (TCR) metric, which represents the first use of the TCR loss function for detecting moving objects. The further challenge the MTINet, we also evaluated its performance with simulated sensor movement mimicking effects of image stabilization processes. Finally, we also propose a modification of the Reed-Xiaoli detector &#x005B;3&#x005D; (originally developed for anomaly detection in hyperspectral data) to enable it to detect temporal anomalies caused by moving objects, and refer to it as the temporal-ARX (t-ARX) algorithm. We find that the t-ARX algorithm can achieve better probability of detection at lower false alarm rates, while the MTINet is superior at finding difficult targets at higher false alarm rates. We then show that the combination of the two algorithms achieves the best overall performance.

Deep learning-based single image face depth data enhancement

Face recognition can benefit from the utilization of depth data captured using low-cost cameras, in particular for presentation attack detection purposes. Depth video output from these capture devices can however contain defects such as holes or general depth inaccuracies. This work proposes a deep learning face depth enhancement method in this context of facial biometrics, which adds a security aspect to the topic. U-Net-like architectures are utilized, and the networks are compared against hand-crafted enhancer types, as well as a similar depth enhancer network from related work trained for an adjacent application scenario. All tested enhancer types exclusively use depth data as input, which differs from methods that enhance depth based on additional input data such as visible light color images. Synthetic face depth ground truth images and degraded forms thereof are created with help of PRNet, to train multiple deep learning enhancer models with different network sizes and training configurations. Evaluations are carried out on the synthetic data, on Kinect v1 images from the KinectFaceDB, and on in-house RealSense D435 images. These evaluations include an assessment of the falsification for occluded face depth input, which is relevant to biometric security. The proposed deep learning enhancers yield noticeably better results than the tested preexisting enhancers, without overly falsifying depth data when non-face input is provided, and are shown to reduce the error of a simple landmark-based PAD method.

Traffic thermal infrared texture generation based on siamese semantic CycleGAN

Thermal infrared texture generation is a promising infrared imaging framework for various applications. A novel thermal infrared texture generation algorithm, based on siamese semantic CycleGAN (SS-CycleGAN), is proposed for thermal infrared systems. Different from traditional infrared simulation frameworks, SS-CycleGAN depends on no extra environmental information, such as air temperature, humidity and radiation properties of objects. In other words, visible images could be directly transformed into thermal infrared images like using style transfer algorithms, after traffic scene has been fully understood through training CNN. In this paper, style transfer is firstly introduced for generating thermal textures from color visible images. Siamese semantic loss for visible-infrared transformation is designed and introduced to generate object-oriented thermal infrared textures, while maintaining high definition. Compared to other style transfer algorithms, SS-CycleGAN could generate reasonable thermal infrared textures with clear edge details, in traffic scenes.

Quantitative infrared image analysis of simultaneous upstream and downstream microgravity flame spread over thermally thin cellulose fuel in low speed forced flow

Quantitative image analysis of infrared (IR) measurements are compared to visible color images for simultaneous upstream and downstream flame spread over thermally thin cellulose samples in low speed forced flow in microgravity. These results are a unique set of data that provide a wealth of information to guide future modeling efforts. Test conditions at ambient pressure were conducted in 21–50% O2 by volume (N2 balance) and forced flow velocities from 2 cm/s to 20 cm/s. The tests conditions encompass strictly upstream flame spread to simultaneous upstream and downstream flame spread. Steady upstream flame spread is observed while the downstream flame only begins to spread at the highest flow velocities due to the oxygen shadow cast by the upstream flame (oxygen depleted region downstream of the flame). Blackbody surface temperature gradients for the upstream flame are an order of magnitude larger than the downstream gradients. Upstream preheat lengths decrease with flow while downstream preheat lengths increase. Downstream pyrolysis lengths reach steady state for most cases and increase with convective heating while upstream pyrolysis lengths increase with time. Surface thermocouple (TC) histories were non-dimensionalized to obtain a characteristic surface temperature profile using new scaling analyses. Applying the ellipticity of the leading edge to the scaling reveals the local flow velocity at the leading edge is on the order of diffusive velocities and the flame thermal expansion acts as a significant barrier to divert the flow. A surface energy balance reveals that the peak heat flux for both upstream and downstream flames increases with oxygen concentration and forced flow velocity. The upstream flame heat flux is equally divided between fuel heat up and surface radiation. The downstream heat flux goes almost exclusively to surface radiation. Comparisons of these results to previous applicable research are provided throughout the text and generally agree well.

Deep Learning Thermal Image Translation for Night Vision Perception

Context enhancement is critical for the environmental perception in night vision applications, especially for the dark night situation without sufficient illumination. In this article, we propose a thermal image translation method, which can translate thermal/infrared (IR) images into color visible (VI) images, called IR2VI. The IR2VI consists of two cascaded steps: translation from nighttime thermal IR images to gray-scale visible images (GVI), which is called IR-GVI; and the translation from GVI to color visible images (CVI), which is known as GVI-CVI in this article. For the first step, we develop the Texture-Net, a novel unsupervised image translation neural network based on generative adversarial networks. Texture-Net can learn the intrinsic characteristics from the GVI and integrate them into the IR image. In comparison with the state-of-the-art unsupervised image translation methods, the proposed Texture-Net is able to address some common challenges, e.g., incorrect mapping and lack of fine details, with a structure connection module and a region-of-interest focal loss. For the second step, we investigated the state-of-the-art gray-scale image colorization methods and integrate the deep convolutional neural network into the IR2VI framework. The results of the comprehensive evaluation experiments demonstrate the effectiveness of the proposed IR2VI image translation method. This solution will contribute to the environmental perception and understanding in varied night vision applications.

Infrared-Visible Cross-Modal Person Re-Identification with an X Modality

This paper focuses on the emerging Infrared-Visible cross-modal person re-identification task (IV-ReID), which takes infrared images as input and matches with visible color images. IV-ReID is important yet challenging, as there is a significant gap between the visible and infrared images. To reduce this ‘gap’, we introduce an auxiliary X modality as an assistant and reformulate infrared-visible dual-mode cross-modal learning as an X-Infrared-Visible three-mode learning problem. The X modality restates from RGB channels to a format with which cross-modal learning can be easily performed. With this idea, we propose an X-Infrared-Visible (XIV) ReID cross-modal learning framework. Firstly, the X modality is generated by a lightweight network, which is learnt in a self-supervised manner with the labels inherited from visible images. Secondly, under the XIV framework, cross-modal learning is guided by a carefully designed modality gap constraint, with information exchanged cross the visible, X, and infrared modalities. Extensive experiments are performed on two challenging datasets SYSU-MM01 and RegDB to evaluate the proposed XIV-ReID approach. Experimental results show that our method considerably achieves an absolute gain of over 7% in terms of rank 1 and mAP even compared with the latest state-of-the-art methods.

Fusion Algorithm for Near-Infrared and Color Visible Images Based on Tetrolet Transform

To address the problems of low contrast,loss of detail,and color distortion after near-infrared and color visible image fusion,an algorithm for near-infrared and color visible image fusion based on the Tetrolet transform and pulse coupled neural network(PCNN)is proposed.First,the color visible light source image is transformed into a hue saturation intensity space,where each component is relatively independent,and its brightness component is decomposed into a near-infrared image by Tetrolet decomposition.Subsequently,a fusion rule for expectation maximization likelihood estimation of the potential distribution from a given incomplete dataset is proposed.A self-adaptive PCNN model with a Sobel operator that automatically adjusts the threshold is used as a fusion rule,and the processed high-frequency and low-frequency images are fused by Tetrolet inverse transformation as brightness images.An adaptive stretching method for the saturation component is proposed to solve the problem of image saturation decline.The processed components are mapped back to red–green–blue space to complete the fusion.The proposed algorithm was compared with several efficient fusion algorithms.The experimental results show that the image obtained by this method has clear details and improved color contrast.It has obvious advantages in image saturation,color restoration performance,structural similarity,and contrast.

Infrared and Visible Image Fusion Based on Nonparametric Segmentation

Image fusion is a process of combining an image sequence of the same scene into a single image for better human perception &amp; targeting. The thermal energy reflected from outstanding objects under poor lighting conditions and visible information that yield spatial details needs to be fused for improving the performance of surveillance systems. In this paper, we present a fusion technique that is helpful in surveillance systems for detecting targets when the background and the targets are of the same color. A nonparametric segmentation based weight map computation technique is proposed to extract target details from infrared (IR) imagery. The optimal threshold based on local features is selected automatically for target detection. With this, the extracted salient information of targets is blended to visible image without introducing distortions. The main advantage of the new technique is that it is based on a single-scale binary map (SSBM) fusion approach. The binary weight maps are computed for the fusion of separable IR target with visible imagery. An extension to IR and visible color image fusion is also suggested for target localization. Several simulation results are demonstrated for different data sets to support the validity of the proposed technique

Near-Infrared Fusion via Color Regularization for Haze and Color Distortion Removals

Different from conventional haze removal methods based on a single image, near-infrared imaging can provide two types of multimodal images: one is the near-infrared image and the other is the visible color image. These two images have different characteristics regarding color and visibility. The captured near-infrared image is haze-free, but it is grayscale, whereas the visible color image has colors, but it contains haze. There are serious discrepancies in terms of brightness and image structures between the near-infrared image and the visible color image. Due to this discrepancy, the direct use of the near-infrared image for haze removal causes a color distortion problem during near-infrared fusion. The key objective for the near-infrared fusion is therefore to remove the color distortion as well as the haze. To achieve this objective, this paper presents a new near-infrared fusion model that combines the proposed new color and depth regularizations with the conventional haze degradation model. The proposed color regularization sets the color range of the unknown haze-free image based on the combination of the two colors of the colorized near-infrared image and the captured visible color image. That is, the proposed color regularization can provide color information for the unknown haze-free color image. The new depth regularization enables the consecutively estimated depth maps not to be largely deviated, thereby transferring natural-looking colors and high visibility of the colorized near-infrared image into the preliminary dehazed version of the captured visible color image with color distortion and edge artifacts. Experimental results show that the proposed color and depth regularizations can help remove the color distortion and the haze simultaneously. The effectiveness of the proposed color regularization for the near-infrared fusion is verified by comparing it with other conventional regularizations.

Synthetic aperture radar and color visible images fusion algorithm based on region division strategy

Multisensor image fusion can be used as an advanced technique for image enhancement. A fusion method based on region division strategy is proposed for synthetic aperture radar (SAR) and color visible images. The SAR target feature detection function is first constructed to divide the images into the SAR target region and non-SAR target region. Meanwhile, SAR image despeckling is considered during the generation of this detection function. Then, two different fusion rules are designed for these two regions. More specifically, the fusion rule adopted in the SAR target region is fusing the high-frequency feature within the SAR image, and the fusion rule employed in the non-SAR target region is maintaining the spectral and detail information of the color visible image. Experimental results on several pairs of SAR and color visible images demonstrate the effectiveness of the proposed method. Compared with the conventional image fusion methods, the proposed method provides better results for all evaluation criteria, including spectral distortions, feature similarity, mutual information, and peak signal-to-noise ratio.

Near-infrared fusion via a series of transfers for noise removal

Near-infrared imaging has been considered as a solution for providing high quality photographs in dim lighting conditions. This imaging system captures two types of multimodal images, namely, near-infrared gray image (NGI) and visible color image (VCI). NGI is noise free and grayscale, whereas the VCI has colors and contains noise. Moreover, there are significant edge and brightness discrepancies between NGI and VCI. In order to deal with this problem, a new transfer-based fusion method is proposed for noise removal. Different from conventional fusion approaches, the proposed method conducts a series of transfers, namely, contrast, detail, and color transfers. First, the proposed contrast and detail transfers aim at solving the serious discrepancy problem to create a new noise-free and detail-preserving NGI. Second, the proposed color transfer models the unknown colors from the denoised VCI through a linear transform, and then transfers the natural-looking colors into the newly generated NGI. The experimental results show that the proposed transfer-based fusion method is highly successful in solving the discrepancy problem. The edges and textures are clearly described and the noise is completely removed from the fused images. Most of all, the proposed method is superior to the conventional fusion methods, guided filtering, and state-of-the-art fusion methods based on scale map and layer decomposition.

Near-Infrared Coloring via a Contrast-Preserving Mapping Model.

Near-infrared gray images captured along with corresponding visible color images have recently proven useful for image restoration and classification. This paper introduces a new coloring method to add colors to near-infrared gray images based on a contrast-preserving mapping model. A naive coloring method directly adds the colors from the visible color image to the near-infrared gray image. However, this method results in an unrealistic image because of the discrepancies in the brightness and image structure between the captured near-infrared gray image and the visible color image. To solve the discrepancy problem, first, we present a new contrast-preserving mapping model to create a new near-infrared gray image with a similar appearance in the luminance plane to the visible color image, while preserving the contrast and details of the captured near-infrared gray image. Then, we develop a method to derive realistic colors that can be added to the newly created near-infrared gray image based on the proposed contrast-preserving mapping model. Experimental results show that the proposed new method not only preserves the local contrast and details of the captured near-infrared gray image, but also transfers the realistic colors from the visible color image to the newly created near-infrared gray image. It is also shown that the proposed near-infrared coloring can be used effectively for noise and haze removal, as well as local contrast enhancement.

Multisensor image fusion approach utilizing hybrid pre-enhancement and double nonsubsampled contourlet transform

A multisensor image fusion approach established on the hybrid-domain image enhancement and double nonsubsampled contourlet transform (NSCT) is proposed. The hybrid-domain pre-enhancement algorithm can promote the contrast of the visible color image. Different fusion rules are, respectively, selected and applied to obtain fusion results. The double NSCT framework is introduced to obtain better fusion performance than the general single NSCT framework. Experimental outcomes in fused images and performance results demonstrate that the presented approach is apparently more advantageous.