Detail Layer Research Articles

Infrared polarization and intensity image fusion method based on multi-decomposition LatLRR

With the development of infrared imaging technology, the study of infrared polarization and infrared intensity image fusion has great potential for application. In this paper, we propose a fusion method based on multi-decomposition latent low-rank representation (LatLRR) and combine it with dual- simplified pulse coupled neural network (D-SPCNN) for the improvement of detail layer fusion. First, the source images are decomposed into the low-rank part and saliency part using LatLRR, and the weight map of the low-rank part is calculated as the final base layer using the luminance weighting strategy. Then, the saliency part is decomposed by LatLRR again to get the detail layers after multiple decompositions, and the saliency part is used as the input of D-SPCNN to output the activated detail weight map. After that, the final detail layer is obtained by summing the results of the multiple times detail weighting strategy. Finally, the detail layer and the base layer are combined and reconstructed to get the fused image. Our method has an excellent effect of improving brightness, while retaining sufficient image details. The experimental results show that our proposed method obtains better performance in both objective evaluation and subjective metrics when compared with other methods.

DSG-Fusion: Infrared and visible image fusion via generative adversarial networks and guided filter

The goal of infrared and visible image fusion is to generate an informative image which preserves texture details and infrared targets. Most generative adversarial network (GAN) based methods take a concatenation of two source images as the input, hence the extracted feature is insufficient for preserving background and detail information. To this end, we propose a novel GAN based fusion framework, termed as double-stream guided filter network (DSG-Fusion). Given the diverse modalities of infrared and visible images, the generator of DSG network extracts features of two images through two independent data flows. To address the problem of extracting representative background information and force the DSG network focus on details, we integrate guided filter into the generator to obtain base and detail layers of source images. The base layers are concatenated with the corresponding source images to extract deeper features, while detail layers participate in the fusion procedure directly. Thus, DSG-Fusion can extract texture and intensity information sufficiently, and more background and detail information are preserved. Furthermore, a DSG loss consisting of intensity and structural similarity (SSIM) is designed to impel the network to generate a better fused image. Extensive experimental results show that DSG-Fusion achieves better performance comparing with five representative methods.

Laplacian face blending

AbstractDesigning realistic tridimensional facial models is a challenging task, not only due to the effort and artistic abilities required but also because human visual perception is very tuned to the processing of facial features. For this reason, rather than creating face models from scratch, artists usually start from a scanned model of a real person. In this work, we present a novel method for blending human faces in order to create a new one. In a nutshell, our proposal uses Laplacian smoothing to segregate layers of details from one or more faces, which are then integrated into a base face with the help of an interactive and visual editor. In particular, our method supports blending multiple faces and multiple sub‐regions in those faces. Since our approach is intuitive and relatively easy to implement, it can be integrated into artistic pipelines aiming at designing human face models from preexisting ones.

Infrared and visible image fusion based on visibility enhancement and hybrid multiscale decomposition

Infrared and visible image fusion technology aims to integrate the heat source information of infrared image into the visible image to generate a more informative image. Many fusion methods proposed in recent years have problems such as loss of detailed information and low contrast. In this paper, we propose a novel infrared and visible image fusion method based on visibility enhancement and hybrid multiscale decomposition. Firstly, we propose an effective image pre-processing method to increase the details and improve the quality of the source image. Then, the pre-processed images are decomposed by ℓ1−ℓ0 decomposition model to obtain the base and detail layers. Thirdly, for the base layer fusion part, we propose a meaningful method based on the weight of the visual saliency illumination map (VSIM), which not only preserves the contrast information but also does it guarantee the overall structure of the fusion result. For the detail layer fusion part, we take the advantage of convolutional neural network (CNN) to obtain the decision map of the fused detail layer. Next, we employ Laplacian and Gaussian pyramids to decompose the detail layers and decision map, respectively and then fuse them by a synthetic detail fusion strategy. Finally, we reconstruct the base and detail layers to generate the final fusion result. Experiment results demonstrate that our results are more in line with the human visual system (HVS) and outperform some state-of-the-art methods on quantitative metrics.

Detailed analyses of microstructure of photoreceptor layer at different severities of occult macular dystrophy by ultrahigh-resolution SD-OCT

PurposeTo analyze the microstructures of the photoreceptor layer in detail in eyes with occult macular dystrophy (OMD, Miyake's disease) by ultrahigh-resolution spectral-domain optical coherence tomography (UHR-SD-OCT). ObservationsTwenty-eight normal subjects and 5 patients with OMD of different severities were studied. Cross-sectional images through the fovea were recorded with a UHR-SD-OCT system with a depth resolution of <2.0 μm. In patients with OMD, the UHR-SD-OCT images revealed abnormal photoreceptor microstructures which were not detected in the conventional SD-OCT images. The UHR-SD-OCT images showed that the interdigitation zone (IZ) was not present and the outer segments were hyperreflective with hyperreflective dots (HRDs) aligned like string of pearls during the earlier stages. There was a disruption of the ellipsoid zone (EZ) which appeared as clusters of larger HRDs, and these HRDs became less apparent with increasing time. The outer segments became hyporeflective and rod IZ became apparent with longer duration of the disease process. Conclusions and ImportanceThe UHR-SD-OCT images show detailed characteristics of the photoreceptor microstructures of different severities during the progression of OMD. These detailed observations will help in understanding the mechanisms involved in the retinal pathology and should provide important information for their treatments.

Lung CT image enhancement based on total variational frame and wavelet transform

AbstractBenefitting from the development of computer vision, computed tomography (CT) images have been used for assisting doctor's clinical diagnosis and improving the diagnostic efficiency. However, there exist some issues in medical images, such as low contrast, obscure detail, and complex noise due to the restriction of the system and the equipment in the process of imaging, medical images. To resolve these issues, a novel lung CT image enhancement method based on total variational framework combined with wavelet transform is proposed. Firstly, low‐frequency structure layer with low contrast and high‐frequency details layer with complex noise signals are acquired by decomposing the original image using total variational framework. Then, through the analysis of the histogram distribution characteristics of CT image, structure layer requires contrast enhancement; at the same time, the detail layer performs wavelet transform adaptive threshold denoising to remove noise. Finally, weight fusion of processed structure layer and details layer is performed to obtain the final fusion enhancement CT images. Experimental results show that the proposed method can enhance the contrast of clinical lung CT images, improve the clarity of details, and effectively suppress artifacts and complex noises. Contrast and sharpness—objective indicators—prove the proposed method's advantages. Subjectively, the proposed method performs superior over other existing CT enhancement methods, which achieves a better visual recognition.

Infrared and visible image fusion based on visibility enhancement and norm optimization low-rank representation

Infrared and visible image fusion aims to fuse infrared targets and visible details in a composite scene. Recently, many fusion methods have been proposed, but most cannot balance characteristics of infrared and visible images well. We propose an infrared and visible image fusion method. First, to enhance the quality of the infrared and visible images to assist the subsequent prefusion and decomposition, a preprocessing method based on singular value decomposition combined with high dynamic range compression and guided filter (GF) is proposed. Then the preprocessing results are fused to produce the prefusion result by multiscale structural image decomposition-based fusion method. Next, the prefusion result is decomposed by the multilevel image decomposition method based on latent low-rank representation (MDLatLRR). Then the source images are decomposed by MDLatLRR. For the base layer, we combine the visual saliency map with GF to obtain the final fused base layer. For the detail layers, we propose a nonlinear function to highlight the infrared target and use weighted least square to optimize the details. We also adopt a structural similarity-based weight map via L2-norm to improve the fusion performance. The experimental results demonstrate that the proposed method outperforms some state-of-the-art methods both qualitatively and quantitatively.

An Improved Infrared and Visible Image Fusion Using an Adaptive Contrast Enhancement Method and Deep Learning Network with Transfer Learning

Deep learning (DL) has achieved significant attention in the field of infrared (IR) and visible (VI) image fusion, and several attempts have been made to enhance the quality of the final fused image. It produces better results than conventional methods; however, the captured image cannot acquire useful information due to environments with poor lighting, fog, dense smoke, haze, and the noise generated by sensors. This paper proposes an adaptive fuzzy-based preprocessing enhancement method that automatically enhances the contrast of images with adaptive parameter calculation. The enhanced images are then decomposed into base and detail layers by anisotropic diffusion-based edge-preserving filters that remove noise while smoothing the edges. The detailed parts are fed into four convolutional layers of the VGG-19 network through transfer learning to extract features maps. These features maps are fused by multiple fusion strategies to obtain the final fused detailed layer. The base parts are fused by the PCA method to preserve the energy information. Experimental results reveal that our proposed method achieves state-of-the-art performance compared with existing fusion methods in a subjective evaluation through the visual experience of experts and statistical tests. Moreover, the objective assessment parameters are conducted by various parameters (FMI, SSIMa, API, EN, QFAB, and NFAB) which were used in the comparison method. The proposed method achieves 0.2651 to 0.3951, 0.5827 to 0.8469, 56.3710 to 71.9081, 4.0117 to 7.9907, and 0.6538 to 0.8727 gain for FMI, SSIMa, API, EN, and QFAB, respectively. At the same time, the proposed method has more noise reduction (0.3049 to 0.0021) that further justifies the efficacy of the proposed method than conventional methods.

Lossless Compression of Hyperspectral Imagery by Assimilating Decorrelation and Pre-processing with Efficient Displaying Using Multiscale HDR Approach

The instinctive essence of the Hyperspectral imagery cube is its immense information having both spatial and spectral correlation. In the interest of reducing the storage and bandwidth requisites, an effective lossless Hyperspectral compression system is proposed. The imagery is enforced to preprocessing stage preceding decorrelation. Preprocessing stage comprises band normalization and band ordering techniques. A technique named Greedy heap sorting is addressed to sort the bands. The proposed plan accords with a Compression ratio (CR) of 8.12 and bits per pixel (bpp) of 1.67. The performance of the system is comparable to earlier algorithms for lossless Hyperspectral image compression concerning compression ratio and bpp. An experiment conducted on AVIRIS images substantiates that the methodology presented surpasses the IP3-OBPS-BPS method by a percentage increase of 116.44 in CR and a percentage decrease of 58.49 in bpp. The multiscale High Dynamic Range (HDR) approach is used to project Hyperspectral images on devices with Low Dynamic Range (LDR) devices. The Bilateral filter is used for the decomposition of the image into multiple base layers and detail layer. The PSNR obtained is 37.4911 for the compressed HDR image, which signifies the better quality of the reconstructed image with a 4.8% compression ratio.

Two-Scale Multimodal Medical Image Fusion Based on Structure Preservation

Medical image fusion has an indispensable value in the medical field. Taking advantage of structure-preserving filter and deep learning, a structure preservation-based two-scale multimodal medical image fusion algorithm is proposed. First, we used a two-scale decomposition method to decompose source images into base layer components and detail layer components. Second, we adopted a fusion method based on the iterative joint bilateral filter to fuse the base layer components. Third, a convolutional neural network and local similarity of images are used to fuse the components of the detail layer. At the last, the final fused result is got by using two-scale image reconstruction. The contrast experiments display that our algorithm has better fusion results than the state-of-the-art medical image fusion algorithms.

Identification of chromatin loops from Hi-C interaction matrices by CTCF-CTCF topology classification.

Genome-wide profiling of long-range interactions has revealed that the CCCTC-Binding factor (CTCF) often anchors chromatin loops and is enriched at boundaries of the so-called Topologically Associating Domains, which suggests that CTCF is essential in the 3D organization of chromatin. However, the systematic topological classification of pairwise CTCF–CTCF interactions has not been yet explored. Here, we developed a computational pipeline able to classify all CTCF–CTCF pairs according to their chromatin interactions from Hi-C experiments. The interaction profiles of all CTCF–CTCF pairs were further structurally clustered using self-organizing feature maps and their functionality characterized by their epigenetic states. The resulting clusters were then input to a convolutional neural network aiming at the de novo detecting chromatin loops from Hi-C interaction matrices. Our new method, called LOOPbit, is able to automatically detect significant interactions with a higher proportion of enhancer-promoter loops compared to other callers. Our highly specific loop caller adds a new layer of detail to the link between chromatin structure and function.

TSE_Fuse: Two stage enhancement method using attention mechanism and feature-linking model for infrared and visible image fusion

Infrared and visible image fusion is a common solution that joints multispectral modals to address weaknesses of observation in a complex environment to achieve all-weather and full-time perception. The core concept of pixel-level fusion is to obtain complementary features and remove redundant factors. The existing works based on enhancement aim to improve the details of raw information, but most of them easily suffer from noise while in the enhancement process since they ignore well-designed feature selection. To further suppress noise and bridge infrared and visible latent cues, we designed a two-stage enhancement (TSE) framework by using an attention mechanism and feature-linking model (FLM). First of all, we construct a novel decomposition scheme that combines structure adaptive total-variational (SATV) and l1 sparsity term, which is called SATV-l1, to extract two scale detail layers and base layer. Furthermore, the l1 sparsity term is exploited on the base layer to compute its piecewise smoothness property owing to the initial base layer containing texture features. In the final stage, we introduce the TSE to fulfill the detail layer and reconstruct the fused image. Extensive experimental validations on public datasets are performed, demonstrating the robustness and effectiveness of our approach.

Infrared and visible image fusion via rolling guidance filter and weight map

The main task of image fusion is to extract the advantage information from the source images. Multiscale transform (MST) is commonly used in image fusion, but it is difficult to integrate detail information and structural information from source images. Therefore, it is important to integrate the information in each image patch group. Different from MST, edge preserving technology obtains more edge information than MST by extracting spatial information. Based on this superior feature, we propose an infrared and visible image fusion method via rolling guidance filter (RGF) and weight map. First, source images are decomposed into multiscale detail layers and structural base layers by RGF; second, to avoid the loss of local information, the modified Laplacian energy weight matrix is used as detail weight maps to merging the detail layers; then to reduce the loss of contours of edge objects, a structural weight maps are used to merging the base layers; finally, these two different weight maps are applied to reconstruct the fused image. Simulation results indicate that our fusion approach performs better in merging detail information and structural information than some classic MST fusion algorithms and the latest fusion methods.

Towards an Improved Large-Scale Gridded Population Dataset: A Pan-European Study on the Integration of 3D Settlement Data into Population Modelling

Large-scale gridded population datasets available at the global or continental scale have become an important source of information in applications related to sustainable development. In recent years, the emergence of new population models has leveraged the inclusion of more accurate and spatially detailed proxy layers describing the built-up environment (e.g., built-area and building footprint datasets), enhancing the quality, accuracy and spatial resolution of existing products. However, due to the consistent lack of vertical and functional information on the built-up environment, large-scale gridded population datasets that rely on existing built-up land proxies still report large errors of under- and overestimation, especially in areas with predominantly high-rise buildings or industrial/commercial areas, respectively. This research investigates, for the first time, the potential contributions of the new World Settlement Footprint—3D (WSF3D) dataset in the field of large-scale population modelling. First, we combined a Random Forest classifier with spatial metrics derived from the WSF3D to predict the industrial versus non-industrial use of settlement pixels at the Pan-European scale. We then examined the effects of including volume and settlement use information into frameworks of dasymetric population modelling. We found that the proposed classification method can predict industrial and non-industrial areas with overall accuracies and a kappa-coefficient of ~84% and 0.68, respectively. Additionally, we found that both, integrating volume and settlement use information considerably increased the accuracy of population estimates between 10% and 30% over commonly employed models (e.g., based on a binary settlement mask as input), mainly by eliminating systematic large overestimations in industrial/commercial areas. While the proposed method shows strong promise for overcoming some of the main limitations in large-scale population modelling, future research should focus on improving the quality of the WFS3D dataset and the classification method alike, to avoid the false detection of built-up settlements and to reduce misclassification errors of industrial and high-rise buildings.

The fusion of infrared and visible images via decomposition-based structure transfer and local saliency detection

Most of existing image fusion methods neglect the differences between multi-source images and extract characteristics without any discrimination, which negatively impacts the fused result, particularly in infrared (IR) and visible image fusion. This paper proposes a two-scale fusion method using total variation (TV) model with structure transfer and local saliency detection. Firstly, we use rolling guidance filter (RGF) to perform two-scale decomposition on the input images. Secondly, we combine the base layers with TV minimization based on structure transfer. Then we construct the weight map with local saliency detection and guide filter, and combine the detail layers by weight average. At last, the fused result is reconstructed with the fused base layer and the fused detail layer. The fusion result retains more target location and appearance information, which is beneficial to subsequent processing. The comparisons with 7 other methods reveal that our method achieves a superior performance.

The effect of hierarchical structure of the surface dust layer on the modelling of comet gas production

ABSTRACT A hierarchical model of microstructure of cometary dust seems to accurately capture the morphological complexity of these particles as observed by the Rosetta mission to the 67P/Churyumov–Gerasimenko. The main aim of this work is to investigate how uncertainties in the knowledge of the microstructure of the surface layer affect our estimates of gas production. New models that incorporate scale of inhomogenieties in a sound theoretical framework are used for hierarchical dust layers. The studied layers are constructed in two steps: first we design ballistic aggregates as the building units and then, using these porous blocks, assemble the layers constrained by the known porosity range. The mean pore size and permeability are studied. Modelling is performed for various values of porosity, grain size, and layer thickness. The simulation results are embedded in the thermal model, explicitly including a radiation thermal conductivity and a resistance of the dust layer for the gas flow. It is shown that the average pore size is satisfactorily approximated by a linear function of the ratio of the effective porosity of the layer to the degree of filling. Simple fittings were obtained for the layer permeability. Our results indicate that in the expected range of nucleus porosity, the gas production rate is weakly dependent on the detailed layer microstructure, and appropriate effective values of homogeneous dust layers can be used to evaluate the gas activity. We also note that adding complex elements into the model yields unavoidable statistical uncertainties within several tens of per cent.

Infrared and Visible Image Fusion Based on Dual-Kernel Side Window Filtering and S-Shaped Curve Transformation

The fusion of infrared and visible images combines the advantages of thermal radiation information in infrared images and texture information in visible images. To preserve salient targets and obtain better fusion results, this article proposes a novel infrared and visible image fusion method based on dual-kernel side window filtering and detail optimization with S-shaped curve transformation. First, a dual-kernel side window box filter (DSWBF) with adaptive filter kernel size is designed to extract the base and detail layers of the source images. Then, a saliency-based fusion rule is proposed for the base layers to highlight the salient regions of the infrared image. Next, a detail optimization module is constructed based on an S-shaped curve transformation and guided filter to optimize the detail layer of the infrared image. The optimized detail layer is then merged with the detail layer of the visible image to obtain the fused detail layer. Finally, the fusion image is created by reconstructing the fused base and detail layers. Experimental results on the two public datasets indicate that compared with the state-of-the-art methods, the proposed method can obtain better fusion performance in terms of subjective and objective evaluations.

A New Image Fusion Method for ERT-EMT Dual-Modality System Used in Gas–Liquid–Solid Three-Phase Fluidized Bed

The electrical resistance tomography and electromagnetic tomography (ERT-EMT) dual-modality system can be used to reconstruct the image of cross-sectional three-phase distribution for gas-liquid-solid fluidized bed by fusing the reconstructed images of ERT and EMT. However, due to the small size of solid particles, the complexity of fluidization process, and the inconsistency of media distribution recognized by ERT and EMT, the existing fusion algorithms of dual-modality electrical tomography, which either divide the media distribution into binary distribution, or use the consistency of the media distribution to be recognized to improve the image quality, are not applicable. Aiming at the problem, a new image fusion method based on guided image filtering (GIF) and image statistics is proposed in this paper. Firstly, the GIF method is used to decompose the images of ERT and EMT into base layers containing large scale variations in intensity and detail layers with small scale details. Then, the weights corresponding to the base layers and detail layers are calculated with an image statistics method and GIF method, respectively. Through the process, the useful information in the reconstructed images of ERT and EMT are fully utilized. The simulation and experimental results demonstrate that the proposed fusion method can accurately reconstruct the gas-liquid-solid distribution.

Adaptive Infrared and Visible Image Fusion Based on Visual Saliency and Hierarchical Bayesian

An adaptive infrared and visible image fusion method based on visual saliency and hierarchical Bayesian (AVSHB) which preserves the highest similarity between fused images and source images is proposed in this paper. Firstly, an effective salient edge preserving filter named SEPF is developed to decompose each source image into a base layer and a detail layer. A ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> norm gradient minimization is firstly derived and then embedded into a two-scale acceleration scheme in SEPF. Benefiting from the SEPF, the edges of salient regions can be preserved without distortion. Then, an adaptive fusion scheme is proposed which fully considers the characteristics of each layer. More concretely, we design a two-scale fusion strategy based on a visual saliency map (VSM) for the base layers, and a hierarchical Bayesian fusion model is derived for the detail layers. The experimental results on the TNO and RoadScene datasets and Nato_camp image sequence demonstrate that AVSHB favorably outperforms 16 related state-of-the-art fusion methods qualitatively and quantitatively. AVSHB can generate improved fusion results with sufficiently retaining salient targets and rich details from the source images.

Luminance Learning for Remotely Sensed Image Enhancement Guided by Weighted Least Squares

Low/high or uneven luminance results in low contrast of remotely sensed images (RSIs), which makes it challenging to analyze their contents. In order to improve the contrast and preserving fine weak details of RSIs, this letter proposes a novel enhancement framework to correct luminance guided by weighted least squares (WLS), including the following key parts. First, an image is separated into a base layer and a detail layer by employing the WLS. Then, a learning network is proposed to correct luminance for the base layer enhancement. Next, an enhancement operator for improving the detail layer is computed by using the original image and the enhanced base layer. Finally, the output image is obtained with a fusion of the enhanced base and detail components. Both quantitatively and qualitatively experimental results verify that the proposed method performs better than the state of the arts in contrast improvement and detail preservation.