Subjective Visual Evaluation Research Articles

Multi-scale convolutional neural networks and saliency weight maps for infrared and visible image fusion

Image fusion is the fusion of multiple images from the same scene to produce a more informative image, and infrared and visible image fusion is an important branch of image fusion. To tackle the issues of diminished luminosity in the infrared target, inconspicuous target features, and blurred texture of the fused image after the fusion of infrared and visible images. This paper introduces a novel effective fusion framework that merges multi-scale Convolutional Neural Networks (CNN) with saliency weight maps. First, the method measures the source image features to estimate the initial saliency weight map. Then, the initial weight map is segmented and optimized using a guided filter before being further processed by CNN. Next, a trained Siamese convolutional network is used to solve the two key problems of activity measure and weight assignment. Meanwhile, a multi-layer fusion strategy is designed to effectively retain the luminance of the infrared target and the texture information in the visible background. Finally, adaptive adjustment of the fusion coefficients is achieved by employing saliency. The experimental results show that the method outperforms the state-of-the-art algorithms in terms of both subjective visual quality and objective evaluation effects.

A Multi-Scale Recursive Attention Feature Fusion Network for Image Super-Resolution Reconstruction Algorithm

In recent years, deep convolutional neural networks (CNNs) have made significant progress in single-image super-resolution (SISR) tasks. Despite their good performance, the single-image super-resolution task remains a challenging one due to problems with underutilization of feature information and loss of feature details. In this paper, a multi-scale recursive attention feature fusion network (MSRAFFN) is proposed for this purpose. The network consists of three parts: a shallow feature extraction module, a multi-scale recursive attention feature fusion module, and a reconstruction module. The shallow features of the image are first extracted by the shallow feature extraction module. Then, the feature information at different scales is extracted by the multi-scale recursive attention feature fusion network block (MSRAFFB) to enhance the channel features of the network through the attention mechanism and fully fuse the feature information at different scales in order to improve the network’s performance. In addition, the image features at different levels are integrated through cross-layer connections using residual connections. Finally, in the reconstruction module, the upsampling capability of the deconvolution module is used to enlarge the image while extracting its high-frequency information in order to obtain a sharper high-resolution image and achieve a better visual effect. Through extensive experiments on a benchmark dataset, the proposed network model is shown to have better performance than other models in terms of both subjective visual effects and objective evaluation metrics.

Assessing Excessive Keratinization in Acral Areas through Dermatoscopy with Cross-Polarization and Parallel-Polarization: A Dermatoscopic Keratinization Scale.

Excessive epidermal hyperkeratosis in acral areas is a common occurrence in dermatology practice, with a notable prevalence of approximately 65% in the elderly, especially in plantar lesions. Hyperkeratosis, characterized by thickening of the stratum corneum, can have various causes, including chronic physical or chemical factors, genetic predispositions, immunological disorders, and pharmaceutical compounds. This condition can significantly impact mobility, increase the risk of falls, and reduce the overall quality of life, particularly in older individuals. Management often involves creams containing urea to soften hyperkeratotic areas. Currently, subjective visual evaluation is the gold standard for assessing hyperkeratosis severity, lacking precision and consistency. Therefore, our research group proposes a novel 6-point keratinization scale based on dermatoscopy with cross-polarization and parallel-polarization techniques. This scale provides a structured framework for objective assessment, aiding in treatment selection, duration determination, and monitoring disease progression. Its clinical utility extends to various dermatological conditions involving hyperkeratosis, making it a valuable tool in dermatology practice. This standardized approach enhances communication among healthcare professionals, ultimately improving patient care and research comparability in dermatology.

Natureness‐preserving tone mapping operator based on improved guided filter and adaptive Gamma curve

AbstractMapping high dynamic range (HDR) images to low dynamic range (LDR) display devices while performing enhancement processing is a challenging task. To address the issues of chromatic drift, the imbalance between preserving naturalness and enhancing details during the mapping process, this paper proposed a tone mapping operator (TMO) based on an improved guided filter with multi‐layered processing. Firstly, the operator tackled the problem of insufficient extraction of detail layer information in complex scene images by redefining the formula of the guided filter and introducing adaptive regularization parameters. This enhancement enabled more effective extraction of details compared to traditional fixed‐parameter guided filters. Secondly, the algorithm employed a centre‐surround approach to subdivide and processed the detail layers individually, effectively suppressing halos and artefacts. Thirdly, image entropy was utilized to guide the fusion and enhancement processing of the detail layers, preventing excessive or insufficient enhancement. Finally, this paper proposed an adaptive gamma correction algorithm and a chromatic consistency algorithm to improve overall contrast and hue saturation. The results demonstrate convincing performance in terms of subjective visual evaluation and image quality assessment.

Low-Light Image Enhancement Method for Electric Power Operation Sites Considering Strong Light Suppression

Insufficient light, uneven light, backlighting, and other problems lead to poor visibility of the image of an electric power operation site. Most of the current methods directly enhance the low-light image while ignoring local strong light that may appear in the electric power operation site, resulting in overexposure and a poor enhancement effect. Aiming at the above problems, we propose a low-light image enhancement method for electric power operation sites by considering strong light suppression. Firstly, a sliding-window-based strong light judgment method was designed, which used a sliding window to segment the image, and a brightness judgment was performed based on the average value of the deviation and the average deviation of the subimages of the grayscale image from the strong light threshold. Then, a light effect decomposition method based on a layer decomposition network was used to decompose the light effect of RGB images with the presence of strong light and eliminate the light effect layer. Finally, a Zero-DCE (Zero-Reference Deep Curve Estimation) low-light enhancement network based on a kernel selection module was constructed to enhance the low-light images with reduced or no strong light interference. Comparison experiments using the electric power operation private dataset and the SICE (Single Image Contrast Enhancement) Part 2 public dataset showed that our proposed method outperformed the current state-of-the-art low-light enhancement methods in terms of both subjective visual effects and objective evaluation metrics, which effectively improves the image quality of electric power operation sites in low-light environments and provides excellent image bases for other computer vision tasks, such as the estimation of operators’ posture.

Association between three-year mortality after transcatheter aortic valve implantation and paravalvular regurgitation graded by videodensitometry in comparison with visual grading

BackgroundEstimation of regurgitant fraction by videodensitometry (VD-AR) of aortic root angiograms is a new tool for objective grading of paravalvular regurgitation (PVR) after transcatheter aortic valve implantation (TAVI). Stratification with boundaries at 6% and 17% has been proposed to reflect “none/trace”, “mild” and “moderate or higher” PVR.ObjectiveWe sought to investigate the association of strata of VD-AR with 3-year mortality and to compare VD-AR with visual grading of angiograms.MethodsWe interrogated our database for patients undergoing transfemoral TAVI from 2008 to 2018. Vital status of the patients was obtained from population registers. To test differences in survival and estimate adjusted hazard ratios (HRs) we fitted Cox models.ResultsOur retrospective study included 699 patients with evaluable angiograms at completion of the TAVI procedure. Cumulative 3-year mortality was 35.0% in 261 (37.3%) patients with VD-AR < 6%, 33.9% in 325 (46.5%) patients with VD-AR between 6 and 17% (HR [95% confidence interval] 1.06 [0.80–1.42]; P = 0.684) and 47.2% in 113 (16.2%) patients with VD-AR > 17% (HR 1.57 [1.11–2.22]; P = 0.011). Visually, PVR was graded as “none/trace” in 470 (67.2%) patients, as “mild” in 219 (31.3%) and as “moderate” in 10 (1.4%). Both mild PVR and moderate PVR on visual grading were significantly associated with mortality (HRs 1.31 [1.12–1.54]; P = 0.001 and 1.92 [1.13–3.24]; P = 0.015; respectively).ConclusionsVD-AR > 17%, but not VD-AR 6–17%, was independently associated with mortality. Compared with subjective visual evaluation, VD-AR resulted in a smaller proportion of patients with PVR classified as prognostically relevant.Graphical

Improved Retinex-Theory-Based Low-Light Image Enhancement Algorithm

Researchers working on image processing have had a hard time handling low-light images due to their low contrast, noise, and brightness. This paper presents an improved method that uses the Retinex theory to enhance low-light images, with a network model mainly composed of a Decom-Net and an Enhance-Net. Residual connectivity is fully utilized in both the Decom-Net and Enhance-Net to reduce the possible loss of image details. Additionally, Enhance-Net introduces a positional pixel attention mechanism that directly incorporates the global information of the image. Specifically, Decom-Net serves to decompose the low-light image into illumination and reflection maps, and Enhance-Net serves to increase the brightness of the illumination map. Finally, via adaptive image fusion, the reflectance map and the enhanced illuminance map are fused to obtain the final enhanced image. Experiments show better results in terms of both subjective visual aspects and objective evaluation indicators. Compared to RetinexNet, the proposed method shows improvements in the full-reference evaluation metrics, including a 4.6% improvement in PSNR, a 1.8% improvement in SSIM, and a 10.8% improvement in LPIPS. Additionally, it achieved an average improvement of 17.3% in the no-reference evaluation metric NIQE.

An improved hibbard interpolation algorithm based on edge gradient feature detection using Sobel operator

Color interpolation algorithms have important applications in color image processing, and many interpolation algorithms have been proposed one after another. The bilinear interpolation algorithm, Cok interpolation algorithm and gradient edge interpolation are three classical interpolation algorithms, however, they may cause problems such as zipper effect and Mohr's stripe effect. In this paper, this paper optimize the gradient edge algorithm, calculate the gradient by using Sobel operator and fit the direction of the edge in a weighted way, and propose a gradient edge weighted interpolation algorithm based on Sobel operator to improve the performance of the interpolation algorithm. Through the validation of color image recovery, the results show that the subjective visual effect and objective evaluation indexes such as structural similarity index, peak signal-to-noise ratio and minimum mean square error of the improved algorithm proposed in this paper are better than the traditional algorithm, which is of great significance to the improvement of the interpolation algorithm.

Infrared and visible image fusion based on domain transform filtering and sparse representation

Infrared and visible image fusion is an important branch in the field of information fusion. Fused images can provide more comprehensive and objective interpretations of complex scenes with confusing or hidden information. This paper proposes a novel infrared and visible image fusion method based on domain transform filtering (DTF) and sparse representation (SR). First, infrared and visible images are decomposed using a low-pass filter into base and detail layers, respectively. An SR-based rule is designed to fuse the detail layers. To improve the feature expression ability of a redundant dictionary, we introduce a multiscale detail enhancement technique to preprocess the training data. Additionally, a fast but effective rule is proposed based on salient structure extraction and DTF base layer fusion. The structure and border information of source images can be successfully transformed into the fused image. The proposed method can effectively retain the salient structure and edge information of the source images in the fused image. Experimental results demonstrate that the proposed method can outperform state-of-the-art methods in terms of both subjective visual and objective quantitative evaluations. The source code of the proposed method is released at https://github.com/ixilai/DTF-SR.

RTFusion: A Multimodal Fusion Network with Significant Information Enhancement.

Multimodal medical fusion images are important for clinical diagnosis because they can better reflect the location of disease and provide anatomically detailed information. Existing medical image fusion methods can cause significant information loss in fusion images to varying degrees. Therefore, we designed a residual transformer fusion network (RTFusion): a multimodal fusion network with significant information enhancement. We use the residual transformer to make the image information interact remotely to ensure the global information of the image and use the residual structure to enhance the feature information to prevent information loss. Then the channel attention and spatial attention module (CASAM) is added to the fusion process to enhance the significant information of the fusion image, and the feature interaction module is used to promote the interaction of specific information of the source image. Finally, the loss function of the block calculation is designed to drive the fusion network to retain rich texture details, structural information, and color information, to optimize the subjective visual effect of the image. Extensive experiments show that our method can better recover the significant information of the source image and outperform other advanced methods in subjective visual description and objective metric evaluation. In particular, the color information and texture information are balanced to enhance the visual effect of the fused image.

GL-Segnet: Global-Local representation learning net for medical image segmentation.

Medical image segmentation has long been a compelling and fundamental problem in the realm of neuroscience. This is an extremely challenging task due to the intensely interfering irrelevant background information to segment the target. State-of-the-art methods fail to consider simultaneously addressing both long-range and short-range dependencies, and commonly emphasize the semantic information characterization capability while ignoring the geometric detail information implied in the shallow feature maps resulting in the dropping of crucial features. To tackle the above problem, we propose a Global-Local representation learning net for medical image segmentation, namely GL-Segnet. In the Feature encoder, we utilize the Multi-Scale Convolution (MSC) and Multi-Scale Pooling (MSP) modules to encode the global semantic representation information at the shallow level of the network, and multi-scale feature fusion operations are applied to enrich local geometric detail information in a cross-level manner. Beyond that, we adopt a global semantic feature extraction module to perform filtering of irrelevant background information. In Attention-enhancing Decoder, we use the Attention-based feature decoding module to refine the multi-scale fused feature information, which provides effective cues for attention decoding. We exploit the structural similarity between images and the edge gradient information to propose a hybrid loss to improve the segmentation accuracy of the model. Extensive experiments on medical image segmentation from Glas, ISIC, Brain Tumors and SIIM-ACR demonstrated that our GL-Segnet is superior to existing state-of-art methods in subjective visual performance and objective evaluation.

DGC‐UWnet: Underwater image enhancement based on computation‐efficient convolution and channel shuffle

AbstractUnderwater image enhancement is receiving increasing attention due to many of the vision research being applied to underwater scenes. To eliminate the impact of complex underwater scenes on imaging, underwater image enhancement algorithm has become an effective solution. However, underwater image enhancement models face a challenge of lightening the model while improving generalizability. Here, DGC‐UWnet is proposed to go for both lightweight and enhancement effect. The proposed model is designed by using depthwise convolution, group convolution and channel shuffle (DGC). Ablation experiment shows that compared with standard convolution, DGC decreases model parameters and computational complexity, and improves the generalizability of the model. Qualitative and quantitative comparative experimental results show that comprehensive performances of the model can catch up with or even surpass state‐of‐the‐art (SOTA) algorithms in terms of processing speed, subjective visual perception and objective evaluation metrics. In addition, application test results prove that DGC‐UWnet can be used as the pre‐processing for underwater applications of other visual algorithms such as improving performance of YOLOv5l.

Single Image Dehazing Based on Contrastive Learning and Transformer

For the single image dehazing problem, an end-to-end multi-stage dehazing algorithm is designed. The algorithm contains two distinct parts to extract features. For shallow features, texture-level information is mined by stacking pixel and channel attention mechanisms. The proposed method uses multi-head self-attention (MHSA) to capture high-level features. MHSA improves dehazing performance by mining the dependencies of a wide range of abstract information. The superiority of the transformer architecture is extended with cascaded attention mechanisms and convolutions to improve feature extraction capabilities. Multilayer perceptron (MLP) is used in the decoding stage to equalize the context information. Furthermore, a contrastive loss function that introduces multiple negative samples and correction terms is proposed. The correction term is generated according to the difference between the precise and blurred images, which can enhance the training effect when dealing with different concentrations of dehaze. The training result of this loss function assists the model in approximating clear images and staying away from blurry images. According to the experimental results compared with other methods under the same conditions, the proposed method achieves good results in both subjective visual effects and objective evaluation indicators. The proposed contrast loss function also improves the dehazing performance of the algorithm.

Short‐term and long‐term memory self‐attention network for segmentation of tumours in 3D medical images

AbstractTumour segmentation in medical images (especially 3D tumour segmentation) is highly challenging due to the possible similarity between tumours and adjacent tissues, occurrence of multiple tumours and variable tumour shapes and sizes. The popular deep learning‐based segmentation algorithms generally rely on the convolutional neural network (CNN) and Transformer. The former cannot extract the global image features effectively while the latter lacks the inductive bias and involves the complicated computation for 3D volume data. The existing hybrid CNN‐Transformer network can only provide the limited performance improvement or even poorer segmentation performance than the pure CNN. To address these issues, a short‐term and long‐term memory self‐attention network is proposed. Firstly, a distinctive self‐attention block uses the Transformer to explore the correlation among the region features at different levels extracted by the CNN. Then, the memory structure filters and combines the above information to exclude the similar regions and detect the multiple tumours. Finally, the multi‐layer reconstruction blocks will predict the tumour boundaries. Experimental results demonstrate that our method outperforms other methods in terms of subjective visual and quantitative evaluation. Compared with the most competitive method, the proposed method provides Dice (82.4% vs. 76.6%) and Hausdorff distance 95% (HD95) (10.66 vs. 11.54 mm) on the KiTS19 as well as Dice (80.2% vs. 78.4%) and HD95 (9.632 vs. 12.17 mm) on the LiTS.

A sub-pixel convolution-based improved bidirectional feature pyramid network for pansharpening

ABSTRACT Due to the dramatic scale changes in different ground object characteristics, effectively utilizing multi-scale features of remote sensing images represents a major challenge. This letter proposes a sub-pixel convolution-based improved bidirectional feature pyramid network (SCIBFPN) to address the problem. To reduce the information loss in the process of image pre-processing and multi-scale feature fusion, this letter introduces sub-pixel convolution instead of up-sampling. To obtain richer spatial and spectral information, an improved bidirectional feature pyramid network-based sub-pixel convolution is combined with a residual network (ResNet) for feature extraction. In addition, to further enhance the spatial structure of the fused images, a panchromatic (PAN) image is used as a guide to direct the injection of spatial information. Experimental results from two real datasets show that the proposed method outperforms state-of-the-art methods in terms of both objective metrics and subjective visual evaluation.

Infrared and Visible Image Fusion via Interactive Compensatory Attention Adversarial Learning

The existing generative adversarial fusion methods generally concatenate source images or deep features, and extract local features through convolutional operations without considering their global characteristics, which tends to produce a limited fusion performance. Toward this end, we propose a novel interactive compensatory attention fusion network, termed ICAFusion. In particular, in the generator, we construct a multi-level encoder-decoder network with a triple path, and design infrared and visible paths to provide additional intensity and gradient information for the concatenating path. Moreover, we develop the interactive and compensatory attention modules to communicate their pathwise information, and model their long-range dependencies through a cascading channel-spatial model. The generated attention maps can more focus on infrared target perception and visible detail characterization, and are used to reconstruct the fusion image. Therefore, the generator takes full advantage of local and global features to further increase the representation ability of feature extraction and feature reconstruction. Extensive experiments illustrate that our ICAFusion obtains superior fusion performance and better generalization ability, which precedes other advanced methods in the subjective visual description and objective metric evaluation. Our codes will be public at <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><uri>https://github.com/Zhishe-Wang/ICAFusion</uri></i> .

Multi-focus image fusion based on HOSVD and parameter adaptive PCNN in fast local Laplacian filtering domain

To augment edge preservation capabilities and detail enhancement in multi-focus image fusion, a new Multi-focus image fusion algorithm is proposed in this paper. Firstly, we select fast local laplacian filtering (FLLF) as the multi-scale decomposition tool to decompose the source image into an approximate image and multiple detail images. Secondly, higher-order singular value decomposition (HOSVD) is applied to fuse the approximate image. Thirdly, the parameter-adaptive pulse coupled-neural network (PA-PCNN) model is used to fuse the detailed images. Finally, exploiting the combination of the fusion results of the approximate image and the detail images, the final fusion image is obtained using the FLLF inverse transform. The experimental results show that the proposed method achieves excellent performance in both subjective visual perception and objective index evaluation.

MSE-Fusion: Weakly supervised medical image fusion with modal synthesis and enhancement

Existing multi-modal image fusion methods utilize multi-modal images as input that require multiple imaging of patients causing harm to patients’ bodies and large costs, moreover, image fusion needs a large number of registered images which is time-consuming and difficult to get, and has unclear texture and structure of the fused images. Therefore, a weakly supervised medical image fusion method with modal synthesis and enhancement is proposed. In modal synthesis, a weakly supervised approach is used to train the model to decrease the requirements of registered images, and MR images are used as input to synthesize CT images through a deep-structure and shallow-detail generator by training to reduce the required input modal and make the texture and structure clearer. In image enhancement, MR images are passed through a trained generator to generate enhanced MR images which enhance the texture and structure of the MR images. And then using the synthesized CT and enhanced MR images together with the original PET images as input to achieve tri-modal image fusion. Compared with 13 state-of-the-art modal synthesis and image fusion methods on the same datasets, the performance of the proposed method on 7 objective evaluation metrics is significantly improved. The subjective visual effect and objective evaluation metrics of our method are better than those of the compared image fusion methods.

Rethinking multi-exposure image fusion with extreme and diverse exposure levels: A robust framework based on Fourier transform and contrastive learning

Multi-exposure image fusion (MEF) is an important technique for generating high dynamic range images. However, most existing MEF studies focus on fusing a moderately over-exposed image and a moderately under-exposed image, and they are not robust in fusing images with extreme and diverse exposure levels. In this paper, we propose a robust MEF framework based on Fourier transform and contrastive learning. Specifically, we develop a Fourier transform-based pixel intensity transfer strategy to synthesize images with diverse exposure levels from normally exposed natural images and train an encoder–decoder network to reconstruct the original natural image. In this way, the encoder and decoder can learn to extract features from images with diverse exposure levels and generate fused images with normal exposure. We propose a contrastive regularization loss to further enhance the capability of the network in recovering normal exposure levels. In addition, we construct an extreme MEF benchmark dataset and a random MEF benchmark dataset for a more comprehensive evaluation of MEF algorithms. We extensively compare our method with fifteen competitive traditional and deep learning-based MEF algorithms on three benchmark datasets, and our method outperforms the other methods in both subjective visual effects and objective evaluation metrics. Our code, datasets and all fused images will be released.

Multi-modal medical image super-resolution fusion based on detail enhancement and weighted local energy deviation

Multi-modal medical image fusion (MMIF) integrates medical images of different modalities into an image with rich information to boost the accuracy and efficiency of clinical diagnosis and treatment. There are two main problems in medical image fusion: 1) It is difficult to balance the computational efficiency and fusion quality; 2) in the clinic, it is necessary to observe medical images in high resolution. To overcome these problems, a multi-modal medical image super-resolution (SR) fusion (MMISRF) method is proposed based on detail enhancement and weighted local energy deviation (WLED). The method has two major novelties. For the first problem, to improve the efficiency, the proposed method decomposes the SR image into one base layer and one detail layer through a two-scale decomposition method. Then, for the detail layer, a fusion rule is proposed based on detail enhancement and information refinement to enhance the detail information and retain the salient feature information. For the base layer, a WLED-based rule is designed to better preserve the energy information from the source images to the fused image. The final fused image is obtained by combining the fused detail and base layers. For the second problem, this paper introduces the bicubic interpolation-based SR into the field of MMIF for the first time. The experimental results indicate that the proposed MMISRF outperforms the state-of-the-art approaches in terms of subjective visual effect and objective evaluation. Furthermore, the proposed method is more effective for MMIF task at different resolutions.