Visual Quality Metrics Research Articles

Non-Uniform-Illumination Image Enhancement Algorithm Based on Retinex Theory

To address the issues of fuzzy scene details, reduced definition, and poor visibility in images captured under non-uniform lighting conditions, this paper presents an algorithm for effectively enhancing such images. Firstly, an adaptive color balance method is employed to address the color differences in low-light images, ensuring a more uniform color distribution and yielding a low-light image with improved color consistency. Subsequently, the image obtained is transformed from the RGB space to the HSV space, wherein the multi-scale Gaussian function is utilized in conjunction with the Retinex theory to accurately extract the lighting components and reflection components. To further enhance the image quality, the lighting components are categorized into high-light areas and low-light areas based on their pixel mean values. The low-light areas undergo improvement through an enhanced adaptive gamma correction algorithm, while the high-light areas are enhanced using the Weber–Fechner law for optimal results. Then, each block area of the image is weighted and fused, leading to its conversion back to the RGB space. And a multi-scale detail enhancement algorithm is utilized to further enhance image details. Through comprehensive experiments comparing various methods based on subjective visual perception and objective quality metrics, the algorithm proposed in this paper convincingly demonstrates its ability to effectively enhance the brightness of non-uniformly illuminated areas. Moreover, the algorithm successfully retains details in high-light regions while minimizing the impact of non-uniform illumination on the overall image quality.

Intra-block pyramid cross-scale network for thermal radiation effect correction of uncooled infrared images.

Thermal radiation effects can greatly degrade the image quality of uncooled infrared focal plane array detection systems. In this paper, we propose a thermal radiation effect correction network based on intra-block pyramid cross-scale feature extraction and fusion. First, an intra-block pyramid residual attention module is introduced to obtain fine-grained features from long-range IR images by extracting cross-scale local features within the residual block. Second, we propose a cross-scale gated fusion module to efficiently integrate the shallow and abstract features at multiple scales of the encoder and decoder through gated linear units. Finally, to ensure accurate correction of thermal radiation effects, we add double-loss constraints in the spatial-frequency domain and construct a single-input, multi-output network with multiple supervised constraints. The experimental results demonstrate that our proposed method outperforms state-of-the-art correction methods in terms of both visual quality and quantitative evaluation metrics.

Visual perception-oriented quality assessment for high dynamic range stereoscopic omnidirectional video system

High dynamic range stereoscopic omnidirectional video (HSOV) system provides users with an immersive visual experience, but the distortions generated in the process of visual content generation, coding, and visualization also brings challenges to visual quality assessment. This paper proposes a visual perception-oriented quality metric for HSOV system. Considering the computational efficiency, for frames within each second, several key frames are selected so that features reflecting distortions can be extracted from the viewports and equi-angular cubemap (EAC) projection surfaces of these frames. In viewport feature extraction module, the sum and difference maps of the left and right views of the key frames are used to simulate the binocular perception. Considering that distortions will affect the structure, color and statistical properties of the image, the gradient, local binary pattern and statistical features are extracted, respectively. To reduce projection distortion, EAC projection is used to instead of other projections normally used in quality assessment, and the Sigma domain feature extraction and color feature extraction are performed on EAC projection map. In temporal feature extraction module, wavelet decomposition is used to perform temporal domain bandpass filtering on partial continuous frames to calculate the naturalness features. Finally, the above features of the frames at different scales are pooled through random forest model to predicate the quality score of HSOV. Experimental results on the NBU-HSOVD dataset show that the proposed metric is superior to some blind image/video quality assessment metrics and has higher consistency with the human visual perception.

Transfer learning-based attenuation correction for static and dynamic cardiac PET using a generative adversarial network.

The goal of this work is to demonstrate the feasibility of directly generating attenuation-corrected PET images from non-attenuation-corrected (NAC) PET images for both rest and stress-state static or dynamic [13N]ammonia MP PET based on a generative adversarial network. We recruited 60 subjects for rest-only scans and 14 subjects for rest-stress scans, all of whom underwent [13N]ammonia cardiac PET/CT examinations to acquire static and dynamic frames with both 3D NAC and CT-based AC (CTAC) PET images. We developed a 3D pix2pix deep learning AC (DLAC) framework via a U-net + ResNet-based generator and a convolutional neural network-based discriminator. Paired static or dynamic NAC and CTAC PET images from 60 rest-only subjects were used as network inputs and labels for static (S-DLAC) and dynamic (D-DLAC) training, respectively. The pre-trained S-DLAC network was then fine-tuned by paired dynamic NAC and CTAC PET frames of 60 rest-only subjects to derive an improved D-DLAC-FT for dynamic PET images. The 14 rest-stress subjects were used as an internal testing dataset and separately tested on different network models without training. The proposed methods were evaluated using visual quality and quantitative metrics. The proposed S-DLAC, D-DLAC, and D-DLAC-FT methods were consistent with clinical CTAC in terms of various images and quantitative metrics. The S-DLAC (slope = 0.9423, R2 = 0.947) showed a higher correlation with the reference static CTAC as compared to static NAC (slope = 0.0992, R2 = 0.654). D-DLAC-FT yielded lower myocardial blood flow (MBF) errors in the whole left ventricular myocardium than D-DLAC, but with no significant difference, both for the 60 rest-state subjects (6.63 ± 5.05% vs. 7.00 ± 6.84%, p = 0.7593) and the 14 stress-state subjects (1.97 ± 2.28% vs. 3.21 ± 3.89%, p = 0.8595). The proposed S-DLAC, D-DLAC, and D-DLAC-FT methods achieve comparable performance with clinical CTAC. Transfer learning shows promising potential for dynamic MP PET.

ЕФЕКТИВНІ МЕТОДИ ІНТЕЛЕКТУАЛЬНОЇ ОБРОБКИ ЗОБРАЖЕНЬ ТА ВІДЕО НА ОСНОВІ МЕТРИК ВІЗУАЛЬНОЇ ЯКОСТІ ДЛЯ ПЕРСПЕКТИВНИХ ЗАСТОСУВАНЬ

This chapter contains the results obtained during execution of Ukrainian-Polish Project in 2021 intended on design of methods and means for processing grayscale and multichannel images and video using visual quality metrics. The combined metrics have been

Lightweight Infrared and Visible Image Fusion via Adaptive DenseNet with Knowledge Distillation

The fusion of infrared and visible images produces a complementary image that captures both infrared radiation information and visible texture structure details using the respective sensors. However, the current deep-learning-based fusion approaches mainly tend to prioritize visual quality and statistical metrics, leading to an increased model complexity and weight parameter sizes. To address these challenges, we propose a novel dual-light fusion approach using adaptive DenseNet with knowledge distillation to learn and compress from pre-existing fusion models, which achieves the goals of model compression through the use of hyperparameters such as the width and depth of the model network. The effectiveness of our proposed approach is evaluated on a new dataset comprising three public datasets (MSRS, M3FD, and LLVIP), and both qualitative and quantitative experimental results show that the distillated adaptive DenseNet model effectively matches the original fusion models’ performance with smaller model weight parameters and shorter inference times.

МЕТОДИ ІНТЕЛЕКТУАЛЬНОЇ ОБРОБКИ ЗОБРАЖЕНЬ ТА ВІДЕО НА ОСНОВІ МЕТРИК ВІЗУАЛЬНОЇ ЯКОСТІ ДЛЯ ПЕРСПЕКТИВНИХ ЗАСТОСУВАНЬ

This chapter contains the results obtained during execution of Ukrainian-Polish Project in 2020 intended on design of methods and means for processing grayscale and multichannel images and video using visual quality metrics. The combined metrics have been

BPG-Based Lossy Compression of Three-Channel Noisy Images with Prediction of Optimal Operation Existence and Its Parameters

Nowadays, there is a clear trend toward increasing the number of remote-sensing images acquired and their average size. This leads to the need to compress the images for storage, dissemination, and transfer over communication lines where lossy compression techniques are more popular. The images to be compressed or some of their components are often noisy. They must therefore be compressed taking into account the properties of the noise. Due to the noise filtering effect obtained during lossy compression of noisy images, an optimal operating point (OOP) may exist. The OOP is a parameter that controls the compression for which the quality of the compressed image is closer (closest) to the corresponding noise-free image than the quality of the noisy (original, uncompressed) image according to some quantitative criterion (metric). In practice, it is important to know whether the OOP exists for a given image, because if the OOP exists, it is appropriate to perform the compression in the OOP or at least in its neighborhood. Since the real image is absent in practice, it is impossible to determine a priori whether the OOP exists or not. Here, we focus on three-channel-remote-sensing images and show that it is possible to easily predict the existence of the OOP. Furthermore, it is possible to predict the metric values or their improvements with appropriate accuracy for practical use. The BPG (better portable graphics) encoder is considered a special case of an efficient compression technique. As an initial design step, the case of additive white Gaussian noise with equal variance in the three components is considered. While previous research was mainly focused on predicting the improvement (reduction) of the PSNR and PSNR-HVS-M metrics, here we focus on the modern visual quality metrics, namely PSNR-HA and MDSI. We also discuss what to do if, according to the prediction, an OOP is absent. Examples of lossy compression of noisy three-channel remote sensing images are given. It is also shown that the use of three-dimensional compression provides a compression ratio increase by several times compared with component-wise compression in the OOP.

A joint convolution auto-encoder network for infrared and visible image fusion

Background: Leaning redundant and complementary relationships is a critical step in the human visual system. Inspired by the infrared cognition ability of crotalinae animals, we design a joint convolution auto-encoder (JCAE) network for infrared and visible image fusion. Methods: Our key insight is to feed infrared and visible pair images into the network simultaneously and separate an encoder stream into two private branches and one common branch, the private branch works for complementary features learning and the common branch does for redundant features learning. We also build two fusion rules to integrate redundant and complementary features into their fused feature which are then fed into the decoder layer to produce the final fused image. We detail the structure, fusion rule and explain its multi-task loss function. Results: Our JCAE network achieves good results in terms of both visual quality and objective evaluation metrics.

Infrared and Visible Image Fusion Method Based on a Principal Component Analysis Network and Image Pyramid

The aim of infrared (IR) and visible image fusion is to generate a more informative image for human observation or some other computer vision tasks. The activity-level measurement and weight assignment are two key parts in image fusion. In this paper, we propose a novel IR and visible fusion method based on the principal component analysis network (PCANet) and an image pyramid. Firstly, we use the lightweight deep learning network, a PCANet, to obtain the activity-level measurement and weight assignment of IR and visible images. The activity-level measurement obtained by the PCANet has a stronger representation ability for focusing on IR target perception and visible detail description. Secondly, the weights and the source images are decomposed into multiple scales by the image pyramid, and the weighted-average fusion rule is applied at each scale. Finally, the fused image is obtained by reconstruction. The effectiveness of the proposed algorithm was verified by two datasets with more than eighty pairs of test images in total. Compared with nineteen representative methods, the experimental results demonstrate that the proposed method can achieve the state-of-the-art results in both visual quality and objective evaluation metrics.

GPSR: Gradient-Prior-Based Network for Image Super-Resolution

Recent deep learning has shown great potential in super-resolution (SR) tasks. However, most deep learning-based SR networks are optimized via pixel-level loss (i.e., L1, L2, and MSE), which forces the networks to output the average of all possible predictions, leading to blurred details. Especially in SR tasks with large scaling factors (i.e., ×4, ×8), the limitation is further aggravated. To alleviate this limitation, we propose a Gradient-Prior-based Super-Resolution network (GPSR). Specifically, a detail-preserving Gradient Guidance Strategy is proposed to fully exploit the gradient prior to guide the SR process from two aspects. On the one hand, an additional gradient branch is introduced into GPSR to provide the critical structural information. On the other hand, a compact gradient-guided loss is proposed to strengthen the constraints on the spatial structure and to prevent the blind restoration of high-frequency details. Moreover, two residual spatial attention adaptive aggregation modules are proposed and incorporated into the SR branch and the gradient branch, respectively, to fully exploit the crucial intermediate features to enhance the feature representation ability. Comprehensive experimental results demonstrate that the proposed GPSR outperforms state-of-the-art methods regarding both subjective visual quality and objective quantitative metrics in SR tasks with large scaling factors (i.e., ×4 and ×8).

RESEARCH PROGRESS OF DEEP LEARNING IN LOW-DOSE CT IMAGE DENOISING.

Low-dose computed tomography (CT) will increase noise and artefacts while reducing the radiation dose, which will adversely affect the diagnosis of radiologists. Low-dose CT image denoising is a challenging task. There are essential differences between the traditional methods and the deep learning-based methods. This paper discusses the denoising approaches of low-dose CT image via deep learning. Deep learning-based methods have achieved relatively ideal denoising effects in both subjective visual quality and quantitative objective metrics. This paper focuses on three state-of-the-art deep learning-based image denoising methods, in addition, four traditional methods are used as the control group to compare the denoising effect. Comprehensive experiments show that the deep learning-based methods are superior to the traditional methods in low-dose CT images denoising.

Domain Adaptation for Underwater Image Enhancement.

Recently, learning-based algorithms have shown impressive performance in underwater image enhancement. Most of them resort to training on synthetic data and obtain outstanding performance. However, these deep methods ignore the significant domain gap between the synthetic and real data (i.e., inter-domain gap), and thus the models trained on synthetic data often fail to generalize well to real-world underwater scenarios. Moreover, the complex and changeable underwater environment also causes a great distribution gap among the real data itself (i.e., intra-domain gap). However, almost no research focuses on this problem and thus their techniques often produce visually unpleasing artifacts and color distortions on various real images. Motivated by these observations, we propose a novel Two-phase Underwater Domain Adaptation network (TUDA) to simultaneously minimize the inter-domain and intra-domain gap. Concretely, in the first phase, a new triple-alignment network is designed, including a translation part for enhancing realism of input images, followed by a task-oriented enhancement part. With performing image-level, feature-level and output-level adaptation in these two parts through jointly adversarial learning, the network can better build invariance across domains and thus bridging the inter-domain gap. In the second phase, an easy-hard classification of real data according to the assessed quality of enhanced images is performed, in which a new rank-based underwater quality assessment method is embedded. By leveraging implicit quality information learned from rankings, this method can more accurately assess the perceptual quality of enhanced images. Using pseudo labels from the easy part, an easy-hard adaptation technique is then conducted to effectively decrease the intra-domain gap between easy and hard samples. Extensive experimental results demonstrate that the proposed TUDA is significantly superior to existing works in terms of both visual quality and quantitative metrics.

Proposal of a new fidelity measure between computed image quality and observers quality scores accounting for scores variability

Assessment of the visual quality of colour images is usually a difficult process, validated through hard-to-carry-out psychophysical experiments, used to record observer quality scores. Visual image quality metrics aim to maximise the agreement between computed indexes and observer scores, or opinions. Therefore, in this area, it is of critical importance to have appropriate measures of this agreement (i.e. performance) between the computed image quality metric values and observer’s quality scores, both for the development, as well as for the use of image quality metrics. Among the measures of agreement, the most used one nowadays is the well-known Pearson correlation coefficient, while Spearman rank correlation coefficient is also commonly used. The aim of this paper is two-fold. First, to introduce the Standardised Residual Sum of Squares (STRESS) as an alternative metric for the agreement between computed image quality and observers quality scores and analyse its properties and advantages in front of Pearson, Spearman and Kendall correlation coefficients; Second, to introduce a new version of STRESS (called USTRESS) that takes observers’ scores variability into account. The results on synthetic and real datasets support that STRESS has a series of benefits in front of the classical approaches and that the inclusion of uncertainty in STRESS has an important effect on the results, quantified by statistical significance tests. A free to download MATLAB code version of USTRESS is available at https://viplab.webs.upv.es/resources/

Relationships between the Visual Quality and Color Patterns: Study in Peri-Urban Forests Dominated by Cotinus coggygria var. cinerea Engl. in Autumn in Beijing, China

The spatial pattern of color patches plays a crucial role in affecting the visual quality of peri-urban forests dominated by Cotinus coggygria var. cinerea Engl. in autumn. The impact mechanism has been studied to facilitate algorithm-based automatic visual quality estimation. The color patterns of 120 photographs were calculated after color quantization and automatic color substitution. The scenic beauty of the forest was estimated by 698 respondents. Multiple correlations between visual quality and color pattern metrics were explored with stepwise regression. Principal component analysis (PCA) was also employed to investigate the impact mechanism of color patterns on visual quality. Number of patches (NP), largest patch index (LPI), mean patch area (AREA_MN), patch size standard deviation (AREA_SD), and Shannon’s evenness index (SHEI) were the main factors affecting the visual quality of the Cotinus coggygria forest. AREA_MN correlated positively with visual quality, while NP, LPI, AREA_SD, and SHEI correlated negatively. Moreover, AREA_SD had the most significant impact on the visual quality of the landscape, while SHEI, LPI, and AREA_MN had the second-highest impact. The evenness and the size of color patches significantly affected the visual quality of the forest landscapes. Balancing the diversity and evenness of color patches plays a decisive role in creating a forest landscape with high visual quality.

Generative Memory-Guided Semantic Reasoning Model for Image Inpainting

The critical challenge of single image inpainting stems from accurate semantic inference via limited information while maintaining image quality. Typical methods for semantic image inpainting train an encoder-decoder network by learning a one-to-one mapping from the corrupted image to the inpainted version. While such methods perform well on images with small corrupted regions, it is challenging for these methods to deal with images with large corrupted area due to two potential limitations. 1) Such one-to-one mapping paradigm tends to overfit each single training pair of images; 2) The inter-image prior knowledge about the general distribution patterns of visual semantics, which can be transferred across images sharing similar semantics, is not explicitly exploited. In this paper, we propose the Generative Memory-guided Semantic Reasoning Model (GM-SRM), which infers the content of corrupted regions based on not only the known regions of the corrupted image, but also the learned inter-image reasoning priors characterizing the generalizable semantic distribution patterns between similar images. In particular, the proposed GM-SRM first pre-learns a generative memory from the whole training data to explicitly learn the distribution of different semantic patterns. Then the learned memory are leveraged to retrieve the matching semantics for the current corrupted image to perform semantic reasoning during image inpainting. While the encoder-decoder network is used for guaranteeing the pixel-level content consistency, our generative priors are favorable for performing high-level semantic reasoning, which is particularly effective for inferring semantic content for large corrupted area. Extensive experiments on Paris Street View, CelebA-HQ, and Places2 benchmarks demonstrate that our GM-SRM outperforms the state-of-the-art methods for image inpainting in terms of both visual quality and quantitative metrics.

A Method Noise-Based Convolutional Neural Network Technique for CT Image Denoising

Medical imaging is a complex process that capitulates images created by X-rays, ultrasound imaging, angiography, etc. During the imaging process, it also captures image noise during image acquisition, some of which are extremely corrosive, creating a disturbance that results in image degradation. The proposed work addresses the challenge to eliminate the corrosive Gaussian additive white noise from computed tomography (CT) images while preserving the fine details. The proposed approach is synthesized by amalgamating the concept of method noise with a deep learning-based framework of a convolutional neural network (CNN). The corrupted images are obtained by explicit addition of Gaussian additive white noise at multiple noise variance levels (σ = 10, 15, 20, 25). The denoised images obtained are then evaluated according to their visual quality and quantitative metrics, such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). These metrics for denoised CT images are then compared with their respective values for the reference CT image. The average PSNR value of the proposed method is 25.82, the average SSIM value is 0.85, and the average computational time is 2.8760. To better understand the proposed approach’s effectiveness, an intensity profile of denoised and original medical images is plotted and compared. To further test the performance of the proposed methodology, the results obtained are also compared with that of other non-traditional methods. The critical analysis of the results shows the commendable efficiency of the proposed methodology in denoising the medical CT images corrupted by Gaussian noise. This approach can be utilized in multiple pragmatic areas of application in the field of medical image processing.

Automatic calibration of crack and flaking diseases in ancient temple murals

Many precious ancient murals are seriously deteriorated due to long-term environmental influences and man-made destructions. How to effectively protect ancient murals and restore these murals’ original appearance has become an urgent problem for field experts. Modern computer technology makes it possible to virtually restore the deteriorated areas in ancient murals. However, most existing mural restoration approaches require manual calibration of the deteriorated areas, which is very difficult and time-consuming. It has been noticed that the earth layer flaking and cracks are the most common problems of ancient temple murals. This paper proposes an automatic calibration method for the earth layer flaking and cracking deterioration of murals by taking temple murals from the Ming Dynasty in Zhilin Temple as the study object. First, we extract the texture and line features of the deteriorated murals by using multi-dimensional gradient detection in the HSV space. Then, a guided filter operation is employed to highlight the disease (deteriorated) areas and meanwhile suppress other unwanted areas, which helps to extract the flaked areas or cracked lines from the digital murals. The filtered images are segmented by using an automatic threshold to obtain the initial masks of the mural disease areas. Next, we use a two-dimensional tensor voting technique to connect the discontinuous edge curves of the extracted disease areas. Afterwards, the masks of flaking and cracking areas can be generated after morphological processing. Finally, we obtained the calibration results by adding the masks to the original digital murals. Experimental results show that our method can rapidly and accurately calibrate the cracks and the earth layer flaking diseases in the ancient murals. As compared to existing calibration approaches, our method can achieve better performance in subjective visual quality and objective evaluation metrics. Moreover, the method does not need human-computer interaction. This research work provides a solid foundation for the following virtual and practical restoration of ancient murals.

End-to-End Retinex-Based Illumination Attention Low-Light Enhancement Network for Autonomous Driving at Night.

Low-light image enhancement is a preprocessing work for many recognition and tracking tasks for autonomous driving at night. It needs to handle various factors simultaneously including uneven lighting, low contrast, and artifacts. We propose a novel end-to-end Retinex-based illumination attention low-light enhancement network. Specifically, our proposed method adopts multibranch architecture to extract rich features for different depth levels. Meanwhile, we consider the features from different scales in built-in illumination attention module. We encode reflectance features and illumination features into latent space based on Retinex in each submodule, which could cater for highly ill-posed image decomposition tasks. It aims to enhance the desired illumination features under different receptive fields. Subsequently, we propose a memory gate mechanism to learn adaptively long-term and short-term memory. Their weight could control how many high-level and low-level features should be reserved. This method could improve the image quality from both different feature scales and feature levels. Comprehensive experiments on BDD10K and cityscapes datasets demonstrate that our proposed method outperforms various types of methods in terms of visual quality and quantitative metrics. We also show that our proposed method has certain antinoise capability and generalizes well without fine-tuning when dealing with unseen images. Meanwhile, our restoration performance is comparable to that of advanced computationally intensive models.1

Optimizing Local Alignment along the Seamline for Parallax-Tolerant Orthoimage Mosaicking

Orthoimage mosaicking with obvious parallax caused by geometric misalignment is a challenging problem in the field of remote sensing. Because the obvious objects are not included in the digital terrain model (DTM), large parallax exists in these objects. A common strategy is to search an optimal seamline between orthoimages, avoiding the majority of obvious objects. However, stitching artifacts may remain because (1) the seamline may still cross several obvious objects and (2) the orthoimages may not be precisely aligned in geometry when the accuracy of the DTM is low. While applying general image warping methods to orthoimages can improve the local geometric consistency of adjacent images, these methods usually significantly modify the geometric properties of orthophoto maps. To the best of our knowledge, no approach has been proposed in the field of remote sensing to solve the problem of local geometric misalignments after orthoimage mosaicking with obvious parallax. In this paper, we creatively propose a method to optimize local alignment along the seamline after seamline detection. It consists of the following main processes. First, we locate regions with geometric misalignments along the seamline based on the similarity measure. Second, for any one region, we find one-dimensional (1D) feature matches along the seamline using a semi-global matching approach. The deformation vectors are calculated for these matches. Third, these deformation vectors are robustly and smoothly propagated into the buffer region centered on the seamline by minimizing the associated energy function. Finally, we directly warp the orthoimages to eliminate the local parallax under the guidance of dense deformation vectors. The experimental results on several groups of orthoimages show that our proposed approach is capable of eliminating the local parallax existing in the seamline while preserving most geometric properties of digital orthophoto maps, and that it outperforms state-of-the-art approaches in terms of both visual quality and quantitative metrics.