High Perceptual Quality Research Articles

Cascaded Learning with Generative Adversarial Networks for Low Dose CT Denoising.

CT machines can be tuned in order to reduce the radiation dose used for imaging, yet reducing the radiation dose results in noisy images which are not suitable in clinical practice. In order for low dose CT to be used effectively in practice this issue must be addressed. Generative Adversarial Networks (GAN) have been used widely in computer vision research and have proven themselves as a powerful tool for producing images with high perceptual quality. In this work we use a cascade of two neural networks, the first is a Generative Adversarial Network and the second is a Deep Convolutional Neural Network. The first network generates a denoised sample which is then fine-tuned by the second network via residue learning. We show that our cascaded method outperforms related works and more effectively reconstructs fine structural details in low contrast regions of the image.

Benchmark of Bitrate Adaptation in Video Streaming

The HTTP adaptive streaming technique opened the door to cope with the fluctuating network conditions during the streaming process by dynamically adjusting the volume of the future chunks to be downloaded. The bitrate selection in this adjustment inevitably involves the task of predicting the future throughput of a video session, owing to which various heuristic solutions have been explored. The ultimate goal of the present work is to explore the theoretical upper bounds of the QoE that any ABR algorithm can possibly reach, therefore providing an essential step to benchmarking the performance evaluation of ABR algorithms. In our setting, the QoE is defined in terms of a linear combination of the average perceptual quality and the buffering ratio. The optimization problem is proven to be NP-hard when the perceptual quality is defined by chunk size and conditions are given under which the problem becomes polynomially solvable. Enriched by a global lower bound, a pseudo-polynomial time algorithm along the dynamic programming approach is presented. When the minimum buffering is given higher priority over higher perceptual quality, the problem is shown to be also NP-hard, and the above algorithm is simplified and enhanced by a sequence of lower bounds on the completion time of chunk downloading, which, according to our experiment, brings a 36.0% performance improvement in terms of computation time. To handle large amounts of data more efficiently, a polynomial-time algorithm is also introduced to approximate the optimal values when minimum buffering is prioritized. Besides its performance guarantee, this algorithm is shown to reach 99.938% close to the optimal results, while taking only 0.024% of the computation time compared to the exact algorithm in dynamic programming.

Image translation with dual‐directional generative adversarial networks

Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between input images and output images. However, due to the unstable training and limited training samples, many existing GAN-based works have difficulty in producing photo-realistic images. Herein, dual-directional generative adversarial networks are proposed, which consist of four adversarial networks, to produce images of high perceptual quality. In this framework, self-reconstruction strategy is used to construct auxiliary sub-networks, which impose more effective constraints on encoder-generator pairs. Using this idea, this model can increase the use ratio of paired data conditioned on the same dataset and obtain well-trained encoder-generator pairs with the help of the proposed cross-network skip connections. Moreover, the proposed framework not only produces realistic images but also addresses the problem where condition GAN produces sharp images containing many small, hallucinated objects. Training on multiple supervised datasets, convincing evidences are shown to prove that this model can achieve compelling results by latently learning a common feature representation. Qualitative and quantitative comparisons against other methods, demonstrate the effectiveness and superiority of the method.

Masquerade attack on biometric hashing via BiohashGAN

Masquerade attack on biometric hashing, which reconstructs the original biometric image from the given hashcode, has been given much attention recently. It is mainly used to validate the security of biometric recognition system or expand existing biometric databases like face or iris. However, an existing state-of-the-art method tends to ignore the perceptual quality of synthesized biometric images in the attack, and consequently, the synthetic images can be easily differentiated from real images. To obtain the high-perceptual-quality image which can simultaneously pass the validation of recognition system, we introduce a new target combining semantic invariability in hashing space and perceptual similarity in biometric space. In order to simulate the mapping from images to hashcodes and tackle the derivative problem related to discrete hashcodes in hashing space, we propose a DNN-based network named SimHashNet. Then we incorporate the SimHashNet into a generative adversarial network as our model named BiohashGAN to generate synthetic images form hashcodes. Experiment result on dataset CASIA-IrisV4.0-Interval and CMU PIE demonstrates that the synthetic images obtained from our model can pass the validation of recognition system and simultaneously maintain high perceptual quality.

Multisensor Remote Sensing Imagery Super-Resolution with Conditional GAN

Despite the promising performance on benchmark datasets that deep convolutional neural networks have exhibited in single image super-resolution (SISR), there are two underlying limitations to existing methods. First, current supervised learning-based SISR methods for remote sensing satellite imagery do not use paired real sensor data, instead operating on simulated high-resolution (HR) and low-resolution (LR) image-pairs (typically HR images with their bicubic-degraded LR counterparts), which often yield poor performance on real-world LR images. Second, SISR is an ill-posed problem, and the super-resolved image from discriminatively trained networks with l p norm loss is an average of the infinite possible HR images, thus, always has low perceptual quality. Though this issue can be mitigated by generative adversarial network (GAN), it is still hard to search in the whole solution-space and find the best solution. In this paper, we focus on real-world application and introduce a new multisensor dataset for real-world remote sensing satellite imagery super-resolution. In addition, we propose a novel conditional GAN scheme for SISR task which can further reduce the solution-space. Therefore, the super-resolved images have not only high fidelity, but high perceptual quality as well. Extensive experiments demonstrate that networks trained on the introduced dataset can obtain better performances than those trained on simulated data. Additionally, the proposed conditional GAN scheme can achieve better perceptual quality while obtaining comparable fidelity over the state-of-the-art methods.

Performance Analysis of Diversity Selection Combining on Radio Frequency for Image Transmission

This paper describes a simulation of the image transmission system with diversity selection combining (DSC) method on radio frequency. DSC is used to obtain multiple data streams corresponding to the transmitted image at the receiver in order to combat errors during image transmission on wireless channels. These individual image data streams are combined to form a composite image with higher perceptual quality. When DSC is used, it is expected that it can affect the number of errors, so that the signal transmitted to the wireless channels becomes more robust against interference caused by transmission channels. By using DSC technique, it can increase the quality of image transmission at the receiver. The result of the experiment which was carried out to test bit error rate (BER) and peak signal to noise ratio (PSNR) on Lena image with rate 0.8 bpp by using DSC on radio frequency in Rayleigh fading channel showed that there was a decrease in BER at the average of 0.0152 or 3.39%, while for PSNR there was an increase at the average of 1.6737 dB or 6.73%when comparing to the system without DSC.

Suitability of VVC and HEVC for Video Telehealth Systems

Video compression in medical video streaming is one of the key technologies associated with mobile healthcare. Seamless delivery of medical video streams over a resource constrained network emphasizes the need of a video codec that requires minimum bitrates and maintains high perceptual quality. This paper presents a comparative study between High Efficiency Video Coding (HEVC) and its potential successor Versatile Video Coding (VVC) in the context of healthcare. A large-scale subjective experiment comprising of twenty-four non-expert participants is presented for eight different test conditions in Full High Definition (FHD) videos. The presented analysis highlights the impact of compression artefacts on the perceptual quality of HEVC and VVC processed videos. Our results and findings show that VVC clearly outperforms HEVC in terms of achieving higher compression, while maintaining high quality in FHD videos. VVC requires upto 40% less bitrate for encoding an FHD video at excellent perceptual quality. We have provided rate-quality curves for both encoders and a degree of overlap across both codecs in terms of perceptual quality. Overall, there is a 71% degree of overlap in terms of quality between VVC and HEVC compressed videos for eight different test conditions.

Group Perceptual Quality Optimization for Multi-Channel Image Encoding Systems Based on Adaptive Hyper Networks

Images and short videos that produced by social networks surge in recent years. Image/Video encoders, such as JPEG and H.264, are indispensably involved to reduce the transmitting bandwidth. However, based on our observation, the encoding parameters and their candidates are often preset to fixed values (or fixed candidate values) in real-world scenarios, which might not be the optimal bandwidth allocation strategy. Considering that, we propose an efficient group quality optimization (GQO) framework for multi-channel image/video encoding systems in which the encoding parameters are configured in a perceptual-quality-driven manner. The GQO framework employs adaptive hyper network to predict the relationships between encoding parameters, transmitting resources, and perceptual qualities, i.e., just taking the pristine image as input, the adaptive hyper network could accurately yield a global overview of perceptual quality and transmitting resource varied along encoding parameters. A step-by-step optimization procedure is then employed to search the optimal encoding parameter for each channel so that overall perceptual quality could be maximized under limited transmitting resource. Experimental results demonstrate the proposed GQO framework could achieve higher perceptual quality whilst maintain the same bandwidth compared to traditional allocation strategies where encoding parameters are preset.

RETRACTED: Research on segmentation of sports images based on GMMLS method

The transmission is a major economic game broadcast, not just playing games, video files, and can be tracked by image segmentation. However, from the current situation, it can see the video segmentation technology is still immature. Image segmentation is a technique that divides the original image into several different categories. Many of the possible solutions, it can be made available for image segmentation into different motion image segmentation qualities and each, some of the categories. The first step in the preprocessing is to remove the noise from the image. The second level is designed to be based on the proposed Gaussian Mixed Model Level Set (GMMLS), which uses multi-level threshold technology for image segmentation. The GMMLS is expected to realize an image processing method because it is substantially parallel to and consistent with the regional image division processing. And the third process, feature extraction, completion and final motion image classification. Experiments have demonstrated the feasibility of artificial life methods in both luminance and color images. Experimental results show that the algorithm provides fast segmentation with high perceptual segmentation quality while moving the image split view.

PVC-SLP: Perceptual Vibrotactile-Signal Compression Based-on Sparse Linear Prediction

Developing a signal compression technique that is able to achieve a low bit rate while maintaining high perceptual signal quality is a classical signal processing problem vigorously studied for audio, speech, image, and video type of signals. Yet, until recently, there has been limited effort directed toward the compression of vibrotactile signals, which represent a crucial element of rich touch (haptic) information. A vibrotactile signal; produced when stroking a textured surface with a tool-tip or bare-finger; like other signals contains a great deal of redundant and imperceptible information that can be exploited for efficient compression. This paper presents PVC-SLP, a vibrotactile perceptual coding approach. PVC-SLP employs a model of tactile sensitivity; called ASF (Acceleration Sensitivity Function); for perceptual coding. The ASF is inspired by the four channels model that mediate the perception of vibrotactile stimuli in the glabrous skin. The compression algorithm introduces sparsity constraints in a linear prediction scheme both on the residual and the predictor coefficients. The perceptual quantization of the residual is developed through the use of ASF. The quantization parameters of the residual and the predictor coefficients were jointly optimized; by means of both squared error and perceptual quality measures; to find the sweet spot of the rate-distortion curve. PVC-SLP coding performance is evaluated using two publicly available databases that collectively comprise 1281 vibrotactile signals covering 193 material classes. Furthermore, we compare PVC-SLP with a recent vibrotactile compression method and show that PVC-SLP perceptually outperforms existing method by a sizable margin. Most recently, PVC-SLP has been selected to become part of the haptic codec standard currently under preparation by IEEE P1918.1.1, aka Haptic Codecs for the Tactile Internet.

MCFD: A Hardware-Efficient Noniterative Multicue Fusion Demosaicing Algorithm

Color demosaicing is an essential step in the camera image processing pipeline, especially on hardware platforms for real-time video applications. Though many demosaicing algorithms have been proposed in the past two decades, there remains a substantial gap between industrial needs and academic research. On one hand, industry requires a high perceptual quality, artifact-free, and low-cost demosaicing algorithm that is noniterative and small window based ready to be implemented on a hardware platform with a limited line buffer. On the other hand, academia is targeting high PSNR/SSIM, with the computation cost and line buffer receiving second priority, and often a frame buffer and iterative operation are used. The cost, large line buffer and iteration requirement make most existing demosaicing algorithms inapplicable on hardware platforms. In this paper, we introduce a novel low-cost demosaicing algorithm to narrow the gap. We keep the operation window size and computation cost as the first priority and achieve both high PSNR/SSIM and visual perceptual quality compared to previous state-of-the-art methods on standard test datasets. We fully investigate the a priori knowledge of natural scene raw images and find several key cues that are beneficial to demosaicing. Our demosaicing algorithm is a smart fusion of these useful cues. Furthermore, we solve typical demosaicing issues that occur in many traditional methods, including false color artifacts, T-section closing, and zippering artifacts. The proposed method has no learning stage, no iteration operation, a small line buffer and a limited number of parameters, so it can easily be applied to hardware platforms.

SDP-GAN: Saliency Detail Preservation Generative Adversarial Networks for High Perceptual Quality Style Transfer.

The paper proposes a solution to effectively handle salient regions for style transfer between unpaired datasets. Recently, Generative Adversarial Networks (GAN) have demonstrated their potentials of translating images from source domain X to target domain Y in the absence of paired examples. However, such a translation cannot guarantee to generate high perceptual quality results. Existing style transfer methods work well with relatively uniform content, they often fail to capture geometric or structural patterns that always belong to salient regions. Detail losses in structured regions and undesired artifacts in smooth regions are unavoidable even if each individual region is correctly transferred into the target style. In this paper, we propose SDP-GAN, a GAN-based network for solving such problems while generating enjoyable style transfer results. We introduce a saliency network, which is trained with the generator simultaneously. The saliency network has two functions: (1) providing constraints for content loss to increase punishment for salient regions, and (2) supplying saliency features to generator to produce coherent results. Moreover, two novel losses are proposed to optimize the generator and saliency networks. The proposed method preserves the details on important salient regions and improves the total image perceptual quality. Qualitative and quantitative comparisons against several leading prior methods demonstrates the superiority of our method.

$$\hbox {S}^2\hbox {RGAN}$$: sonar-image super-resolution based on generative adversarial network

As an important display mode of underwater environments, the sonar image has limitations on the resolution, which often leads to problems with low resolution of underwater objects. Therefore, the image super-resolution algorithm is needed to transform the images from low-resolution to high-resolution. It can improve the visual effect and contribute to subsequent processing such as 3D reconstruction and object recognition. This paper proposes a method for sonar image super-resolution based on generative adversarial networks (GAN). By comparing the super-resolution effects of various interpolation and convolutional neural network algorithms on sonar images, a Residual-in-Residual Dense Block network is employed as the generator of GAN since it has the low distortion and high perceptual quality. Because the sonar image training set does not have enough data, the generator utilizes the transfer learning on the sonar images to produce an optimized network model which is more suitable for super-resolution of sonar image. The VGG19 network is employed as the discriminator. In addition, the perceptual loss is introduced into the loss function of $$\hbox {S}^2\hbox {RGAN}$$ to further improve the perceptual quality of super-resolution images. The experimental results indicate that the proposed $$\hbox {S}^2\hbox {RGAN}$$ shows excellent performance. The generated super-resolution images of $$\hbox {S}^2\hbox {RGAN}$$ have the remarkable advantages of both lower distortion and higher perceptual quality comparing with other methods. Because $$\hbox {S}^2\hbox {RGAN}$$ focuses more on the reality and overall visual effect of super-resolution sonar images, it is suitable for various underwater situations.

Image compression for quality 3D reconstruction

A 3D mesh can be reconstructed from multiple viewpoint images or from a single structured light image. Lossy compression of such images by standard techniques such as JPEG at high compression ratios lead to 3D reconstruction being adversely affected by artifacts and missing vertices. In this paper we demonstrate an improved algorithm capable of high compression ratios without adversely affecting 3D reconstruction and with minimum data loss. The compression algorithm starts by applying block DCT over the input image, and the transformed data being quantized using an optimized quantization matrix. The quantized coefficients of each block are arranged as a 1D array and saved with other block’s data in a larger matrix of coefficients. The DC coefficients are subject to a first order difference whose values are referred to as residual array. The AC coefficients are reduced by eliminating zeros and saving the non-zero values in a reduced coefficients array using a mask of 0 (for a block of zeros) and 1 (for a block of non-zeros). Finally, arithmetic coding is applied to both coefficients and residual arrays. At decompression stage, the coefficients matrix is regenerated by scanning the coefficients array and examining the headers to substitute zero and non-zero data. This matrix is then added to the residual array to obtain the original DC values. The IDCT is then applied to obtain the original image. The proposed algorithm has been tested with images of varying sizes in the context of 3D reconstruction. Results demonstrate that our proposed algorithm is superior to traditional JPEG at higher compression ratios with high perceptual quality of images and the ability to reconstruct the 3D models more effectively, both for structured light images and for sequences of multiple viewpoint images.

Learning wavelet coefficients for face super-resolution

Face image super-resolution imaging is an important technology which can be utilized in crime scene investigations and public security. Modern CNN-based super-resolution produces excellent results in terms of peak signal-to-noise ratio and the structural similarity index (SSIM). However, perceptual quality is generally poor, and the details of the facial features are lost. To overcome this problem, we propose a novel deep neural network to predict the super-resolution wavelet coefficients in order to obtain clearer facial images. Firstly, this paper uses prior knowledge of face images to manually emphases relevant facial features with more attention. Then, a linear low-rank convolution in the network is used. Finally, image edge features from canny detector are applied to enhance super-resolution images during training. The experimental results show that the proposed method can achieve competitive PSNR and SSIM and produces images with much higher perceptual quality.

Deep Battery Saver: End-to-End Learning for Power Constrained Contrast Enhancement

Due to the problems of power-hungry displays and limited battery life in electronic devices, the concept of “green computing,” which entails a reduction in power consumption, is proposed. One often seen green computing is the power-constrained contrast enhancement (PCCE), yet it is much more challenging because of the noticeable local intensity suppressions in images. This paper aims at developing an image-quality-lossless end-to-end learning network called deep battery saver to achieve power savings in emissive displays, i.e., produce power-saved images with high perceptual quality and less power consumption. Built upon the end-to-end network of the displayed image, we propose a variational loss function for enhancing the visual quality and suppressing the power consumption, simultaneously. The basic idea is to integrate both high-level perceptual losses and low-level pixel losses by a deep residual convolutional neural network (CNN) over a devised variational loss function with strong human perceptual consistency. Such deep residual CNN network leads to a visually pleasing image representation during the suppression of power consumption. Experimental results demonstrated the superiority of our deep battery saver to existing PCCE methods.

Image Super-Resolution Reconstruction Using Generative Adversarial Networks Based on Wide-Channel Activation

In recent years, residual learning has shown excellent performance on convolutional neural network (CNN)-based single-image super-resolution (SISR) tasks. However, CNN-based SISR approaches have focused mainly on the design of deep architectures, and the rectified linear units (ReLUs) used in these networks hinder shallow-to-deep information transfer. As a result, these methods are unable to utilize some shallow information, and improving model performance is difficult. To solve the above issues, this paper proposes an image SR reconstruction method based on a generative adversarial network with a residual dense architecture. First, before ReLU activation, the number of feature channels is expanded by a factor of 6~9 using a 1 × 1 convolutional layer, which improves the utilization of shallow information. Next, the original discriminator is replaced with a relativistic average discriminator, thereby improving the authenticity of the discriminative network. Finally, preactivation features are used to improve the perceptual loss, thus providing stronger monitoring for brightness consistency and texture restoration. Experimental results show that the proposed algorithm improves the utilization of shallow information in a deep network. Structural similarity (SSIM) index evaluations show that the overall utilization of shallow information is increased by 105.52%. In addition, the average runtime is 0.42 sec/frame, nearly 3.6 times faster than those of traditional methods. Moreover, the recovered images have an average natural image quality evaluator value of 3.4 and high perceptual quality, showing that the proposed method is suitable for image reconstruction applications in fields such as agriculture and medicine.

Linking Bitstream Information to QoE: A Study on Still Images Using HEVC Intra Coding

The coding tools used in image and video encoders aim at high perceptual quality for low bitrates. Analyzing the results of the encoders in terms of quantization parameter, image partitioning, prediction modes or residuals may provide important insight into the link between those tools and the human perception. As a first step, this contribution analyzes the possibility to transcode reference images of three well-known image databases, i.e. IRCCyN/IVC, LIVE and TID2013, from their original, older formats to HEVC; thus creating a homogeneous database of 327 HEVC encoded images accompanied with bitstream parameters and values obtained from objective and subjective assessments. Secondly, it analyzes some of the HEVC intra coding parameters regarding their influence on the image quality by using machine learning, namely Support Vector Machine - Regression.

SD-GAN: Saliency-Discriminated GAN for Remote Sensing Image Superresolution

Recently, convolutional neural networks have shown superior performance in single-image superresolution. Although existing mean-square-error-based methods achieve high peak signal-to-noise ratio (PSNR), they tend to generate oversmooth results. Generative adversarial network (GAN)-based methods can provide high-resolution (HR) images with higher perceptual quality, but produce pseudotextures in images, which generally leads to lower PSNR. Besides, different regions in remote sensing images (RSIs) reflect discrepant surface topography and visual characteristics. This means a uniform reconstruction strategy may not be suitable for all targets in RSIs. To solve these problems, we propose a novel saliency-discriminated GAN for RSI superresolution. First, hierarchical weakly supervised saliency analysis is introduced to compute a saliency map, which is subsequently employed to distinguish the diverse demands of regions in the following generator and discriminator part. Different from previous GANs, the proposed residual dense saliency generator takes saliency maps as a supplementary condition in the generator. Simultaneously, combining the characteristic of RSIs, we design a new paired discriminator to enhance the perceptual quality, which measures the distance between generated images and HR images in salient areas and nonsalient areas, respectively. Comprehensive evaluations validate the superiority of the proposed model.

A novel Hexa data encoding method for 2D image crypto-compression

We proposed a novel method for 2D image compression-encryption whose quality is demonstrated through accurate 2D image reconstruction at higher compression ratios. The method is based on the DWT-Discrete Wavelet Transform where high frequency sub-bands are connected with a novel Hexadata crypto-compression algorithm at compression stage and a new fast matching search algorithm at decoding stage. The novel crypto-compression method consists of four main steps: 1) A five-level DWT is applied to an image to zoom out the low frequency sub-band and increase the number of high frequency sub-bands to facilitate the compression process; 2) The Hexa data compression algorithm is applied to each high frequency sub-band independently by using five different keys to reduce each sub-band to1/6of its original size; 3) Build a look up table of probability data to enable decoding of the original high frequency sub-bands, and 4) Apply arithmetic coding to the outputs of steps (2) and (3). At decompression stage a fast matching search algorithm is used to reconstruct all high frequency sub-bands. We have tested the technique on 2D images including streaming from videos (YouTube). Results show that the proposed crypto-compression method yields high compression ratios up to 99% with high perceptual quality images.