Perceived Quality Degradation Research Articles

Blind Image Quality Assessment via Deep Response Feature Decomposition and Aggregation

Image quality is related to image content and distortion information. Most learning-based image quality assessment (IQA) methods extract quality-oriented features with auxiliary tasks like detecting the distortion type and level. However, the perceptual quality degradation caused by the same distortion type and level varies substantially for different content in an image. To deal with this problem, in this paper, we propose a blind IQA method based on Deep Response fEAture decoMposition and aggregation (DREAM), which considers two factors affecting the image quality simultaneously. First, we use a convolutional neural network (CNN) to extract the basic features from the input image. Second, several parallel fully connected (FC) layers are employed to decompose these basic features into response features related to the image content, distortion type, and distortion level. Third, the graph attention network (GAT) is leveraged to aggregate these response features corresponding to the visual quality. Finally, a regression network is used to predict the quality score. The success of our method lies in the feature decomposition to obtain the response features of different content to a specific distortion in the given distorted image and the quality-oriented features obtained by feature aggregation using the internal relation of these response features. Experimental results indicate that our proposed DREAM achieves state-of-the-art (SOTA) performance on both synthetic and authentic distortion IQA datasets.

Frequency Spectrum Modification Process-Based Anti-Collusion Mechanism for Audio Signals.

The collusion attack combines multiple multimedia files into one new file to erase the user identity information. The traditional anti-collusion methods (which aim to trace the traitors) can defend the collusion attack, but they cannot well defend some hybrid collusion attacks (e.g., a collusion attack combined with desynchronization attacks). To address this issue, we propose a frequency spectrum modification process (FSMP) to defend the collusion attack by significantly downgrading the perceptual quality of the colluded file. The severe perceptual quality degradation can demotivate the attackers from launching the collusion attack. Because FSMP is orthogonal to the existing traitor-trace-based methods, it can be combined with the existing methods to provide a double-layer protection against different attacks. In FSMP, after several signal processing procedures (e.g., uneven framing and smoothing), multiple signals (called FSMP signals) can be generated from the host signal. Launching collusion attack using the generated FSMP signals would lead to the energy disturbance and attenuation effect (EDAE) over the colluded signals. Due to the EDAE, FSMP can significantly degrade the perceptual quality of the colluded audio file, thereby thwarting the collusion attack. In addition, FSMP can well defend different hybrid collusion attacks. Theoretical analysis and experimental results confirm the validity of the proposed method.

Perceptual Assessment of Image and Depth Quality of Dynamically Depth-Compressed Scene for Automultiscopic 3D Display.

This article discusses the depth range which automultiscopic 3D (A3D) displays should reproduce for ensuring an adequate perceptual quality of substantially deep scenes. These displays usually need sufficient depth reconstruction capabilities covering the whole scene depth, but due to the inherent hardware restriction of these displays this is often difficult, particularly for showing deep scenes. Previous studies have addressed this limitation by introducing depth compression that contracts the scene depth into a smaller depth range by modifying the scene geometry, assuming that the scenes were represented as CG data. The previous results showed that reconstructing only a physical depth of 1 m is needed to show scenes with much deeper depth and without large perceptual quality degradation. However, reconstructing a depth of 1 m is still challenging for actual A3D displays. In this study, focusing on a personal viewing situation, we introduce a dynamic depth compression that combines viewpoint tracking with the previous approach and examines the extent to which scene depths can be compressed while keeping the original perceptual quality. Taking into account the viewer's viewpoint movements, which were considered a cause of unnaturalness in the previous approach, we performed an experiment with an A3D display simulator and found that a depth of just 10 cm was sufficient for showing deep scenes without inducing a feeling of unnaturalness. Next, we investigated whether the simulation results were valid even on a real A3D display and found that the dynamic approach induced better perceptual quality than the static one even on the real A3D display and that it had a depth enhancing effect without any hardware updates. These results suggest that providing a physical depth of 10 cm on personalized A3D displays is general enough for showing any deeper 3D scenes with appealing subjective quality.

Intermediate-Layer Transferable Adversarial Attack With DNN Attention

The widespread deployment of deep learning models in practice necessitates an assessment of their vulnerability, particularly in security-sensitive areas. As a result, transfer-based adversarial attacks have elicited increasing interest in assessing the security of deep learning models. However, adversarial samples usually exhibit poor transferability over different models because of overfitting of the particular architecture and feature representation of a source model. To address this problem, the Intermediate Layer Attack with Attention guidance (IAA) is proposed to alleviate overfitting and enhance the black-box transferability. The IAA works on an intermediate layer <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> of the source model. Guided by the model’s attention to the features of layer <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> , the attack algorithm seeks and undermines the key features that are likely to be adopted by diverse architectures. Significantly, IAA focuses on improving existing white-box attacks without introducing significant visual perceptual quality degradation. Namely, IAA maintains the white-box attack performance of the original algorithm while significantly enhancing its black-box transferability. Extensive experiments on ImageNet classifiers confirmed the effectiveness of our method. The proposed IAA outperformed all state-of-the-art benchmarks in various white-box and black-box settings.

Concealment Support and Error Resilience for HEVC to Improve Consumer Quality of Experience

Consumer electronics have started to support 8K/4K/UHD/SUHD videos in order to improve the perceived QoE. It is beneficial to assess the consumer’s perceived quality degradation at the output prior to transmitting the video in an error prone channel, to include concealment support and error resilient features to minimize these artifacts. While existing error resilient schemes have attempted to include source coding error and the channel’s error propagation, they have ignored the error resilience and concealment accuracy at the decoder when encoding the video which allows for end-to-end error capture and mitigation to improve consumer QoE. This paper proposes a two-pass codec encoding scheme that determines the motion vectors and the coding modes which not only mitigate the source coding error and the channel error propagation, but also improve the accuracy of the error concealment operation. The proposed method incorporates an end-to-end probabilistic distortion model and utilizes the concealment error in the subsequently encoding frames while estimating the motion vectors for the current frame. The experiment achieves up to 1.47dB PSNR gain for diverse video contents in comparison to the state-of-the-art techniques at a 10% packet error rate. This objective gain was also supported by the user perceived QoE.

Zonal MPEG-2

This paper presents a simple but effective approach to improve the performance of MPEG-2. This study is motivated by (i) the energy compaction property in DCT and (ii) the property of human visual system. Both (i) and (ii) indicate that low-frequency coefficients are more important than others in the DCT transformed blocks. Moreover, the quantized transformed coefficients in high-frequency region are zero or approximate zero in general. Those coefficients contribute nothing or little to PSNR and may waste some bit rate in the MPEG-2. Therefore, the zonal coding scheme is applied to the MPEG-2 where some high-frequency coefficients are dis- carded. The proposed approach is called zonal MPEG-2 which is totally compatible with the tra- ditional MPEG-2. The zonal MPEG-2 is verified by examples and shown to be able to signifi- cantly reduce the bit rate with acceptable perceptual quality degradation when compared with the traditional MPEG-2.

Mixed distortion image enhancement method based on joint of deep residuals learning and reinforcement learning

The images distortion leads to the loss of image information and the degradation of perceptual quality. To solve this problem, we investigate a novel mixed distortion image enhancement method based on the parallel network combines deep residual and reinforcement learning. The no-reference image quality assessment algorithm is used to determine the type and level of distorted images accurately. According to the type of distortion, the mixed distortion images enter one of the subsequent parallel joint learning networks automatically. In the joint learning framework, we prepare different residual networks to handle specialized restoration assignments including deblurring, denoising, or JPEG compression. Simultaneously, reinforcement learning then learns a policy to select the next best restoration tasks to progressively restore the quality of a corrupted image. Our method is capable of restoring images corrupted with complex mixed distortions in a more parameter-efficient manner in comparison to conventional networks. The extensive experiments on synthetic and real-world images validate the superior performances of the proposed method.

A Compressed-Domain Robust Video Watermarking Against Recompression Attack

With the development of communication networks and the widespread use of mobile terminals, videos are increasingly shared and distributed among mobile users. During the process, the video is recompressed to a certain file size on the sending side and sent to the receiving side via the server. This makes robust video watermark with low complexity to resist recompression attacks become an important issue to address. The proposed video watermarking method in this article is specially designed for resisting recompression attacks when quantization parameter (QP) is greatly increased. In the proposed method, by using the texture information and motion information of video, the invariance of video content under different quantization parameters could be found to help improve the anti-recompression attack ability of video watermark. Moreover, the proposed framework does not use a location map which has security risk to locate the watermark. It aims at finding the optimal location of watermark embedding adaptively according to the feature of video content itself. The experiment results indicate that the proposed video watermarking method can not only achieve greater robustness against recompression attack but also effectively limit the degradation in video perceptual quality.

Desynchronization Attacks Resilient Watermarking Method Based on Frequency Singular Value Coefficient Modification

Desynchronization is a very challenging type of attack in audio watermarking. The traditional singular value decomposition (SVD) based audio watermarking methods embed the watermark information by modifying the singular value of individual segment, which have little resistance against desynchronization attacks. In this paper, we propose a novel frequency singular value coefficient (FSVC) feature, which reflects the ratio between the singular values of two consecutive segments and is insensitive to desynchronization attacks, to carry the watermark bits. To our best knowledge, it is the first time that the ratio between singular values is employed for audio watermarking. In the proposed method, the discrete cosine transform (DCT) is performed on two consecutive segments of the host audio signal and SVD is applied to the DCT coefficients of the mid frequency band of each segment to extract the FSVC. Then the watermark bits are embedded by adjusting the values of the FSVC. The watermark embedding procedure is optimized to minimize the perceptual quality degradation and an error buffer is created to enhance the robustness. As a result, the proposed method can achieve a much higher embedding capacity than the existing methods tackling desynchronization attacks. The impact of desynchronization and common signal processing attacks on the proposed watermarking method is mathematically modeled, and the effectiveness of the proposed method against these attacks is theoretically and experimentally validated.

Segmental DCT Coefficient Reversal Based Anti-Collusion Audio Fingerprinting Mechanism

Collusion attacks are challenging to tackle in audio fingerprinting. A new direction to resist collusion attacks is to degrade the perceptual quality of the colluded files so that these files cannot be reused. The existing method in this direction has low embedding capacity and limited anti-collusion performance when the number of colluders is odd. In this letter, we present an anti-collusion mechanism that has a higher embedding capacity and can significantly degrade the perceptual quality of the colluded files regardless of the number of colluders. In the proposed mechanism, we first segment the host audio file into frames and perform the discrete cosine transform (DCT) on each frame. Then multiple fingerprint bits are embedded into each frame by reversing the DCT coefficients of the corresponding frequency band. As a result, when a collusion attack occurs, our proposed embedding mechanism can introduce perceptibly annoying differences between frames in the colluded file, which leads to severe perceptual quality degradation. Theoretical analysis and experimental results validate the superiority of the proposed anti-collusion mechanism.

Size-Invariant Visual Cryptography With Improved Perceptual Quality for Grayscale Image

The reconstructed image of the size-invariant visual cryptography (VC) is inevitably accompanied by the loss of secret image information and the degradation of perceptual quality. Here, the halftone technique comes to the forefront since it can realistically simulate the grayscale image from a discrete binary image. Thus, by combining VC sharing with grayscale image halftone technique, this paper proposes a size-invariant VC scheme for grayscale image underpinned by the efficient direct binary search (EDBS) algorithm, in which the multi-pixel encryption VC sharing is adopted into the EDBS halftone process. Through local optimizations and global iterations, the optimal reconstructed image is obtained. To further enhance the contrast of the reconstructed image with limited computational power, the image information is probabilistically extracted according to the inverse mapping in the codebook. It is theoretically proved that the proposed scheme is as secure as the traditional VC, while its effectiveness is validated through experiments and comparative analyses.

End User Video Quality Prediction and Coding Parameters Selection at the Encoder for Robust HEVC Video Transmission

Along with the rapid increase in the availability of high-quality video formats such as high definition (HD), ultra HD, and high dynamic range, a huge demand for data rates during their transmission has become inevitable. Consequently, the role of video compression techniques has become crucially important in the process of mitigating the data rate requirements. Even though the latest video codec high efficiency video coding (HEVC) has succeeded in significantly reducing the data rate compared to its immediate predecessor H.264/advanced video coding, the HEVC coded videos in the meantime have become even more vulnerable to network impairments. Therefore, it is equally important to assess the consumers’ perceived quality degradation prior to transmitting HEVC coded videos over an error-prone network, and to include error resilient features so as to minimize the adverse effects of those impairments. To this end, this paper proposes a probabilistic model which accurately predicts the overall distortion of the decoded video at the encoder followed by an accurate QP– $\lambda $ relationship which can be used in the rate-distortion optimization (RDO) process. During the derivation process of the probabilistic model, the impacts from the motion vectors, the pixels in the reference frames, and the clipping operations are accounted, and consequently, the model is capable of minimizing the prediction error to as low as 3.11%, whereas the state-of-the-art methods cannot reach below 20.08%, under identical conditions. Furthermore, the enhanced RDO process has resulted in 21.41%–43.59% improvement in the BD-rate compared to the state-of-the-art error resilient algorithms.

Depth perceptual quality assessment for symmetrically and asymmetrically distorted stereoscopic 3D videos

Depth perception is one of the most important features provided by stereoscopic 3D videos, which is also the major difference to 2D videos. Its quality, called depth perceptual quality or depth quality, is one of the key factors that directly affects the quality of experience of 3D videos. Since 3D video shall be compressed, transmitted and reconstructed in 3D video system, distortion introduced from compression and processing affects the video quality as well as the depth perceptual quality. However, understandings of how the distortions affect the depth perception and how to evaluate the depth perceptual quality of 3D video remain limited. In this paper, subjective experiments are firstly conducted for investigating the compression and processing distortions’ impacts on the perceived depth quality of 3D videos. The subjective experiments over the datasets show that the loss of video details can cause the degradation of the depth perceptual quality in monocular and binocular perspectives. In addition, the relationships between video quality and depth quality for both symmetric and asymmetric distorted 3D stereoscopic videos are analyzed. Both the subjective video quality and depth quality scores are released and available for public. Secondly, an objective depth quality assessment algorithm is proposed for measuring the depth perceptual quality degradation of symmetrically and asymmetrically distorted stereoscopic 3D videos. The experimental results demonstrate that the proposed metric has better consistency with the subjective scores of human vision system compared with the state-of-the-art schemes.

High-Performance Self-Synchronous Blind Audio Watermarking in a Unified FFT Framework

This paper presents a blind audio watermarking method that uses two different schemes to hide binary bits and auxiliary information within separate ranges of a fast Fourier transform (FFT) sequence. An adaptive vector norm modulation (AVNM) scheme is introduced to achieve a satisfactory balance of imperceptibility, robustness, and payload capacity. An improved spread spectrum (ISS) scheme is developed to produce a striking correlation peak, which facilitates the detection of synchronization codes in the FFT domain. The combination of robust audio segment extraction and recursive FFT makes it possible to execute these two FFT-based schemes in tandem on a sample-by-sample basis. The experiment results confirm that watermark embedding causes merely a negligible degradation in perceptual quality. A detectability test proved the effectiveness of the ISS scheme in self-synchronization as well as hiding auxiliary data. Three versions of AVNM with capacities ranging from 344.53 to 1033.59 bits per second were demonstrated. Compared with six recently developed schemes, AVNM exhibited advantages in terms of negligible quality distortion, flexible payload capacity, and excellent robustness against a variety of common signal processing attacks.

Perceptual brightness-based inverse tone mapping for high dynamic range imaging

This paper proposes a novel high dynamic range imaging method based on inverse tone mapping for generating a high dynamic range image from a single low dynamic range image. With the increasing expectations for image quality, displays providing high dynamic range images have been released. However, although low dynamic range images are currently commonly used, they do not use the high representation capability of high dynamic range displays. Moreover, high dynamic range contents are scarce. Therefore, high dynamic range imaging techniques are gaining attention. The proposed method involves adaptive specular region detection to determine the bright and dark areas for high dynamic range imaging, output mapping range decision to consider perceptual brightness, final mapping-level decision to prevent degradation of perceptual image quality, and inverse pixel mapping for luminance mapping. The simulation results of the proposed method yielded an HDR-VDP-2 quality score of 59.51, which was higher than those of the conventional methods. Furthermore, the execution time of the proposed algorithm was 0.66 s, which had the lowest complexity compared to the conventional methods. For subjective performance evaluation, we analyzed the perceived image quality on the display with the DRIM, which confirmed the superiority of the proposed method over the conventional methods.

Novel Adaptive Laser Scanning Method for Point Clouds of Free-Form Objects.

Laser scanners are widely used to collect coordinates, also known as point-clouds, of three-dimensional free-form objects. For creating a solid model from a given point-cloud and transferring the data from the model, features-based optimization of the point-cloud to minimize the number if points in the cloud is required. To solve this problem, existing methods mainly extract significant points based on local surface variation of a predefined level. However, comprehensively describing an object’s geometric information using a predefined level is difficult since an object usually has multiple levels of details. Therefore, we propose a simplification method based on a multi-level strategy that adaptively determines the optimal level of points. For each level, significant points are extracted from the point cloud based on point importance measured by both local surface variation and the distribution of neighboring significant points. Furthermore, the degradation of perceptual quality for each level is evaluated by the adjusted mesh structural distortion measurement to select the optimal level. Experiments are performed to evaluate the effectiveness and applicability of the proposed method, demonstrating a reliable solution to optimize the adaptive laser scanning of point clouds for free-forms objects.

Patchwork-Based Multilayer Audio Watermarking

A multilayer watermarking system is a system that is able to embed watermarks to a host media signal repeatedly in an overlaying manner, without incurring troubles in extracting the watermarks in each layer. In this paper, we present a novel patchwork-based audio watermarking algorithm that can embed and extract watermark bits successfully in such a multilayer framework. In the proposed method, a new watermark embedding algorithm is designed to ensure that the embedded watermarks in a certain layer do not affect the detection of watermarks in other layers. Adding multiple layers of watermark bits inevitably reduces the perceptual quality. However, to minimize the perceptual quality degradation in multilayer watermarking, the audio fragments for watermark embedding are selected from a set of specially arranged discrete cosine transform coefficients of the host audio signal. Watermark embedding is achieved by modifying the mean values of selected sample fragments. With the use of an embedding error buffer, the proposed system can withstand a wide range of common attacks. To maintain the balance between the perceptual quality and robustness, watermark embedding strength is adjusted according to the specific layer used. The proposed multilayer scheme ensures the independence of the processing in different layers. The effectiveness of the proposed system is demonstrated and verified by extensive simulation results.

An Efficient Framework for Compressed Domain Watermarking in P Frames of High-Efficiency Video Coding (HEVC)--Encoded Video

Digital watermarking has received much attention in recent years as a promising solution to copyright protection. Video watermarking in compressed domain has gained importance since videos are stored and transmitted in a compressed format. This decreases the overhead to fully decode and re-encode the video for embedding and extraction of the watermark. High Efficiency Video Coding (HEVC/H.265) is the latest and most efficient video compression standard and a successor to H.264 Advanced Video Coding. In this article, we propose a robust watermarking framework for HEVC-encoded video using informed detector. A readable watermark is embedded invisibly in P frames for better perceptual quality. Our framework imposes security and robustness by selecting appropriate blocks using a random key and the spatio-temporal characteristics of the compressed video. A detail analysis of the strengths of different compressed domain features is performed for implementing the watermarking framework. We experimentally demonstrate the utility of the proposed work. The results show that the proposed work effectively limits the increase in video bitrate and degradation in perceptual quality. The proposed framework is robust against re-encoding and image processing attacks.

Gaze-Contingent Computational Displays: Boosting perceptual fidelity

Contemporary digital displays feature multimillion pixels at ever-increasing refresh rates. Reality, on the other hand, provides us with a view of the world that is continuous in space and in time. The discrepancy between viewing the physical world and its sampled depiction on digital displays gives rise to perceptual quality degradation. By measuring or estimating where we look, a new breed of gaze-contingent algorithms aims to exploit the way we visually perceive digital images and videos to remedy visible artifacts. In this article, we provide an overview of recent developments in computational display algorithms that enhance perceived visual quality of conventional video footage when viewed on commodity monitors, projectors, or headmounted displays (HMDs).

Perceptual Quality Driven Frame-Rate Selection (PQD-FRS) for High-Frame-Rate Video

Video of higher frame rates (HFR) reduces the visual artifact in large screen display at the cost of a higher coding bit rate (or transmission bandwidth). In this work, we propose a perceptual quality driven frame rate selection (PQD-FRS) method that assigns a time-varying frame rate to a sequence so as to reduce its transmission cost. The objective of the PQD-FRS method is to offer perceptually indistinguishable experience for a certain percentage of viewers. We first conduct a subjective test to characterize the relationship between human perceived quality and video contents, and build a frame-rate-dependent video quality assessment dataset to serve as the ground truth. Then, we use a machine learning approach for the design of the key module of the PQD-FRS method, called the “satisfied user ratio (SUR) prediction.” The SUR prediction module predicts the percentage of satisfied viewers, who cannot differentiate video quality of a lower and HFR, using the support vector regression. It is confirmed by experimental results that the proposed SUR module can offer a so highly accurate prediction that the PQD-FRS system can dynamically assign a proper frame rate to video without any perceptual quality degradation for a majority of viewers.