Perceptual Redundancy Research Articles

Effects of perceptual redundancy, conceptual redundancy and self-relatedness on categorical responses.

A redundancy gain occurs when perceptually identical stimuli are presented together, resulting in quicker categorization of these paired stimuli than lone stimuli. Similar effects have been reported for paired stimuli within the same conceptual category, particularly if the category is self-related. We recruited 528 individuals across three related studies to investigate whether, during perceptual and conceptual redundancy, such self-bias effects on foreground stimuli are modulated by natural versus urban backgrounds. Here, we highlight our observations pertaining to perceptual and conceptual redundancy effects of the foreground stimuli. In our first experiment, response options were randomised per trial. Results showed reaction time gains for perceptually identical stimuli, but this advantage was not modulated by self/other categorization. However, slower reaction times were observed for conceptually-related stimulus pairs and were influenced by self/other categorization. The second experiment replicated the methods of earlier studies of redundancy and observed comparable results to Experiment 1: a perceptual redundancy gain unmodulated by self/other categorization, yet for conceptual redundancy, no gain/cost but effects of self/other categorization. In the third experiment, self/other categories were substituted with arbitrary A/B categories: Once more, there was a perceptual redundancy gain and no conceptual redundancy gain. Notably, A/B categorisation produced effects equivalent to self/other categorisation. Overall, these findings challenge previous research on the facilitated early processing of conceptually-related stimuli and suggest that self-relatedness may not exert a unique effect on stimulus processing beyond attentional and response preferences during categorization. Our study motivates further research to understand conceptual categorization and redundancy gain effects.

Fault detection with confidence level evaluation for perception module of autonomous vehicles based on long short term memory and Gaussian Mixture Model

The reliability of perception is crucial for developing higher levels of autonomy to enhance the safety and performance of fully autonomous vehicles. As a result, fault detection in the perception module is an essential function for autonomous vehicles. While many approaches rely on hardware redundancy, necessitating additional sensors, achieving analytical redundancy in perception is a pivotal factor in reducing the cost and complexity of the autonomous system. This paper utilizes a motion predictor for surrounding vehicles and a Gaussian Mixture Model (GMM) to evaluate the confidence level of the perception module without requiring additional sensor installations. The motion predictor is designed based on a long short-term memory-based recurrent neural network. The error between the motion prediction and detection results is leveraged to estimate the confidence level of the perception module. In essence, the motion prediction results are treated as redundant sensor measurements. The error distribution is modeled using a GMM, and the cumulative probability density of the GMM is employed to assess the confidence level. The proposed algorithm's effectiveness is evaluated through a driving data-based simulation study with fault injection. This study shows an enhanced sensitivity to injected faults and a reduced time for fault detection. Furthermore, the proposed algorithm provides a quantitative estimate of the performance degradation level of the perception module. This estimation can serve as an indicator of uncertainty that the motion planner should account for.

Surprise-based JND estimation for perceptual quantization in H.265/HEVC codecs

Just noticeable distortion (JND), reflecting the perceptual redundancy directly, has been widely used in image and video compression. However, the human visual system (HVS) is extremely complex and the visual signal processing has not been fully understood, which result in existing JND models are not accurate enough and the bitrate saving of JND-based perceptual compression schemes is limited. This paper presents a novel pixel-based JND model for videos and a JND-based perceptual quantization scheme for HEVC codecs. In particular, positive and negative perception effects of the inter-frame difference and the motion information are analyzed and measured with an information-theoretic approach. Then, a surprise-based JND model is developed for perceptual video coding (PVC). In our PVC scheme, the frame-level perceptual quantization parameter (QP) is derived on the premise that the coding distortion is infinitely close to the estimated JND threshold. On the basis of the frame-level perceptual QP, we determine the perceptual QP for each coding unit through a perceptual adjustment function to achieve better perceptual quality. Experimental results indicate that the proposed JND model outperforms existing models significantly, the proposed perceptual quantization scheme improves video compression efficiency with better perceptual quality and lower coding complexity.

Just Noticeable Difference Model for Images with Color Sensitivity.

The just noticeable difference (JND) model reflects the visibility limitations of the human visual system (HVS), which plays an important role in perceptual image/video processing and is commonly applied to perceptual redundancy removal. However, existing JND models are usually constructed by treating the color components of three channels equally, and their estimation of the masking effect is inadequate. In this paper, we introduce visual saliency and color sensitivity modulation to improve the JND model. Firstly, we comprehensively combined contrast masking, pattern masking, and edge protection to estimate the masking effect. Then, the visual saliency of HVS was taken into account to adaptively modulate the masking effect. Finally, we built color sensitivity modulation according to the perceptual sensitivities of HVS, to adjust the sub-JND thresholds of Y, Cb, and Cr components. Thus, the color-sensitivity-based JND model (CSJND) was constructed. Extensive experiments and subjective tests were conducted to verify the effectiveness of the CSJND model. We found that consistency between the CSJND model and HVS was better than existing state-of-the-art JND models.

Perceptual redundancy and semantic grouping effects among real-world objects in visual working memory come from different processes

Perceptual redundancy and semantic grouping effects among real-world objects in visual working memory come from different processes

Perceptual ultra-high definition video coding based on adaptive just noticeable distortion model

Ultra-high definition (UHD) video is gradually applied into many fields. The huge amount of data in UHD video limits the realization of these applications. Perception video coding can exploit the visual redundancy, and improve the compression performance. So far, the perpetual video coding based on just noticeable distortion (JND) model cannot fully exploit the perceptual redundancy of ultra-high definition (UHD) video/image in 10-bit depth. Thus, an adaptive JND model based on perceptual noise Bayesian network (PNBN) is firstly established for UHD video/image. Then, combined perceptual noise channel encoding/decoding with human visual characteristics, hidden side information of image is dynamically estimated by PNBN. Thirdly, the adaptive JND model is considered both luminance masking effect and texture complexity, which can adaptively exploit redundancy in different texture regions of image. Finally, an effective perceptual video coding (PVC) scheme is proposed for versatile video coding. A novel distortion compensation factor is appropriately designed for rate-distortion optimization. Experimental results show that the proposed PVC scheme achieves significant bitrate reduction with better subjective video quality than the state-of-the-art PVC schemes.

Spatial-frequency HEVC multiple description video coding with adaptive perceptual redundancy allocation

Multiple description coding (MDC) approaches improve the error-resilient performance of video transmission by introducing redundancy. The existing multiple description video coding (MDVC) schemes are rarely designed for particular coding structure of HEVC, and the characteristics of the human visual system (HVS) are seldom considered. In this paper, a spatial-frequency multiple description video coding with adaptive perceptual redundancy allocation framework, named SF-PMDVC, is proposed for HEVC. For descriptions generation, after polyphase down-sampling in spatial domain, a transformation based on integer discrete cosine transform (DCT) is expended to adapt to the flexible coding unit partitioning process in HEVC, and the frequency coefficients are segmented and mapped to reduce the bitrate of each description. To further improve the performance of MDVC, an adaptive perceptual redundancy allocation strategy based on visual saliency is proposed, which improves the coding efficiency adapting to the visual perception. Experimental results show that the proposed scheme improves the error resiliency of HEVC by achieving superior objective and subjective reconstructed video quality as compared to the state-of-the-art MDVC methods for HEVC.

Perceptual redundancy model for compression of screen content videos

Screen content video (SCV) consists primarily of text areas, computer graphics and other computer-generated content and possesses unique perceptual characteristics. To compress SCVs more effectively with less reduction in subjective quality, perceptual characteristics of SCVs are analyzed and a perceptual redundancy (PR) model for SCV compression is proposed, including spatial PR (SPR), temporal PR (TPR) and foveated PR (FPR) model. In SPR modeling, the SCV is divided into sharp edge (SE) areas and non-SE areas, then SPR is estimated separately. In TPR modeling, both inter-frame luminance adaptation effect and motion masking effect are taken into account. In FPR modeling, each frame of SCV is classified into abrupt frames, relative motion frames or static frames. Then fixation points of different kinds of frames are predicted using different methods, and FPR is modeled considering foveated masking effect and visual attention. Finally, the perceptual redundancy of SCV is estimated based on the product of SPR, TPR and FPR. It is experimentally demonstrated that compared to the state-of-the-art models, the authors' model could obtain more accurate estimates of PR. Moreover, the model is incorporated into SCV compression with an adaptive perceptual quantizer. An average of 7.42% bits could be saved with less decline in subjective quality.

Efficient Partition Decision Based on Visual Perception and Machine Learning for H.266/Versatile Video Coding

H.266/Versatile Video Coding (VVC) is the latest international video coding standard to encode ultra-high-definition video effectively. The quadtree with nested multi-type tree (QT-MTT) structure provides various sizes of coding tree partitioning and allows the nested binary tree (BT) split and ternary tree (TT) split at each QT level. Furthermore, numerous advanced coding tools are equipped in the H.266/VVC encoder. However, the encoding time increases tremendously. Previous researches regarding the fast coding algorithm of H.266/VVC seldom mention perceptual redundancy. This paper utilizes the human vision model of just noticeable difference to extract the visually distinguishable pixels that may affect the visual perception. We observe that the distributions acquired by the horizontal and vertical projections of visually distinguishable pixels within the coding unit are related to their corresponding MTT splitting modes. Therefore, the distributions representing the perceptual information of human vision are used to be the input features of machine learning. Fast MTT decision determined by the random forest models of machine learning is proposed to quickly select the partition for intra coding. Experimental results demonstrate that the proposed method can effectively accelerate intra coding process while maintaining good bitrate and video quality based on the properties of the visual perception. The proposed algorithm provides better performance than the previous work.

Perceptual Redundancy Estimation of Screen Images via Multi-Domain Sensitivities

Visual redundancy detection is essential for image and video communication. Human visual system (HVS) is difficult to perceive the pixel magnitude change below a certain visibility threshold which is also known as just-noticeable-difference (JND). In this letter, we present an efficient JND estimation approach for screen content images by considering high-frequency sensitivity and orientation sensitivity correction. Specifically, to better quantify the visual redundancy, we investigate the visibility threshold based on the high-frequency distortion sensitivity. To obtain the orientation sensitivity correction, we divide the screen image pixels into three levels based on the oblique effect that considers the sensitive integrity of edges. Compared with several state-of-the-art JNDs, experimental results show that our method tolerates more perceptual redundancy, and delivers better visual quality under the same injected-noise energy. The implementation of the proposed method is publicly available at https://sites.google.com/site/wangmiaohui/.

Spatial-Frequency Hevc Multiple Description Video Coding with Adaptive Perceptual Redundancy Allocation

Spatial-Frequency Hevc Multiple Description Video Coding with Adaptive Perceptual Redundancy Allocation

Blind Photograph Watermarking with Robust Defocus-Based JND Model

Just noticeable distortion (JND) is widely employed to describe the perception redundancy in the quantization-based watermarking framework. However, the existing JND models are generally constructed to treat every region of the photograph with an equal focus level, whereas the defocus effect has never been considered. In this paper, the defocus feature, which can portray the aesthetic emphasis in the photograph, is provided to improve the perceptual JND model. Firstly, two indicators which consider the block energy in the defocus measurement (DM) are proposed. Then, the defocus feature map (DFM) is obtained by integrating the influence of the circumambient blocks, and it is applied to the proposed JND contrast masking (CM) processing. In this way, a new blind photograph watermarking method, with emphasis on defocus-JND estimation combined with the proposed CM, is presented. Simulations show that the proposed JND is more suitable for watermarking framework than some exiting JND models, and the proposed watermarking scheme with the improved defocus-based JND model has superior robustness compared with some watermarking schemes.

An adaptive spatio-temporal perception aware quantization algorithm for AVS2

Adaptive quantization proves to be an effective tool to improve coding performance. In this paper, we propose an adaptive spatiotemporal perception aware quantization algorithm to increase subjective coding performance. To measure the spatiotemporally perceptual redundancy, the perceptual complexity models are firstly established with spatial and temporal characteristics respectively. With the help of the models, the adaptive spatial and temporal quantization parameter (QP) offsets are then calculated for each coding tree unit (CTU), respectively. Finally, the perceptually optimal Lagrange multiplier of each CTU is determined with the spatial–temporal QP offset. Experimental results show that the proposed algorithm reduces 8.6% and 8.4% Bjontegaard-Delta Rate (BD-Rate) with Structural Similarity Index Metric (SSIM) in average over the second generation of Audio Video Coding Standard (AVS2) reference software RD17.0 in Low-Delay-P (LDP) and Random-Access (RA) configurations, respectively. The subjective assessment proves that the proposed algorithm can reduce the bitrates with the same subjective quality significantly.

Validation of a novel functional test for assessing metamorphopsia using epiretinal membranes as a model

Current tests for assessing metamorphopsia do not account for confounders such as perceptual filling-in and spatial redundancy, which affect its sensitivity and repeatability. This proof-of-concept study aimed to assess the performance of a novel laboratory-based psychophysical test (Line Sag Test, LST) which addresses these issues for quantification of metamorphopsia in idiopathic epiretinal membranes. The LST quantifies perpendicular metamorphopsia at three eccentricities (3°, 6°, and 9°) along eight meridians (45° steps). Metamorphopsia was assessed using the LST and Amsler grid and the hit rates of both tests for detecting metamorphopsia were compared. Normal metamorphopsia scores using the LST did not differ significantly from 0 and fell within one step-size (p = 0.500). The LST detected significantly more cases of metamorphopsia than the Amsler grid (14/21 versus 3/21) (p = 0.003). Similarly, significantly more cases of visual distortions in asymptomatic iERMs were detected using the LST than the Amsler grid (11/18 versus 0/18) (p = 0.008). The LST has a higher hit rate compared to the Amsler grid (67% versus 14%). This work demonstrates a psychophysically-robust functional test addressing perceptual confounders is more sensitive for quantifying and localising metamorphopsia in macular disease, particularly in asymptomatic disease.

Learning-Based JND-Directed HDR Video Preprocessing for Perceptually Lossless Compression With HEVC

The final consumer of videos is mostly human. Therefore, if videos can be compressed by fully utilizing the perception characteristics of human visual systems (HVS), the bitrates of the compressed videos can be significantly reduced with subjective visual quality degradation as little as possible. Based on this, we newly propose a learning-based Just Noticeable Distortion (JND)-directed preprocessing scheme for perceptual video compression, especially for 10-bit High Dynamic Range (HDR) videos, which is called the HDR-JNDNet. Our HDR-JNDNet effectively suppresses the perceptual redundancy of 10-bit HDR video signals so that the compression efficiency can be significantly enhanced for the HEVC main10 profile encoder. To our best knowledge, our work is the first approach to training a CNN-based model to directly generate the JND-directed suppressed frames of 10-bit HDR video with the negligible perceptual quality difference between the decoded frames for the original HDR video input with and without the preprocessing by our HDR-JNDNet. Via intensive experiments, when the HDR-JNDNet is applied as preprocessing for the HDR video input before compression, it allows to remarkably save the required bitrates up to the maximum (average) 40.66% (18.37%) for 4K-UHD/HDR test videos, with little subjective video quality degradation without increasing the computational complexity.

Asymmetric Foveated Just-Noticeable-Difference Model for Images With Visual Field Inhomogeneities

Just noticeable difference (JND) models reveal visibility limitation of human visual system (HVS) and exploit perceptual redundancy in the visual signal. Conventional just noticeable difference (JND) model supposes the visual acuity is homogeneous with or without considerations on the isoeccentric locations. Recent research on vision science indicates that the effect of eccentricity on visual sensitivity is not homogeneous across the visual field (VF). Relevant studies have found visual acuity at the isoeccentric locations is better along the horizontal meridian than along the vertical meridian, i.e., horizontal-vertical anisotropy (HVA). Along the vertical meridian, better performance has been reported for the inferior meridian compared to the superior meridian, i.e., vertical-meridian asymmetry (VMA). These properties of HVS should be taken into account to yield a better JND estimation, for better exploiting the perceptual redundancy. In this paper, we use two hemi-ellipses to represent the iso-acuity contour, which can better express the HVA and VMA effects. Moreover, we design psychophysical experiments under different test conditions, where the results show the hemi-ellipse model performs better when compared to the circular model. Finally, we build an asymmetric foveated JND model using two hemi-ellipses for approximation. When comparing with traditional JND models, experimental results demonstrate the effectiveness of the proposed model.

P‐3.3: High Definition OLED‐on‐Silicon Micro‐display Based on Human Visual Characters

As users demand higher refresh rates and frame rates, the amount of data in the virtual reality field is also growing. There is a large visual perception redundancy in the image source transmitted by the micro‐display system. In this paper, an OLED‐on‐silicon micro‐display system based on human visual characters is designed to solve the problem of high data bandwidth transmission in head‐mounted virtual reality applications. The experimental results show the multi‐resolution image fusion compression algorithm based on the visual perception of human eye has the good data compression ratio. The proposed method meets the visual perception of the human eye. It has good rendering quality, which meets the real‐time rendering requirements of the OLED‐on‐silicon micro‐display in the virtual reality embedded environment.

High-Definition Video Compression System Based on Perception Guidance of Salient Information of a Convolutional Neural Network and HEVC Compression Domain

The new generation of the high-efficiency video coding (HEVC) video coding standard has improved compression performance and brought considerable coding complexity. Therefore, reducing the perception redundancy of video to obtain a better compression effect is the new direction of video development at present. In this paper, the HEVC is improved and enhanced from the aspects of a video saliency algorithm based on an attention mechanism and a video compression algorithm based on perception priority. In terms of video saliency, this paper proposes a spatial saliency algorithm based on a convolutional neural network and a temporal saliency algorithm based on motion vector. The saliency algorithm can combine the motion estimation results of each block during the HEVC compression on the basis of convolutional neural network and carry out adaptive dynamic fusion of the two, to complete the saliency map of the input video. In the aspect of the video compression algorithm with perception priority, this paper proposes a more flexible QP selection method, which selects its corresponding QP according to the saliency value of CU. At the same time, we propose a new rate-distortion optimization algorithm, which integrates the current block’s saliency feature into the traditional rate-distortion calculation method, to guide the allocation of bits and achieve the purpose of perception priority. The experimental results proved the superiority of the proposed method over the state-of-the-art perceptual coding algorithms in terms of saliency detection and perceptual compression quality.

Invisible Adversarial Attack against Deep Neural Networks: An Adaptive Penalization Approach

Recent studies demonstrated that deep neural networks (DNNs) are vulnerable to adversarial examples, which would seriously threaten security-sensitive applications. Existing works synthesized the adversarial examples by perturbing the original/benign images by leveraging the $\mathcal {L}_p$ L p -norm to penalize the perturbations, which restricts the pixel-wise distance between the adversarial images and correspondingly benign images. However, they added perturbations globally to the benign images without explicitly considering their content/spacial structure, resulting in noticeable artifacts especially in those originally clean regions, e.g., sky and smooth surface. In this paper, we propose an invisible adversarial attack, which synthesizes adversarial examples that are visually indistinguishable from benign ones. We adaptively distribute the perturbation according to human sensitivity to a local stimulus in the benign image, i.e., the higher insensitivity, the more perturbation. Two types of adaptive adversarial attacks are proposed: 1) coarse-grained and 2) fine-grained. The former conducts $\mathcal {L}_p$ L p -norm regularized by the novel spatial constraints, which utilizes the rich information of the cluttered regions to mask perturbation. The latter, called Just Noticeable Distortion (JND)-based adversarial attack, utilizes the proposed JND $_p$ p metric for better measuring the perceptual similarity, and adaptively sets penalty by weighting the pixel-wise perceptual redundancy of an image. We conduct extensive experiments on the MNIST, CIFAR-10 and ImageNet datasets and a comprehensive user study with 50 participants. The experimental results demonstrate that JND $_p$ p is a better metric for measuring the perceptual similarity than $\mathcal {L}_p$ L p -norm, and the proposed adaptive adversarial attacks can synthesize indistinguishable adversarial examples from benign ones and outperform the state-of-the-art methods.

Joint Optimization of Transform and Quantization for High Efficiency Video Coding

In high efficiency video coding (HEVC), transformation and quantization are separately performed to eliminate the perceptual redundancy of visual signals. However, a uniform quantizer can inevitably degrade the compression efficiency of fixed transform matrices due to varying space-frequency characteristics of video content. This paper introduces a joint optimization of transform and quantization approach for video coding. First, we compute a content dependent transform from the reconstructed reference by a fast Karhunen-Loeve transform (KLT). Second, using a template-based rate regularization, we jointly optimize transform and quantization (JOTQ) as a rate constrained optimization problem and obtain a feasible solution to improve coding performance. Finally, we design fast algorithms and early terminations to reduce the computational complexity of JOTQ. The experimental results show that JOTQ outperforms several previous methods by providing Bjontegaard Delta rate reductions of 4.11% and 3.38% on average under the low-delay and random-access configuration, respectively.