Spatial Masking Research Articles

Perceived Coding Distortion Assessment for Streaming Video

For applications involving video streaming, full decoding is usually not acceptable for quality assessment. To address the inherent challenges, an efficient method for coding distortion assessment is proposed in this paper. Building on empirical analysis, the proposed method employs a linear model to assess the coding distortion using the quantization scale. Furthermore, the characteristics of the human visual system are exploited by taking into account the spatial and temporal masking. To estimate the required spatial and temporal complexities in absence of sufficient information, a rate-distortion model is theoretically derived to formulate their relationship with the coding bit-rate. Extensive experimental results have demonstrated the effectiveness of the proposed method for quality assessment with respect to perceived coding distortion.

Can illusory contours and grouping produce spatial masking?

Can illusory contours and grouping produce spatial masking?

Watermarking of Free-view Video

With the advances in image based rendering (IBR) in recent years, generation of a realistic arbitrary view of a scene from a number of original views has become cheaper and faster. One of the main applications of this progress has emerged as free-view TV(FTV), where TV-viewers select freely the viewing position and angle via IBR on the transmitted multiview video. Noting that the TV-viewer might record a personal video for this arbitrarily selected view and misuse this content, it is apparent that copyright and copy protection problems also exist and should be solved for FTV. In this paper, we focus on this newly emerged problem by proposing a watermarking method for free-view video. The watermark is embedded into every frame of multiple views by exploiting the spatial masking properties of the human visual system. Assuming that the position and rotation of the virtual camera is known, the proposed method extracts the watermark successfully from an arbitrarily generated virtual image. In order to extend the method for the case of an unknown virtual camera position and rotation, the transformations on the watermark pattern due to image based rendering operations are analyzed. Based upon this analysis, camera position and homography estimation methods are proposed for the virtual camera. The encouraging simulation results promise not only a novel method, but also a new direction for watermarking research.

The role of higher order image statistics in masking scene gist recognition

In the present article, we investigated whether higher order image statistics, which are known to be carried by the Fourier phase spectrum, are sufficient to affect scene gist recognition. In Experiment 1, we compared the scene gist masking strength of four masking image types that varied in their degrees of second- and higher order relationships: normal scene images, scene textures, phase-randomized scene images, and white noise. Masking effects were the largest for masking images that possessed significant higher order image statistics (scene images and scene textures) as compared with masking images that did not (phase-randomized scenes and white noise), with scene image masks yielding the largest masking effects. In a control study, we eliminated all differences in the second-order statistics of the masks, while maintaining differences in their higher order statistics by comparing masking by scene textures rather than by their phase-randomized versions, and showed that the former produced significantly stronger gist masking. Experiments 2 and 3 were designed to test whether conceptual masking could account for the differences in the strength of the scene texture and phase-randomized masks used in Experiment 1, and revealed that the recognizability of scene texture masks explained just 1% of their masking variance. Together, the results suggest that (1) masks containing the higher order statistical structure of scenes are more effective at masking scene gist processing than are masks lacking such structure, and (2) much of the disruption of scene gist recognition that one might be tempted to attribute to conceptual masking is due to spatial masking.

A Versatile Model for Packet Loss Visibility and its Application to Packet Prioritization

In this paper, we propose a generalized linear model for video packet loss visibility that is applicable to different group-of-picture structures. We develop the model using three subjective experiment data sets that span various encoding standards (H.264 and MPEG-2), group-of-picture structures, and decoder error concealment choices. We consider factors not only within a packet, but also in its vicinity, to account for possible temporal and spatial masking effects. We discover that the factors of scene cuts, camera motion, and reference distance are highly significant to the packet loss visibility. We apply our visibility model to packet prioritization for a video stream; when the network gets congested at an intermediate router, the router is able to decide which packets to drop such that visual quality of the video is minimally impacted. To show the effectiveness of our visibility model and its corresponding packet prioritization method, experiments are done to compare our perceptual-quality-based packet prioritization approach with existing Drop-Tail and Hint-Track-inspired cumulative-MSE-based prioritization methods. The result shows that our prioritization method produces videos of higher perceptual quality for different network conditions and group-of-picture structures. Our model was developed using data from high encoding-rate videos, and designed for high-quality video transported over a mostly reliable network; however, the experiments show the model is applicable to different encoding rates.

Geometrical superresolved imaging using nonperiodic spatial masking

The resolution of every imaging system is limited either by the F-number of its optics or by the geometry of its detection array. The geometrical limitation is caused by lack of spatial sampling points as well as by the shape of every sampling pixel that generates spectral low-pass filtering. We present a novel approach to overcome the low-pass filtering that is due to the shape of the sampling pixels. The approach combines special algorithms together with spatial masking placed in the intermediate image plane and eventually allows geometrical superresolved imaging without relation to the actual shape of the pixels.

Digital Video Watermarking in P-Frames With Controlled Video Bit-Rate Increase

Most video watermarking algorithms embed the watermark in I-frames, but refrain from embedding in P- and B-frames, which are highly compressed by motion compensation. However, P-frames appear more frequently in the compressed video and their watermarking capacity should be exploited, despite the fact that embedding the watermark in P-frames can increase the video bit rate significantly. This paper gives a detailed overview of a common approach for embedding the watermark in I-frames. This common approach is adopted to use P-frames for video watermarking. We show that by limiting the watermark to nonzero-quantized AC residuals in P-frames, the video bit-rate increase can be held to reasonable values. Since the nonzero-quantized AC residuals in P-frames correspond to nonflat areas that are in motion, temporal and texture masking are exploited at the same time. We also propose embedding the watermark in nonzero quantized AC residuals with spatial masking capacity in I-frames. Since the locations of the nonzero-quantized AC residuals is lost after decoding, we develop a watermark detection algorithm that does not depend on this knowledge. Our video watermark detection algorithm has controllable performance. We demonstrate the robustness of our proposed algorithm to several different attacks.

Oblivious Spatio-Temporal Watermarking of Digital Video by Exploiting the Human Visual System

Imperceptibility requirement in video watermarking is more challenging compared with its image counterpart due to the additional dimension existing in video. The embedding system should not only yield spatially invisible watermarks for each frame of the video, but it should also take the temporal dimension into account in order to avoid any flicker distortion between frames. While some of the methods in the literature approach this problem by only allowing arbitrarily small modifications within frames in different transform domains, some others simply use implicit spatial properties of the human visual system (HVS), such as luminance masking, spatial masking, and contrast masking. In addition, some approaches exploit explicitly the spatial thresholds of HVS to determine the location and strength of the watermark. However, none of the former approaches have focused on guaranteeing temporal invisibility and achieving maximum watermark strength along the temporal direction. In this paper, temporal dimension is exploited for video watermarking by means of utilizing temporal sensitivity of the HVS. The proposed method utilizes the temporal contrast thresholds of HVS to determine the maximum strength of watermark, which still gives imperceptible distortion after watermark insertion. Compared with some recognized methods in the literature, the proposed method avoids the typical visual degradations in the watermarked video, while still giving much better robustness against common video distortions, such as additive Gaussian noise, video coding, frame rate conversions, and temporal shifts, in terms of bit error rate.

Archival image indexing with connectivity features using randomized masks

Connectivity properties capture a natural spatial feature of a binary image. Albeit they are easy to compute, more often than not, they alone fail to provide a good characterization of the image because of the fact that several different images may have the same connectivity features. Typically, other types of parameters, e.g., moments, are used to augment the connectivity feature for efficient indexing and retrieval purposes. In this work, an alternative approach is proposed. Instead of considering other diverse features, which are computationally intensive, only the connectivity features of the image are used iteratively using a novel concept of spatial masking. For this purpose, a greedy algorithm for constructing a spatial feature vector of variable length for a binary image, is proposed. The algorithm is based on XOR-ing the image bit-plane with a few pseudo-random synthetic masks, and its novelty lies in computing the feature vector iteratively, depending on the size and diversity of the image database. The classical Euler number and the two primary connectivity features from which it is derived, namely, the number of connected components and the number of holes, are used to finally generate a unique feature vector for each binary image in the database using a fuzzy membership function customized for the given database. The method is particularly suitable for large-sized image archives of a digital library, where each image contains one or more objects. It is found to converge within only three iterations for a postal stamp database consisting of 2598 images, and also for a logo database of 1034 images. A data structure called discrimination tree has been introduced for supporting efficient storage and indexing of the images using the above feature vector.

Human visual system based adaptive digital image watermarking

It is known that there is a trade off between the imperceptibility and robustness of a digital watermarking system. Trying to deal with this problem, a new adaptive digital image watermarking method is proposed. The new spatial masking is built according to the image features such as the brightness, edges, and region activities. With the same watermark embedding energy, the quality of watermarked image using the proposed adaptive masking is much better than the one without using the adaptive masking. The wPSNR is used to evaluate the image quality difference which is fed into the new spatial masking function to control the watermark embedding adaptively. The watermark is detected by a key-dependent method without knowing any information of the original image. In addition, we also extend this proposed spatial masking to the DCT domain by searching the extreme value of the quadratic function subject to the bounds on the variables. In experiments, this scheme shows good performance.

Efficient Spatial Image Watermarking via New Perceptual Masking and Blind Detection Schemes

The aim of this paper is to improve the performance of spatial domain watermarking. To this end, a new perceptual mask and a new detection scheme are proposed. The proposed spatial perceptual mask is based on the cover image prediction error sequence and matches very well with the properties of the human visual system. It exhibits superior performance compared to existing spatial masking schemes. Moreover, it allows for a significantly increased strength of the watermark while, at the same time, the watermark visibility is decreased. The new blind detection scheme comprises an efficient prewhitening process and a correlation-based detector. The prewhitening process is based on the least-squares prediction error filter and substantially improves the detector's performance. The correlation-based detector that was selected is shown to be the most suitable for the problem at hand. The improved performance of the proposed detection scheme has been justified theoretically for the case of linear filtering plus noise attack and through extensive simulations. The theoretical analysis is independent of the proposed mask and the derived expressions can be used for any watermarking technique based on spatial masking. It is shown though that in most cases the detector performs better if the proposed mask is employed.

A perceptual two-pass VBR MPEG-2 video encoder

This paper presents a perceptual two-pass VBR MPEG-2 video encoder suitable for digital storage applications. The proposed algorithm aims to achieve perceptually uniform distortion among adjacent macroblocks within a frame and smoothen the inter-frame visual quality fluctuation as soon as possible. In the first-pass coding, we analyze the video sequence and collect some useful coding parameters to accurately measure the frame coding complexity. In the second-pass coding, the target bits are allocated to all frames optimally according to the coding complexity, certainly with decoder buffer considered. The human visual system characteristics including both the spatial and temporal masking effects are employed to improve the proposed adaptive quantization algorithm. Furthermore, an improved iteration based Qp selection algorithm is proposed to search for the optimal frame-level reference Qp resulting in satisfactory iteration speed and iteration precision. Experimental results show that the proposed algorithm can achieve superior visual quality both objectively and subjectively compared with the reference algorithms.

Just noticeable distortion model and its applications in video coding

We explore a new perceptually-adaptive video coding (PVC) scheme for hybrid video compression, in order to achieve better perceptual coding quality and operational efficiency. A new just noticeable distortion (JND) estimator for color video is first devised in the image domain. How to efficiently integrate masking effects together is a key issue of JND modelling. We integrate spatial masking factors with the nonlinear additivity model for masking (NAMM). The JND estimator applies to all color components and accounts for the compound impact of luminance masking, texture masking and temporal masking. Extensive subjective viewing confirms that it is capable of determining a more accurate visibility threshold that is close to the actual JND bound in human eyes. Secondly, the image-domain JND profile is incorporated into hybrid video encoding via the JND-adaptive motion estimation and residue filtering process. The scheme works with any prevalent video coding standards and various motion estimation strategies. To demonstrate the effectiveness of the proposed scheme, it has been implemented in the MPEG-2 TM5 coder and demonstrated to achieve average improvement of over 18% in motion estimation efficiency, 0.6 dB in average peak signal-to perceptual-noise ratio (PSPNR) and most remarkably, 0.17 dB in the objective coding quality measure (PSNR) on average. Theoretical explanation is presented for the improvement on the objective coding quality measure. With the JND-based motion estimation and residue filtering process, hybrid video encoding can be more efficient and the use of bits is optimized for visual quality.

Foveal contour interaction: detection and discrimination

Contour interaction, the detrimental effect of flanking features on the discrimination of optotypes, has been studied mainly close to the visual acuity limit. We were interested to know how these results compare with those for the detection of targets. According to the simplest model of contour interaction, comparable detection effects would be expected. The case for low-level masking would be further strengthened if the form and nature of the dependence on flank separation and flank polarity followed that typically found in studies of lateral spatial masking [Vision Res. 33, 993 (1993)]. Landolt Cs subtending a visual angle of 0.25 degrees, 0.5 degrees, and 1.0 degrees were presented and contrast thresholds for detecting the presence of the Landolt C and discriminating its orientation were measured in five normal subjects as a function of flank separation and flank polarity. The results obtained for the relationship between detection and discrimination depend on the size of the target used. For small letters, discrimination but not detection was significantly affected by flanking bars. For large letters, detection and discrimination were affected to the same extent. However, in this case the effectiveness of opposite-polarity flanks and the finding that facilitation occurred at close, not far, flank separations suggests that the simplest explanation in terms of masking may not be applicable.

Interocular masking and summation indicate two stages of divisive contrast gain control

Our understanding of early spatial vision owes much to contrast masking and summation paradigms. In particular, the deep region of facilitation at low mask contrasts is thought to indicate a rapidly accelerating contrast transducer (eg a square-law or greater). In experiment 1, we tapped an early stage of this process by measuring monocular and binocular thresholds for patches of 1 cycle deg-1 sine-wave grating. Threshold ratios were around 1.7, implying a nearly linear transducer with an exponent around 1.3. With this form of transducer, two previous models (Legge, 1984 Vision Research 24 385 - 394; Meese et al, 2004 Perception 33 Supplement, 41) failed to fit the monocular, binocular, and dichoptic masking functions measured in experiment 2. However, a new model with two-stages of divisive gain control fits the data very well. Stage 1 incorporates nearly linear monocular transducers (to account for the high level of binocular summation and slight dichoptic facilitation), and monocular and interocular suppression (to fit the profound 42 Oral presentations: Spatial vision Thursday dichoptic masking). Stage 2 incorporates steeply accelerating transduction (to fit the deep regions of monocular and binocular facilitation), and binocular summation and suppression (to fit the monocular and binocular masking). With all model parameters fixed from the discrimination thresholds, we examined the slopes of the psychometric functions. The monocular and binocular slopes were steep (Weibull s˜3-4) at very low mask contrasts and shallow (s˜1.2) at all higher contrasts, as predicted by all three models. The dichoptic slopes were steep (s˜3-4) at very low contrasts, and very steep (s>5.5) at high contrasts (confirming Meese et al, loco cit.). A crucial new result was that intermediate dichoptic mask contrasts produced shallow slopes (s˜2). Only the two-stage model predicted the observed pattern of slope variation, so providing good empirical support for a two-stage process of binocular contrast transduction. [Supported by EPSRC GR/S74515/01]

Visual perception studies to improve the perceived sharpness of television images

Over the past century a number of factors have been observed to improve the quality of sampled images. Three of these phenomena are reviewed. The reasons for improvement based on perception studies are given, the improvements are quantified, and how they can be used in a practical television system is shown. Interlaced scanning reduces vertical resolution for several reasons involving spatial and temporal masking effects in visual perception. A progressive scan avoids these limitations. Because of the oblique effect in vision and the statistical orientation of lines in scenes, diagonal sampling reduces the required number of pixels in an image. Quantitatively, our measurements show that the number of pixels is reduced by a factor of 1.4 for the same perceived sharpness. In addition, by taking advantage of the octave-wide tuning bands in visual perception, our measurements show that the perceived resolution in the vertical direction for a progressive scan can be double that of an interlaced scan. Using diagonal sampling, a 1920×1080 image with progressive scan at 60 frames/s requires the same transmission bit rate as a 1920×1080 Cartesian sampled image scanned interlaced at 30 frames/s. This results in an image that appears to be much sharper than the 1080 line interlaced format without the interlace artifacts.

Quality measurements for compressed video transmitted over a lossy packet network

We evaluate the degradation in a video sequence that suffers from increasing packet loss ratio (PLR), by using several kinds of quality measures, namely, mean square error (MSE); A max (the root MSE, which is weighted by the maximum signal change between pairs of pixels—the current pixel and eight surrounding neighbors); the video perceptual distortion measure (VPDM); and a perceptual distortion metric for digital color video (VPDM2). Packet losses can occur at different positions in the compressed video sequence, for example, in headers, I frames, P frames, and B frames. Each packet loss results in a different amount of distortion in the received video signal. We demonstrate the differences between two packet loss models—IID (identical independent distribution) and burst (a sequence of packets lost together). We also examine the influence of packet loss in a communication network on MPEG2 compressed digital video quality. Three of these measures emulate features of the human visual system: the VPDM, which considers temporal masking of distortion (the human eye notices less degradation within a frame, if it appears during fast changes between consequent frames, such as scene cuts); the A max , which encompasses spatial masking, resulting in small objects within high-frequency neighborhoods being less noticeable; and the VPDM2, which takes into account the human vision color channels and temporal and spatial masking. The results point out that loss due to burst degrades the quality of the compressed video stream significantly less than loss due to IID. The video quality can still be tolerable for a 0.1 PLR using the burst loss model, but for the IID loss model, even for lower PLRs (i.e., 0.01), the video is not perceptible to the human viewer.

Engineering of the passband function of a generalized spectrometer

To our knowledge, we demonstrate a new scheme for passband engineering of a grating spectrometer. Through spatial masking of the input beam and translation of the optical components we present shaping, shifting, and scaling of the passband in optical frequency. Specifically, we demonstrate the relationship between the applied spatial masking function and the spectrometer passband may be tuned from an exact Fourier transform to a direct scaling through longitudinal displacement of the spectrometer lens and sampling slit and that this operation is independent of the choice of spectrometer center frequency.

Kernel-based object tracking

A new approach toward target representation and localization, the central component in visual tracking of nonrigid objects, is proposed. The feature histogram-based target representations are regularized by spatial masking with an isotropic kernel. The masking induces spatially-smooth similarity functions suitable for gradient-based optimization, hence, the target localization problem can be formulated using the basin of attraction of the local maxima. We employ a metric derived from the Bhattacharyya coefficient as similarity measure, and use the mean shift procedure to perform the optimization. In the presented tracking examples, the new method successfully coped with camera motion, partial occlusions, clutter, and target scale variations. Integration with motion filters and data association techniques is also discussed. We describe only a few of the potential applications: exploitation of background information, Kalman tracking using motion models, and face tracking.

Spatial masking does not reveal mechanisms selective to combined luminance and red-green color.

Detection thresholds plotted in the L and M cone-contrast plane have shown that there are two primary detection mechanisms, a red-green hue mechanism and a light-dark luminance mechanism. However, previous masking results suggest there may be additional mechanisms, responsive to combined features like bright and red or dark and green. We measured detection thresholds for a 1.2 c deg-1 sine-wave grating in the presence of a spatially matched mask grating which was either stationary, dynamically jittered or flickered. The stimuli could be set to any direction in the L,M plane. The appearance of selectivity for combined hue and luminance arose only in conditions where adding the test to the mask modified the spatial phase offset between the luminance and red-green stimulus components. Sensitivity was very high for detecting this spatial phase offset. When this extra cue was eliminated, masking contours in the L,M plane could be largely described by the classical red-green and luminance mechanisms.