Disparity Vectors Research Articles

Stereo Hysteresis Revisited

One of the most fascinating phenomena in stereopsis is the profound hysteresis effect reported by Fender and Julesz (1967), in which the depth percept persisted with increasing disparity long past the point at which depth was recovered with decreasing disparity. To control retinal disparity without vergence eye movements, they stabilized the stimuli on the retinas with an eye tracker. I now report that stereo hysteresis can be observed directly in printed stereograms simply by rotating the image. As the stereo image rotates, the horizontal disparities rotate to become vertical, then horizontal with inverted sign, and then vertical again before returning to the original orientation. The depth shows an interesting popout effect, almost as though the depth was turning on and off rapidly, despite the inherently sinusoidal change in the horizontal disparity vector. This stimulus was generated electronically in a circular format so that the random-dot field could be dynamically replaced, eliminating any cue to ...

Fast synthesized and predicted just noticeable distortion maps for perceptual multiview video coding

The just noticeable distortion (JND) map is a useful tool for perceptual video coding. However, direct calculation of the JND map incurs high complexity, and the problem is aggravated in multiview video coding. In this paper, two fast methods are proposed to generate the JND maps of multiview videos. In the first method, the JND maps of some anchor views are used to synthesize the JND maps of other views via the depth image based rendering (DIBR), which can be much faster than direct JND computation. In the second method, the motion and disparity vectors obtained during the video coding are employed to predict the JND maps. If the prediction is not satisfactory, the JND block will be refreshed by calculating the JND directly. This method does not need any camera parameters and depth maps. The performances of the two fast JND map generation methods are evaluated in a perceptual MVC framework, where the residuals after spatial, temporal, or inter-view prediction are tuned according to the JND thresholds to save the bits without affecting the perceptual quality. Experimental results show that the JND prediction method has better accuracy and lower complexity. In addition, both fast JND methods lead to negligible degradation of the coding performance, compared to the direct JND method.

Fast Motion and Disparity Estimation With Adaptive Search Range Adjustment in Stereoscopic Video Coding

Stereoscopic video gives viewers more realistic vision than traditional 2D videos by transmitting two different views simultaneously. It doubles the required bandwidth in comparison with single view videos. Motion and disparity estimation therefore play a key role in reducing the bit rate of stereoscopic videos. However, it brings extremely huge computational complexity to an encoder which obstructs it from practical uses. In the past few years, some fast algorithms were proposed where most of them speed up the coding time based on an accurate estimation in one field (motion/disparity field), and then relieve the computational burden in the other field (disparity/motion field). Nevertheless, the complexities of both motion and disparity estimation cannot be fully reduced. In this paper, an iterative motion and disparity estimation algorithm is proposed. The proposed algorithm can determine motion vectors of the right view and disparity vectors of the current stereo pair in an iterative way. The gain of this iterative search is due to the use of a stereo-motion consistency constraint in which the motion and disparity vectors can be estimated by updating the local optimal vectors iteratively. An adaptive search range adjustment through the confidence measure of the constraint is then designed to further strengthen the reliability of each step for the iterative search. Experimental results show that the speed-up gain of the proposed algorithm is 18.76 ~ 229.19 times compared to the full search algorithm with a negligible quality drop.

Early disparity estimation skipping for multi-view video coding

Multi-view video (MVV) and multi-view plus depth video became typical formats for 3D video systems which offer more realistic visual effects for consumers. The standard for encoding of MVV, known as multi-view video coding (MVC), was produced by ISO/IEC and ITU-T and has exhibited significant compression efficiency for MVV signals. However, the high computational complexity of MVC leads to excessive power consumption and limits its applications. In this study, new fast algorithms for MVC are proposed by skipping disparity estimation prediction. According to the characteristics of the motion and disparity vectors, unnecessary disparity estimation is skipped based on the amount of motion that can be measured through the use of motion activity and SKIP mode on the temporal axis. Experimental results showed that by skipping the disparity estimation, the proposed method reduced the encoding computational complexity by nearly 73% for inter-view prediction and 38% in the overall encoding process of B-views. In addition, no noticeable visual quality degradation was observed.

Iterative search strategy with selective bi-directional prediction for low complexity multiview video coding

The multiview video coding (MVC) extension of H.264/AVC is the emerging standard for compression of impressive 3D and free-viewpoint video. The coding structure in MVC adopts motion and disparity estimation to exploit temporal and inter-view dependencies in MVC. It results in a considerable increase in encoding complexity. Most of the computational burden comes from uni-directional and bi-directional prediction. In this paper, an iterative search strategy is designed to speed up the uni-directional prediction in MVC. It can work with an adaptive search range adjustment through a confidence measure of a loop constraint to obtain both motion and disparity vectors jointly. Furthermore, a selective bi-directional prediction algorithm is proposed to enhance the coding performance by analyzing the statistical characteristics of bi-directional prediction in MVC. Experimental results demonstrate that, by using the proposed fast search, the temporal and inter-view redundancies of multiview video can be eliminated sufficiently with low complexity.

Advanced motion vector coding framework for multiview video sequences

The inconsistency between the temporal motion vector and the disparity vector fields causes a drastic loss in coding performance during multiview video coding. In order to resolve this inconsistency problem, we investigate a new approach for motion vector coding. In this paper, degradation in accuracy using a conventional method is experimentally analyzed using mathematical tools. Then, we introduce a new framework for motion vector coding, including modified Inter-mode and Adaptive Direct-mode, in which an advanced motion vector prediction method is employed. The main idea of the proposed motion vector prediction method is to separate each motion vector field from the others. For this purpose, we introduce three alternative motion vectors; a virtual disparity vector, a virtual temporal motion vector and a scaled motion vector. The experimental results show that the proposed method can resolve the inconsistency problem and consequently improve the overall rate-distortion performance. The average bit-rate savings is about 7.3% compared to JMVC 6.0. At maximum, the gain obtained is about 20%, corresponding to a PSNR of 1.42 dB.

Efficient disparity vector prediction schemes with modified P frame for 2D camera arrays

An efficient disparity estimation algorithm for multi-view video sequences, recorded by a two-dimensional camera array in which the cameras are spaced equidistantly, is presented. Because of the strong geometrical relationship among views, the disparity vectors of a certain view can for most blocks be derived from the disparity vectors of other views. A frame constructed using that idea is called a D frame in this work. Three new prediction schemes which contain D frames are proposed for encoding 5 × 3 multi-view video sequences. The schemes are applied to several multi-view image sequences taken from a camera-array and they are compared in terms of quality, bit-rate and complexity. The experimental results show that the proposed prediction schemes significantly decrease the complexity of the encoder at a very low cost of quality and/or bit-rate.

立体视频编码中运动和视差联合估计算法的研究

In order to exploit the intra-view and inter-view dependencies in stereo video coding, a fast joint motion and disparity estimation algorithm is proposed in this paper. An efficient search strategy based on the motion-disparity constraint is developed, in which both the motion and disparity vectors can be obtained simultaneously with low complexity. Moreover, a credible initial predictive vector set is designed with the aid of the temporal and spatial neighboring blocks so that the coding efficiency can be further enhanced. The experimental results demonstrate that the proposed algorithm can reduce more than 96% coding time compared with the JMVM full search with a negligible quality drop.

A fast stereoscopic video coding algorithm based on JMVM

In order to exploit the intra-view and inter-view dependencies in stereo video coding, a fast joint motion and disparity estimation algorithm is proposed in this paper. An efficient search strategy based on the motion-disparity constraint is developed, in which both the motion and disparity vectors can be obtained simultaneously with low complexity. Moreover, a credible initial predictive vector set is designed with the aid of the temporal and spatial neighboring blocks so that the coding efficiency can be further enhanced. The experimental results demonstrate that the proposed algorithm can reduce more than 96% coding time compared with the JMVM full search with a negligible quality drop.

Fast Algorithms for Inter-view Prediction of Multiview Video Coding

The compression efficiency and complexity should be simultaneously taken into consideration in Multiview Video Coding (MVC). In order to reduce the complexity while maintaining the efficiency of MVC, we proposed in this paper two algorithms called Adaptive Search Region Adjustment (ASRA) and Adaptive Selection of Inter-view References (ASIR). In ASRA, we simplified the exiting method to obtain the global disparity vector (GDV), using which the correlations between the disparity vector (DV) and predicted vector were exploited. The statistical property of the absolute difference value of $DV$ and GDV was analyzed and the results were applied to adjust the search range dynamically. In ASIR, we first used the prediction information of the lower temporal layer in the hierarchical B picture prediction structure to reduce the number of inter-view reference pictures, and then applied the results of reference selection after inter_16×16 prediction mode decision to revise our method. The experimental results demonstrate that the algorithms suggested in this paper can reduce the computation up to 25% with little decrease of compression efficiency.

Spatial error concealment for stereoscopic video coding based on pixel matching

In stereoscopic video coding, the interview correlation between the stereo image pair can be used for error concealment. A new spatial error concealment method for stereoscopic video coding based on pixel matching in the decoder is proposed in this paper. The lost macroblocks are recovered by utilizing disparity matching between two-view images on a pixel-by-pixel basis. Firstly, we get the candidate disparity vectors of the four neighboring pixels of the lost pixel by disparity matching in the decoder. Secondly, by calculating the boundary pixel difference, we determine an optimal replacing pixel in the reference image, and then we recover the lost pixel by the optimal pixel in the reference image. Experimental results show that the proposed algorithm performs better comparing to the previous technique.

Stereo Coding with MVC

This paper presents an investigation into the coding efficiency of the multiview video coding (MVC) extension of MPEG-4 AVC/H.264. MVC adds prediction among multiple views to the temporal motion compensation and prediction known as intercoding. Here, we consider the case of only two views or stereo coding. How much coding gain should we expect from the addition of inter-view coding to the motion-only coding of left and right-eye images? To answer this question, we examine fundamental qualitative and quantitative differences between temporal motion vectors and stereo disparity vectors. These differences are related to simple properties of the images themselves, and they affect the entire prediction stage and hence overall coding efficiency. Results suggest that the differences in the statistics of motion vectors compared to those of disparity vectors, along with the statistics of the resulting residuals, offers insight into the overall coding efficiencies currently observed in stereo coding.

View-Adaptive Motion Estimation and Disparity Estimation for Low Complexity Multiview Video Coding

The emerging international standard for multiview video coding (MVC) is an extension of H.264/advanced video coding. In the joint mode of MVC, both motion estimation (ME) and disparity estimation (DE) are included in the encoding process. This achieves the highest coding efficiency but requires a very high computational complexity. In this letter, we propose a fast ME and DE algorithm that adaptively utilizes the inter-view correlation. The coding mode complexity and the motion homogeneity of a macroblock (MB) are first analyzed according to the coding modes and motion vectors from the corresponding MBs in the neighbor views, which are located by means of global disparity vector. According to the coding mode complexity and the motion homogeneity, the proposed algorithm adjusts the search strategies for different types of MBs in order to perform a precise search according to video content. Experimental results demonstrate that the proposed algorithm can save 85% computational complexity on average, with negligible loss of coding efficiency.

Fast motion and disparity estimation for multiview video coding

Multiview video involves a huge amount of data, and as such, efficiently encoding each view is a critical issue for its wider application. In this paper, a fast motion and disparity estimation algorithm is proposed, utilizing the close correlation between temporal and interview reference frames. First, a reliable predictor is found according to the correlation of motion and disparity vectors. Second, an iterative search process is carried out to find the optimal motion and disparity vectors. The proposed algorithm makes use of the prediction vector obtained in the previous motion estimation for the next disparity estimation and achieves both optimal motion and disparity vectors jointly. Experimental results demonstrate that the proposed algorithm can successfully save an average of 86% of computational time with a negligible quality drop when compared to the joint multiview video model (JMVM) full search algorithm. Furthermore, in comparison with the conventional simulcast coding, the proposed algorithm enhances the video quality and also greatly increases coding speed.

Fast disparity estimation using spatio-temporal correlation of disparity field for multiview video coding

Disparity estimation is adopted by multiview video coding (MVC) to reduce the inter-view redundancy. However, it consumes enormous computational load. In this paper, a fast disparity estimation is proposed by using the spatio-temporal correlation and the temporal variation of disparity field. For each macroblock, a temporal prediction of the disparity vector is calculated first by utilizing the smoothed disparity field of the previous coded frame. Then, the search center is selected among the candidates obtained from spatio-temporal neighboring disparity vectors, and deemed to be a preliminary disparity vector. Finally, the search range is predicted adaptively by using the distance between the search center and the temporal prediction of the disparity vector, and then the search is implemented in a limited range. The distance represents the temporal variation of the disparity vector. As compared to the full search algorithm in MVC reference software, experimental results show that an average 96% reduction of the computational complexity is achieved, while the rate-distortion performance remains the same. As compared to the fast search algorithm in MVC reference software, experimental results show that an average 43% reduction of the computational complexity is achieved, and the rate-distortion performance of the proposed algorithm has been improved.

Retinal Correspondence and the Theoretical Horopter

Disparity is usually considered a 2D vector representing deviation from retinal correspondence, and visualized as the theoretical horopter, the set of locations projecting onto correspondence pairs. Disparity is assumed to decompose naturally into two orthogonal components, horizontal and vertical disparity, processed in fundamentally different ways by the visual system. But when disparity is considered to be relative to a non-identical correspondence pattern, simple definitions break down. In general, scalar horizontal and vertical disparities and the disparity vector composed of them are not well defined entities. Retinally, a binocular target is represented by one 2D position vector per eye, or four dimensions total. If disparity is assumed to be the difference between these projection vectors and a retinal correspondence pattern, the resulting entity has eight degrees of freedom twice that of a retinal disparity vector. Only when empirical correspondence obeys certain constraints is disparity reducible to such a vector. But even then it can not be simply split into retinal horizontal and vertical components, as eye movements change epipolar projection geometry. Additionally, we demonstrate that the accepted model forms of the correspondence patterns of the Hering-Hillebrandt deviation and the Helmholtz shear are not compatible across a 2D retina.

Joint depth-motion dense estimation for multiview video coding

The multiview video coding (MVC) extension of H.264/MPEG-4 AVC [1] is one of the most promising visual encoders for three-dimensional television and free viewpoint video applications. In this paper, we propose a joint dense motion/disparity estimation algorithm, designed to replace the classical temporal/inter-view unit within MVC, which uses a block-based motion/disparity estimation. The motion vector fields and the disparity vector fields are therefore simultaneously derived using the stereo-motion consistency constraint in a set theoretic convex optimization framework. The obtained displacement vector fields are then jointly segmented by minimizing a rate-distortion cost function, in line with the multiple reference frame strategy used in H.264/MPEG-4 AVC. Experimental results demonstrate the benefits of the proposed method compared to the separated dense estimation scheme or the block-based estimation technique.

The horizontal disparity direction vs. the stimulus disparity direction in the perception of the depth of two-dimensional patterns

Horizontal binocular disparity has long been the conventional predictor of stereo depth. Surprisingly, an alternative predictor fairs just as well. This alternative predicts the relative depth of two stimuli from the relation between their disparity vectors, without regard to horizontal disparities. These predictions can differ; horizontal disparities accurately predict the perceived depth of a grating and a plaid only when the grating is vertical, while the vector calculation accurately predicts it at all except near-horizontal grating orientations. For spatially two-dimensional stimulus pairs, such as plaids, dots, and textures, the predictions cannot be distinguished when the stimuli have the same disparity direction or when the disparity direction of one of the stimuli is horizontal or has a magnitude of zero. These are the conditions that have prevailed in earlier studies. We tested whether the perceived depth of two-dimensional stimuli depends on relative horizontal disparity magnitudes or on relative disparity magnitudes along a disparity axis. On both measures tested-depth matches and depth-interval matches-the perceived depth of plaids varied with their horizontal disparities and not with disparity direction differences as observed for grating-plaid pairs. Differences in disparity directions as great as 120 degrees did not affect depth judgments. This result, though opposite the grating-plaid data, is consistent with them and provides a view into the construction of orientation-invariant disparity representations.

From disparity to depth: How to make a grating and a plaid appear in the same depth plane

Even though binocular disparity is a very well-studied cue to depth, the function relating disparity and perceived depth has been characterized only for the case of horizontal disparities. We sought to determine the general relationship between disparity and depth for a particular set of stimuli. The horizontal disparity direction is a special case, albeit an especially important one. Non-horizontal disparities arise from a number of sources under natural viewing condition. Moreover, they are implicit in patterns that are one-dimensional, such as gratings, lines, and edges, and in one-dimensional components of two-dimensional patterns, where a stereo matching direction is not well-defined. What function describes perceived depth in these cases? To find out, we measured the phase disparities that produced depth matches between a reference stimulus and a test stimulus. The reference stimulus was two-dimensional, a plaid; the test stimulus was one-dimensional, a grating. We find that horizontal disparity is no more important than other disparity directions in determining depth matches between these two stimuli. As a result, a grating and a plaid appear equal in depth when their horizontal disparities are, in general, unequal. Depth matches are well predicted by a simple disparity vector calculation; they survive changes in component parameters that conserve these vector quantities. The disparity vector rule also describes how the disparities of 1-D components might contribute to the perceived depth of 2-D stimuli.

Multi-View Video Coding Based on Vector Estimation and Weighted Disparity Interpolation

Aiming at fully exploiting the temporal and spatial redundancy and answering the "Call for Proposals" for multi-view video coding (MVC) issued by MPEG, a MVC scheme based on vector field estimation and weighted disparity interpolation is presented. By extending the loop constraint to multi-view images for a parallel camera model and proposing the novel "vector field estimation" scheme, the temporal and spatial redundancy is significantly reduced. Also, weighted disparity interpolation is performed to predict adjacent disparity vectors. Experimental results over multi-view image sets imply that the coding efficiency is improved about 0.2---0.5 dB compared with previous coding approaches such as H.264/AVC simulcast and JMVM.