3D Video Applications Research Articles

Depth Perception Assessment of 3D Videos Based on Stereoscopic and Spatial Orientation Structural Features

Depth quality of stereoscopic three-dimensional (S3D) videos is a significant factor which directly affects the quality of experience (QoE) associated with 3D video applications and services. Nevertheless, there remain limited reports on the investigation of depth perception and depth quality evaluation of S3D videos, which impedes further advancement and deployment of 3D video technology. This paper reports a series of subjective experiments which have been conducted to investigate the depth perception and its related properties of the human visual system (HVS) using S3D video compressed by the H.264/AVC standard. The experimental results reveal that the HVS response in depth perception varies at different frequencies and in varying orientations, and the distortions introduced by video coding can cause the loss of and/or variation in depth perception. By integration of binocular and monocular features (BM) extracted from left and right views of S3D video with respect to depth perception, a depth quality assessment model, herein referred to as BM-DQAM, is devised by training these stereoscopic and spatial orientation structural features with a support vector regression model. It is shown that the BM-DQAM provides a novel no-reference metric for evaluation of the depth quality of S3D videos. Based on two publicly available 3D video databases and the proposed depth perception assessment database, the experimental results show that the BM-DQAM has demonstrated better performance in assessing the depth quality in S3D video viewing than that of other metrics reported in the published literatures, correlating well with the HVS response in the depth perception assessment experiment.

As-Deformable-As-Possible Single-Image-Based View Synthesis Without Depth Prior

Depth-image-based rendering (DIBR) technologies have been widely employed to synthesize novel realistic views from a single image in 3D video applications. However, DIBR-oriented approaches heavily rely on the accuracy of depth maps, usually requiring the depth GT as a prior. Despite that, there might exist extensive float precision losses and invalid holes in the synthesized view due to warping error and occlusion. In this paper, we propose an end-to-end as-deformable-as-possible (ADAP) single-image-based view synthesis solution without depth prior. It addresses the above issues through two stages: alignment and reconstruction, where we first transform the input image to the latent feature space and then reconstruct the novel view in the image domain. In the first stage, the input image is deformed to align with the synthesized view at feature level. To this end, we propose an ADAP alignment mechanism through pixel-level warping to error-level quantization to feature-level alignment, progressively improving the deformable capability in handling challenging motion conditions in real-world scenes. In the second stage, we exploit an occlusion-aware reconstruction module to recover the content details from the deformed feature at pixel level. Extensive experiments demonstrate that our alignment-reconstruction approach is robust to the depth map. Even with a coarsely estimated depth map, our solution outperforms other SoTA schemes in the popular benchmarks.

Multi-layer and Multi-scale feature aggregation for DIBR-Synthesized image quality assessment

Depth-Image-Based Rendering (DIBR) is one of the main fundamental techniques for generating new viewpoints in 3D video applications such as multi-viewpoint video (MVV), free viewpoint video (FVV) and virtual reality (VR). Due to the imperfections of color images, depth maps or texture restoration techniques, several types of distortions occur in synthesized views. However, most of related works evaluated the quality of DIBR-synthesized views by only detecting a specific type of distortion, such as stretching, black holes, blurring, etc., which were unable to accurately evaluate the quality of DIBR-synthesized views. In this paper, a new no-reference image quality assessment method is proposed to evaluate the quality of DIBR-synthesized images by combining multi-layer and multi-scale features of images. To be specific, the distortions introduced by different stages of virtual viewpoint synthesis are first analyzed, and then multi-layer and multi-scale features are extracted to estimate the degree of texture and structure distortions. As a result, individual quality scores associated with two types of distortions (e.g., structural distortion and texture distortion) are aggregated to an overall image quality. Experimental results on two publicly available DIBR datasets show that the method has better performance than the state-of-the-art models.Index Terms: image quality assessment, DIBR-synthesized image, distortion correction, BIQA.

Quality Assessment of View Synthesis Based on Visual Saliency and Texture Naturalness

Depth-Image-Based-Rendering (DIBR) is one of the core techniques for generating new views in 3D video applications. However, the distortion characteristics of the DIBR synthetic view are different from the 2D image. It is necessary to study the unique distortion characteristics of DIBR views and design effective and efficient algorithms to evaluate the DIBR-synthesized image and guide DIBR algorithms. In this work, the visual saliency and texture natrualness features are extracted to evaluate the quality of the DIBR views. After extracting the feature, we adopt machine learning method for mapping the extracted feature to the quality score of the DIBR views. Experiments constructed on two synthetic view databases IETR and IRCCyN/IVC, and the results show that our proposed algorithm performs better than the compared synthetic view quality evaluation methods.

Quality assessment of DIBR-synthesized views: An overview

The Depth-Image-Based-Rendering (DIBR) is one of the main fundamental technique to generate new views in 3D video applications, such as Multi-View Videos (MVV), Free-Viewpoint Videos (FVV) and Virtual Reality (VR). However, the quality assessment of DIBR-synthesized views is quite different from the traditional 2D images/videos. In recent years, several efforts have been made towards this topic, but there is a lack of detailed survey in the literature. In this paper, we provide a comprehensive survey on various current approaches for DIBR-synthesized views. The current accessible datasets of DIBR-synthesized views are firstly reviewed, followed by a summary analysis of the representative state-of-the-art objective metrics. Then, the performances of different objective metrics are evaluated and discussed on all available datasets. Finally, we discuss the potential challenges and suggest possible directions for future research.

Perceptual quality assessment of 3D videos with stereoscopic degradations

In the last decade, several aspects of the 3D video technology have been improved, including the 3D content production, distribution, and display. Still, the level of acceptability and popularity of 3D video applications are strongly correlated to the user Quality of Experience (QoE). Since research in this area depends heavily on data acquired in psychophysical experiments, public databases with typical stereoscopic degradations are considered important tools to researchers. Although currently there are number of available public 3D video quality databases, most of them contain only compression and transmission degradations. In this work, our first goal was to build a database (UnB-3D) that contains stereoscopic distortions. We created a set of Computer Graphics Imaging (CGI) scenes and rendered it using different parameters, generating 3D videos containing stereoscopic degradations at different strengths. Our second goal is to understand how these stereoscopic degradations are perceived by viewers. So, we performed a psychophysical experiment to analyze the perceived quality and comfort of these videos. Finally we conducted the statistical analysis and model generation. Results shows that users that have little familiarity with 3D content have difficulties identifying stereoscopic distortions. Also, the source content has a great influence on the user’s comfort. Similarly, the 3D quality is affected by the spatial and temporal information of the content, specially when the disparity is high.

Efficient Shape Coding for Object-Based 3D Video Applications

Shape is a popular way to define objects and shape coding is a key technique for object-based 3D video applications. In this paper, the issue of efficient shape coding for object-based 3D video applications is addressed, and a novel contour-based and chain-represented scheme is proposed. For a given 3D shape video, contour extraction and preprocessing are first implemented followed by chain-based representation. Then, to achieve high coding efficiency, a chain-based prediction and compensation technique is developed based on joint motion-compensated prediction and disparity-compensated prediction to effectively exploit the intra-view temporal correlation and the inter-view spatial correlation. Experiments are conducted, and the results demonstrate that the proposed scheme is more efficient than the existing methods, including state-of-the-art methods.

Network-Assisted Neural Adaptive Naked-Eye 3D Video Streaming Over Wireless Networks

High quality transmission of Virtual Reality (VR) video depends strictly on bandwidth and delay requirements. It becomes possible with the maturity and commercialization of 5G technology. However, to transmit VR video streaming over wireless communication system, we must cope with the challenge that fluctuation exists in the wireless channel conditions. For example, available channel bandwidth, packet loss rate, and interference level may vary with time due to channel fading and user's mobility. The problem is more prominent when transmitting naked-eye 3D video which generally consists of multiple viewpoints with different resolutions. In order to optimize the quality of experience (QoE) of watching naked-eye VR video over wireless networks, this paper proposes a network-assisted neural adaptive video streaming algorithm (NAVSA). Specifically, we present a modified QoE function to describe the quality of naked-eye 3D streaming quantitatively, which not only considers the quality of naked-eye 3D video itself, but also considers the phenomenon of rebuffering and video fluctuation that occurs during video transmission. Next, with the network-assisted feedback, the physical layer information, the buffer occupancy of the video client, and the size of the next video chunk are collected to train a reinforcement learning model. Based on dynamic adaptive streaming over HTTP (DASH), the model can automatically choose appropriate viewpoints and resolutions corresponding to the current condition of the wireless networks such that the network capacity can be fully explored. To verify the performance of our proposed NAVSA, we simulate 3 naked-eye 3D video application scenarios on the NS3 platform. The results show that the performance of NAVSA is about 5~8% better than some state-of-the-art algorithms in wireless networks.

Combining Local and Global Measures for DIBR-Synthesized Image Quality Evaluation.

Depth-Image-Based-Rendering (DIBR) techniques are significant for three-dimensional (3D) video applications, e.g., 3D television and free viewpoint video (FVV). Unfortunately, the DIBR-synthesized image suffers from various distortions, which induce an annoying viewing experience for the entire FVV. Proposing a quality evaluator for DIBR-synthesized images is fundamental for the design of perceptual friendly FVV systems. Since the associated reference image is usually not accessible, full-reference (FR) methods cannot be directly applied for quality evaluation of the synthesized image. In addition, most traditional no-reference (NR) methods fail to effectively measure the specifically DIBR-related distortions. In this paper, we propose a novel NR quality evaluation method accounting for two categories of DIBR-related distortions, i.e., geometric distortions and sharpness. First, the disoccluded regions, as one of the most obvious geometric distortions, are captured by analyzing local similarity. Then, another typical geometric distortion (i.e., stretching) is detected and measured by calculating the similarity between it and its equal-size adjacent region. Second, considering the property of scale invariance, the global sharpness is measured as the distance between the distorted image and its downsampled version. Finally, the perceptual quality is estimated by linearly pooling the scores of two geometric distortions and sharpness together. Experimental results verify the superiority of the proposed method over the prevailing FR and NR metrics. More specifically, it is superior to all competing methods except APT in terms of effectiveness, but greatly outmatches APT in terms of implementation time.

Reliable 3D video streaming considering region of interest

3D video applications are growing more common as communication technology becomes more predominant nowadays. With such increasing demand for the 3D multimedia services in either the wired or wireless networks, robust methods of video streaming will be introduced to show more favorable efficiency outcomes since packet failure is an integral characteristic of communication networks.This paper aims to introduce a new reliable method of stereoscopic video streaming based on multiple description coding (MDC) strategy. The proposed multiple description coding generates four 3D video descriptions considering the interesting objects contained in the scene. To be able to find the interesting objects in the scene, we use two metrics from the second-order statistics of the depth map image in a block-wise manner. Having detected the objects, the proposed multiple description coding algorithm generates the descriptions for the color video using a nonidentical decimation method with respect to the identified objects. To show how much reliable the proposed MDC method is, this article assumes that due to the unreliable communication channel, only one description, among four encoded descriptions, is delivered to the receiver successfully. Therefore, the receiver needs to estimate the missed descriptions’ data from the available description. Since the human eye is more sensitive to objects than it is to pixels, the proposed method provides a better visual performance in view of its subjective assessment. Although, the objective test results verify the fact that the proposed method provides an improved performance than the Polyphase SubSampling (PSS) multiple description coding and our previous work using pixel variation. Regarding the depth map image, the proposed method generates the multiple descriptions according to the pixel prediction difficulty level. The considerable improvement achieved by the proposed method is shown with the peak signal-to-noise ratio (PSNR) and Structural SIMilarity (SSIM) simulation result.

High-resolution depth map generator for 3D video applications using time-of-flight cameras

3D video applications are easily accessible to consumers in this era. In general, high quality depth maps are crucial for interpolating virtual view images that are vital in 3D video applications. Many applications adopt time-of-flight (ToF) cameras due to their real-time distance capturing. Yet, the raw ToF image exhibit limitations such as inaccurate distance values in object boundaries and areas that can absorb the infrared ray. This paper presents a method that handles this issue. The existing methods do not focus on this problem. The proposed method is tested on a camera system that consists of a color camera and a ToF camera. Assuming immersive applications such as teleconferencing or virtual broadcasting, the object of interest is captured. The errors existing in the ToF images are modified as a preprocessing step. Afterward, the low-resolution ToF image is warped to the viewpoint of the color camera; empty areas are filled considering the neighbor information. The depth map results show improvements over the state-of-the-art method. The proposed method can increase the overall quality of the 3D video system, ultimately making 3D video consumer applications more marketable.

DIBR-synthesized image quality assessment based on morphological multi-scale approach

The depth-image-based rendering (DIBR) algorithms used for 3D video applications introduce new types of artifacts mostly located around the disoccluded regions. As the DIBR algorithms involve geometric transformations, most of them introduce non-uniform geometric distortions affecting the edge coherency in the synthesized images. Such distortions are not handled efficiently by the common image quality assessment metrics which are primarily designed for other types of distortions. In order to better deal with specific geometric distortions in the DIBR-synthesized images, we propose a full-reference metric based on multi-scale image decomposition applying morphological filters. Using non-linear morphological filters in multi-scale image decomposition, important geometric information such as edges is maintained across different resolution levels. Edge distortion between the multi-scale representation subbands of the reference image and the DIBR-synthesized image is measured precisely using mean squared error. In this way, areas around edges that are prone to synthesis artifacts are emphasized in the metric score. Two versions of morphological multiscale metric have been explored: (a) Morphological Pyramid Peak Signal-to-Noise Ratio metric (MP-PSNR) based on morphological pyramid decomposition, and (b) Morphological Wavelet Peak Signal-to-Noise Ratio metric (MW-PSNR) based on morphological wavelet decomposition. The performances of the proposed metrics have been tested using two databases which contain DIBR-synthesized images: the IRCCyN/IVC DIBR image database and MCL-3D stereoscopic image database. Proposed metrics achieve significantly higher correlation with human judgment compared to the state-of-the-art image quality metrics and compared to the tested metric dedicated to synthesis-related artifacts. The proposed metrics are computationally efficient given that the morphological operators involve only integer numbers and simple computations like min, max, and sum as well as simple calculation of MSE. MP-PSNR has slightly better performances than MW-PSNR. It has very good agreement with human judgment, Pearson’s 0.894, Spearman 0.77 when it is tested on the MCL-3D stereoscopic image database. We have demonstrated that PSNR has particularly good agreement with human judgment when it is calculated between images at higher scales of morphological multi-scale representations. Consequently, simplified and in essence reduced versions of multi-scale metrics are proposed, taking into account only detailed images at higher decomposition scales. The reduced version of MP-PSNR has very good agreement with human judgment, Pearson’s 0.904, Spearman 0.863 using IRCCyN/IVC DIBR image database.

Depth Intra Coding for 3D Video Based on Geometric Primitives

This paper presents an advanced depth intra-coding approach for 3D video coding based on the High Efficiency Video Coding (HEVC) standard and the multiview video plus depth (MVD) representation. This paper is motivated by the fact that depth signals have specific characteristics that differ from those of natural signals, i.e., camera-view video. Our approach replaces conventional intra-picture coding for the depth component, targeting a consistent and efficient support of 3D video applications that utilize depth maps or polygon meshes or both, with a high depth coding efficiency in terms of minimal artifacts in rendered views and meshes with a minimal number of triangles for a given bit rate. For this purpose, we introduce intra-picture prediction modes based on geometric primitives along with a residual coding method in the spatial domain, substituting conventional intra-prediction modes and transform coding, respectively. The results show that our solution achieves the same quality of rendered or synthesized views with about the same bit rate as MVD coding with the 3D video extension of HEVC (3D-HEVC) for high-quality depth maps and with about 8% less overall bit rate as with 3D-HEVC without related depth tools. At the same time, the combination of 3D video with 3D computer graphics content is substantially simplified, as the geometry-based depth intra signals can be represented as a surface mesh with about 85% less triangles, generated directly in the decoding process as an alternative decoder output.

HEVC-based 3D holoscopic video coding using self-similarity compensated prediction

Holoscopic imaging, also known as integral, light field, and plenoptic imaging, is an appealing technology for glassless 3D video systems, which has recently emerged as a prospective candidate for future image and video applications, such as 3D television. However, to successfully introduce 3D holoscopic video applications into the market, adequate coding tools that can efficiently handle 3D holoscopic video are necessary. In this context, this paper discusses the requirements and challenges for 3D holoscopic video coding, and presents an efficient 3D holoscopic coding scheme based on High Efficiency Video Coding (HEVC). The proposed 3D holoscopic codec makes use of the self-similarity (SS) compensated prediction concept to efficiently explore the inherent correlation of the 3D holoscopic content in Intra- and Inter-coded frames, as well as a novel vector prediction scheme to take advantage of the peculiar characteristics of the SS prediction data. Extensive experiments were conducted, and have shown that the proposed solution is able to outperform HEVC as well as other coding solutions proposed in the literature. Moreover, a consistently better performance is also observed for a set of different quality metrics proposed in the literature for 3D holoscopic content, as well as for the visual quality of views synthesized from decompressed 3D holoscopic content.

Multi–Scale Synthesized View Assessment Based on Morphological Pyramids

Abstract The Depth-Image-Based-Rendering (DIBR) algorithms used for 3D video applications introduce geometric distortions affecting the edge coherency in the synthesized images. In order to better deal with specific geometric distortions in the DIBR synthesized images, we propose full-reference metric based on multi-scale pyramid decompositions using morphological filters. The non-linear morphological filters used in multi-scale image decompositions maintain important geometric information such as edges across different resolution levels. We show that PSNR has particularly good agreement with human judgment when it is calculated between detailed images at higher scales of morphological pyramids. Consequently, we propose reduced morphological pyramid peak signal-to-noise ratio metric (MP-PSNR), taking into account only mean squared errors between pyramids’ images at higher scales. Proposed computationally efficient metric achieves significantly higher correlation with human judgment compared to the state-of-the-art image quality assessment metrics and compared to the tested metric dedicated to synthesis-related artifacts.

Multi-directional Hole Filling Method for Virtual View Synthesis

In 3D video applications, view synthesis is widely used for generating a virtual viewpoint images. However, the quality of synthesized image is dissimilar to captured color image. Furthermore, noise occurs at object boundary areas in synthesized images. In this paper, we propose a new hole filling method to improve the synthesized image quality. In order to synthesize an image without the noise, we remove the boundary noise by expanding it as a hole region. Based on the expanded hole region, we use proposed hole scanning and directional hole filling method in simultaneous time. Additionally, we compute a cost value among the candidate pixels to find out proper hole filling pixel. We conducted experiments on various test sequences for evaluation of the proposed method. The proposed method produces better objective results than other conventional hole filling methods.

Depth-Based Texture Coding in AVC-Compatible 3D Video Coding

The target of 3D video coding is to compress Multiview Video plus Depth (MVD) format data, which consist of a texture image and its corresponding depth map. In the MVD format, the depth map plays an important role for successful services in 3D video applications, because it enables the user to experience 3D by generating arbitrary intermediate views. The depth map has a strong correlation with its associated texture data, so it can be utilized to improve texture coding efficiency. This paper introduces a novel and efficient depth-based texture coding scheme. It includes depth-based motion vector prediction, block-based view synthesis prediction, and adaptive luminance compensation, which were adopted in an AVC-compatible 3D video coding standard. Simulation results demonstrate that the proposed scheme reduces the total coding bitrates of texture and depth by 19.06% for the coded PSNR and 17.01% for the synthesized PSNR in a P-I-P view prediction structure, respectively.

Hoda Roodaki

With the growing demand for 3D video, efforts are underway to incorporate it in the next generation of broadcast and streaming applications and standards. Scalability is one possible solution to reduce the amount of data in multi-view/3D video in heterogeneous environments. But using Scalable Multi-view Video Coding (SMVC) for multi-view/3D video still has many unresolved challenges. In this thesis, we propose an adaptive framework to use SMVC in various 3D video applications effectively. For this issue, first, the proper scalable modality should be selected according to the application at hand, its related features and requirements. To the best of our knowledge, no work has systematically defined new and proper scalable modalities specifically for multi-view 3D video, so far. Hence, at the first step of the proposed framework, we will suggest a methodology to extract the proper scalable modalities for multi-view/3D video. In addition, while SMVC can help support heterogeneous receivers, the question becomes: how to scale the 3D video content in a given type of scalability and a specified application in order to achieve the highest performance and satisfy the receivers? constraint as much as possible? In other words, the proper mechanism to assign SMVC data to various layers should be clearly determined. This issue is considered as the second step of our proposed framework. This method uses the inter-layer and intralayer disparity concepts. Note that specific features of any given scalable modality should be used to define these concepts in that specific scalable modality. Simulation results indicate that the proposed method achieves relatively better compression rate for each layer, with much less overhead. At the next step of our proposed framework, we propose an analytical view-level rate model for multiview video coding. Our rate model takes into account both previous theoretical results as well as new results specifically obtained for multi-view video and confirmed by comprehensive practical experiments. Simulation results show that our model can predict the rate of each view with relatively high precision and a low estimation error of 12% on average for tested sequences. In addition, the evaluation of the overall visual quality of scalable multi-view video requires a new objective perceptual quality measure specifically designed for scalable multi-view/3D video. Although several subjective and objective quality assessment methods have been proposed for multi-view/3D sequences, no comparable attempt has been made for quality assessment of scalable multi-view/3D video so far. Hence, in this framework, we propose a new methodology to build suitable objective quality assessment metrics for different scalable modalities in multi-view/3D video. Our proposed methodology considers the importance of each layer and its content as a quality of experience factor in the overall quality. Furthermore, in addition to the quality of each layer, the concept of inter-layer and intra-layer disparity is considered as an effective feature to evaluate overall perceived quality more accurately. Our simulation results indicate that the correlation coefficient between our extracted objective quality evaluation metric and subjective quality assessment is 0.8 on average for tested video sequences. At the last step of our proposed framework, we present a novel method for rate-distortion optimization in scalable multi-view video that tries to minimize the perceptual distortion of decoded video under the conditions that the sum of bits generated from different views is constrained within a given bit budget. Since the constraint-based optimization problem is usually computational intensive, our proposed approach considers the concept of intra-layer and inter-layer disparity to reduce this computational complexity. Experimental results show that the proposed approach uses on average 24% and 42% less bitrate than the H.264/AVC rate-distortion optimization for base and base plus enhancement layers, respectively. Although the thesis is in Farsi (Persian), the following English papers capture most of its essence: H. Roodaki, M.R. Hashemi, and S. Shirmohammadi, ?A New Methodology to Derive Objective Quality Assessment Metrics for Scalable Multi-view 3D Video Coding?, ACM Transactions on Multimedia Computing, Communications, and Applications, Vol. 8, No. 3S, Article 44, September 2012, 25 pages. DOI: 10.1145/2348816.2348823 H. Roodaki, M.R. Hashemi, and S. Shirmohammadi, ?Rate-Distortion Optimization for Scalable Multi-View Video Coding?, Proc. IEEE International Conference on Multimedia and Expo, Chengdu, China, July 14-18 2014, 6 pages. DOI: 10.1109/ICME.2014.6890275 H. Roodaki, Z. Iravani, M.R. Hashemi, S. Shirmohammadi, and M. Gabbouj, ?A New Rate Distortion Model for Multi-View/3D Video Coding?, Proc. IEEE International Workshop on Hot Topics in 3D, in Proc. IEEE International Conference on Multimedia and Expo, July 15-19 2013, San Jose, USA, 6 pages. DOI: 10.1109/ICMEW.2013.6618338 H. Roodaki, M.R. Hashemi, and S. Shirmohammadi, ?New Scalable Modalities in Multi-view 3D Video?, Proc. ACM Workshop on Mobile Video, Oslo, Norway, February 27 2013, pp. 25-30. DOI: 10.1145/2457413.2457420

Learning-based saliency model with depth information.

Most previous studies on visual saliency focused on two-dimensional (2D) scenes. Due to the rapidly growing three-dimensional (3D) video applications, it is very desirable to know how depth information affects human visual attention. In this study, we first conducted eye-fixation experiments on 3D images. Our fixation data set comprises 475 3D images and 16 subjects. We used a Tobii TX300 eye tracker (Tobii, Stockholm, Sweden) to track the eye movement of each subject. In addition, this database contains 475 computed depth maps. Due to the scarcity of public-domain 3D fixation data, this data set should be useful to the 3D visual attention research community. Then, a learning-based visual attention model was designed to predict human attention. In addition to the popular 2D features, we included the depth map and its derived features. The results indicate that the extra depth information can enhance the saliency estimation accuracy specifically for close-up objects hidden in a complex-texture background. In addition, we examined the effectiveness of various low-, mid-, and high-level features on saliency prediction. Compared with both 2D and 3D state-of-the-art saliency estimation models, our methods show better performance on the 3D test images. The eye-tracking database and the MATLAB source codes for the proposed saliency model and evaluation methods are available on our website.

Fractal Depth Map Sequence Coding Algorithm with Motion-vector-field-based Motion Estimation

Three-dimensional video coding is one of the main challenges restricting the widespread applications of 3D video and free viewpoint video. In this paper, a novel fractal coding algorithm with motion-vector-field-based motion estimation for depth map sequence is proposed. We firstly add pre-search restriction to rule the improper domain blocks out of the matching search process so that the number of blocks involved in the search process can be restricted to a smaller size. Some improvements for motion estimation including initial search point prediction, threshold transition condition and early termination condition are made based on the feature of fractal coding. The motion-vector-field-based adaptive hexagon search algorithm on the basis of center-biased distribution characteristics of depth motion vector is proposed to accelerate the search. Experimental results show that the proposed algorithm can reach optimum levels of quality and save the coding time. The PSNR of synthesized view is increased by 0.56 dB with 36.97% bit rate decrease on average compared with H.264 Full Search. And the depth encoding time is saved by up to 66.47%. Moreover, the proposed fractal depth map sequence codec outperforms the recent alternative codecs by improving the H.264/AVC, especially in much bitrate saving and encoding time reduction.