Joint Multiview Video Model Research Articles

Multiview video coding with an improved prediction structure for faster random access

As any existing video codec, multiview video coding requires a highly efficient compression as well as a faster random access to any given view. Such efficiency for the coding approach can be achieved by exploiting both the temporal and the inter-view correlation. The temporal and view random access are improved by the increase of intracoded pictures (I pictures), which requires more bit rate. In this research study, we propose and evaluate an improved inter-view prediction structure mainly based on reducing the complexity of view random access at any instance T n and improving the inter-view correlation for P-view. These improvements can be guaranteed by the appropriate choice of the position of I-view, the type of the different views, and the number of each frame type to use. Compared to the prediction scheme, which is based on the hierarchical B pictures structure and used for joint multiview video model, experimental results have shown that the proposed prediction structure provides better bit-rate performance of up to 6.81% with a similar compression quality measured in peak signal-to-noise ratio and significantly improves view random access by up to 25%.

Fast Mode Decision for Multiview Video Coding Using Mode Correlation

Exhaustive mode decision has been exploited in multiview video coding for effectively improving the coding efficiency, but at the expense of yielding much higher computational complexity. In this paper, a fast mode decision algorithm, called the mode correlation-based mode decision (MCMD), is proposed to speed up the encoding process by reducing the number of the modes required to be checked. In our approach, all the prediction modes are first categorized into five motion-activity classes, and only one of them will be chosen to identify the optimal mode in a hierarchical manner, as follows. For each macroblock (MB), the proposed MCMD algorithm always begins with checking whether the rate-distortion cost computed at the SKIP mode (i.e., Class 1) is below an adaptive threshold for providing a possible early termination chance. If this early termination condition is not met, one of the remaining four motion-activity classes will be chosen for further mode checking according to the analysis of the predicted motion vector (PMV) of the current MB. The above-mentioned adaptive threshold and PMV are derived by exploiting the mode correlation between the current MB and a set of adjacent MBs (i.e., region of support) in the current view and its neighboring view. Experimental results have shown that compared with exhaustive mode decision, which is a default approach set in the joint multiview video model (JMVM) reference software, the proposed MCMD algorithm achieves a reduction of the computational complexity by 73.39% on average, while incurring only 0.07 dB loss in peak signal-to-noise ratio (PSNR) and 2.22% increment on the total bit rate.

Adaptive early termination of fast mode decision in MVC based on inter-view correlation

Variable size motion estimation (ME) and disparity estimation (DE) are employed to select the best coding mode for each macroblock (MB) in the current joint multiview video model (JMVM). This technique achieves the highest possible coding efficiency, but it results in extremely large computation complexity which obstructs the multiview video coding (MVC) from practical application. This paper proposes an adaptive early termination of fast mode decision algorithm for MVC. It makes use of the coding information of the corresponding MBs in neighbor view based on inter-view correlation to early terminate the mode decision procedure. Experimental results show that the proposed fast mode decision algorithm can achieve computational 50% computation saving with no significant loss of rate distortion (RD) performance.

High-speed inter-view frame mode decision procedure for multi-view video coding

With the amazing development of visual identification systems, multi-view video which is one of the main types of three-dimensional (3D) video signals, captured by a set of video cameras from various viewpoints, has attracted much interest recently. The multi-view video coding (MVC) uses the joint multi-view video model (JMVM) established on H.264/AVC as the core codec to achieve excellent coding efficiency. However, the concomitant problem is the huge encoding complexity especially due to the heavy Rate-to-Distortion Cost (RDCost) computational load of the mode decision (MD) process, which limits the applications of MVC for mobile terminals and embedded systems. In this paper, a high-speed MD (HSMD) algorithm is proposed for the inter-view frames of multi-view video signal. The redundant candidate modes of each macroblock (MB) in the inter-view frames are eliminated to reduce the original RDCost computational load by utilizing multiple efficiency parameters. The quantitative analysis verifies our proposal in theory and the simulation results show that the proposed HSMD algorithm can reduce the encoding time by over 90% for the inter-view video frames with negligible quality loss compared with the original JMVM codec. The proposed algorithm can be widely employed for real-time multi-view signal encoding especially under the mobile and embedded environments to realize the real-time encoding process.

Optimizing inter-view prediction structures for multi-view video coding using simulated annealing

New video applications, such as 3D video and free viewpoint video, require efficient compression of multi-view video. In addition to temporal redundancy, exploiting the inter-view redundancy is crucial to improve the performance of multi-view video coding. In this paper, we present a novel method to construct the optimal inter-view prediction structure for multi-view video coding using simulated annealing. In the proposed model, the design of the prediction structure is converted to the arrangement of coding order. Then, a simulated annealing algorithm is employed to minimize the total cost for obtaining the best coding order. This method is applicable to arbitrary irregular camera arrangements. As experiment results reveal, the annealing process converges to satisfactory results rapidly and the generated optimal prediction structure outperforms the reference prediction structure of the joint multi-view video model (JMVM) by 0.1–0.8 dB PSNR gains.

Fast Macroblock Mode Selection Algorithm for B Frames in Multiview Video Coding

Intensive computational complexity is an obstacle of enabling multiview video coding for real-time applications. In this paper, we present a fast macroblock (MB) mode selection algorithm for B frames which are based on the computational complexity analyses between the MB mode selection and reference frame selection. Three strategies are proposed to reduce the coding complexity jointly. First, the temporal correlation of MB modes between current MB and its temporal corresponding MBs is utilized to reduce computational complexity in determining the optimal MB mode. Secondly, Lagrangian cost of SKIP mode is compared with that of Inter16×16 modes to early terminate the mode selection process. Thirdly, reference frame correlation among different Inter modes is exploited to reduce the number of reference frames. Experimental results show that the proposed algorithm can promote the encoding speed by 3.71~7.22 times with 0.08dB PSNR degradation and 2.03% bitrate increase on average compared with the joint multiview video model.

Joint coding of multiview video and depth data using virtual view synthesis

AbstractTo compress multiview video and depth information, we synthesize a virtual image for the current view using color and depth data of neighboring views. In this article, we then use a view interpolation prediction scheme at the virtual image to improve the inter‐view prediction. We also propose a solution for overlapping regions and empty holes that are generated during the intermediate view synthesis process due to occlusion and disocclusion situations. Experimental results show that the proposed methods achieve approximately 0.65 dB of Peak Signal‐to‐Noise Ratio (PSNR) gain on average for multiview depth data and 0.17 dB of PSNR gain for multiview video coding, compared with the reference software, Joint Multiview Video Model 1.0. We also show that our method is even more powerful for smaller search ranges. Furthermore, we examine the effects of inter‐view prediction on hierarchical B‐pictures. © 2010 Wiley Periodicals, Inc. Int J Imaging Syst Technol, 20, 370–377, 2010

A Fast Multi-reference Frame Selection Algorithm for Multiview Video Coding

Multiview video coding (MVC) plays an important role in three-dimensional video applications. Joint Video Team developed a joint multiview video model (JMVM) with multi-reference frame technology. Motion and disparity estimation are employed in multi-reference frame technology to provide better rate distortion performance. However, the process of searching blocks with variable sizes for motion and disparity estimation in multi-reference frames significantly increases the computational complexity. After analyzing the statistical features of multi-reference frames in MVC, this paper proposes a fast multi-reference frame selection algorithm based on a dynamic threshold technology correlated with the types of frames. All views of prediction structure in JMVM are categorized into three types, that is, the basic view without inter-view reference relation, the first layer views that only refer anchor frame from upper view, the second layer views that need to refer the basic view and the first layer views. Furthermore, all frames in MVC prediction structure are divided into three categories, the anchor frame without any reference, non-anchor frame in the base view and the first layer view, and the non-anchor frame in other views. Dynamic threshold technique is given to terminate the process of searching multi-reference frames early. The proposed algorithm also reduces the useless candidate reference frames by filtering out those reference frames that are less likely to contain the best matched results and achieves significant speed-up. Experimental results show that the proposed algorithm can achieve 50.13% ~ 72.19% reduction of encoding time in comparison with JMVM7.0, while the proposed algorithm hardly influences the rate distortion performance of MVC.

Fast motion and disparity estimation for multiview video coding

Multiview video involves a huge amount of data, and as such, efficiently encoding each view is a critical issue for its wider application. In this paper, a fast motion and disparity estimation algorithm is proposed, utilizing the close correlation between temporal and interview reference frames. First, a reliable predictor is found according to the correlation of motion and disparity vectors. Second, an iterative search process is carried out to find the optimal motion and disparity vectors. The proposed algorithm makes use of the prediction vector obtained in the previous motion estimation for the next disparity estimation and achieves both optimal motion and disparity vectors jointly. Experimental results demonstrate that the proposed algorithm can successfully save an average of 86% of computational time with a negligible quality drop when compared to the joint multiview video model (JMVM) full search algorithm. Furthermore, in comparison with the conventional simulcast coding, the proposed algorithm enhances the video quality and also greatly increases coding speed.

Depth perceptual region-of-interest based multiview video coding

MultiView Video (MVV) has attracted considerable attention recently since it is capable of providing users with three-dimensional perception and interactive functionalities. However, these MVV data require large mount of storage and bandwidth in network transmission. In this paper, we present a novel Depth Perceptual Region-Of-Interest (DP-ROI) based Multiview Video Coding (RMVC) scheme to extensively improve data compression efficiency by exploiting redundancies in depth perception. Firstly, we define DP-ROI according to the three-dimensional depth sensation of human visual system. Then, a framework of RMVC is developed to improve compression efficiency by properly segmenting the MVV into different macroblock wise DP-ROIs and encoding them separately. And then, we propose three fast depth based DP-ROI extraction and tracking algorithms by jointly using motion, texture, depth as well as previous extracted DP-ROIs. Finally, on the basis of the extracted DP-ROI, bit allocation optimization model is proposed to allocate more bits on DP-ROIs for high image quality and fewer bits on background regions for high compression ratio. Experimental results show that the presented RMVC scheme achieves significant coding gains at high rate while comparing with original joint multiview video model. To be specific, up to 14.22–23.32% bit-rate are saved while 0.16–0.68 dB coding gains are achieved in DP-ROIs at the cost of the image quality degradation in background.

RD-optimized interactive streaming of multiview video with multiple encodings

This paper presents a rate-distortion (RD) optimized interactive streaming method for multiview video pre-compressed by H.264 Joint Multiview Video Model (JMVM). In the proposed method, multiple encodings are first used to facilitate the flexible server–client interaction. Second, a RD-optimized scheduling strategy is provided to guarantee the optimal view-dependent delivery of multiview video. In the RD-optimized scheduling strategy, a distortion model is proposed to estimate the expected end-to-end distortion by accounting for both coding and packet-loss-induced distortions, as well as rendering-induced distortion. With the end-to-end distortion model, the server can select the optimal encoding combination for transmission. Experimental results demonstrate that the proposed method can achieve a significant end-to-end RD performance improvement over the selective streaming methods with simulcast coding or scalable multiview coding. In addition, it has better error-resilience performance to combat with packet-losses over the Internet protocol (IP) networks.

Early SKIP mode decision for MVC using inter-view correlation

In the joint multiview video model (JMVM) proposed by JVT, the variable block-size motion estimation (ME) and disparity estimation (DE) have been employed to determine the best coding mode for each macroblock (MB). These give a high coding efficiency for multiview video coding (MVC), however, they cause a very high computational complexity in encoding system. This paper proposes to reduce the complexities of the ME and DE processes with an early SKIP mode decision algorithm based on the analysis of prediction mode distribution regarding the corresponding MBs in the neighbor view. In this method, the mode decision procedures of most of MBs can be early terminated, and thus much of computation for ME and DE can be greatly reduced. Simulation results demonstrate that our algorithm can achieve computational saving of 46–57% (depending on the tested sequences) with no significant loss of rate-distortion performance.

Color Correction Preprocessing for Multiview Video Coding

In multiview video, a number of cameras capture the same scene from different viewpoints. There can be significant variations in the color of views captured with different cameras, which negatively affects performance when the videos are compressed with inter-view prediction. In this letter, a method is proposed for correcting the color of multiview video sets as a preprocessing step to compression. Unlike previous work, where one of the captured views is used as the color reference, we correct all views to match the average color of the set of views. Block-based disparity estimation is used to find matching points between all views in the video set, and the average color is calculated for these matching points. A least-squares regression is performed for each view to find a function that will make the view most closely match the average color. Experimental results show that when multiview video is compressed with joint multiview video model, the proposed method increases compression efficiency by up to 1.0 dB in luma peak signal-to-noise ratio (PSNR) compared to compressing the original uncorrected video.

다시점 영상 부호화에서 전역 변이 벡터를 이용한 고속 모드 결정

Multi-view video coding (MVC) based on H.264/AVC encodes multiple views efficiently by using a prediction scheme that exploits inter-view correlation among multiple views. However, with the increase of the number of views and use of inter-view prediction among views, total encoding time will be increased in multiview video coding. In this paper, we propose a fast mode decision using both MB(Macroblock)-based region segmentation information corresponding to each view in multiple views and global disparity vector among views in order to reduce encoding time. The proposed method achieves on average 40% reduction of total encoding time with the objective video quality degradation of about 0.04 dB peak signal-to-noise ratio (PSNR) by using joint multi-view video model (JMVM) 4.0 that is the reference software of the multiview video coding standard.

Fast Macroblock Mode Selection Algorithm for Multiview Video Coding

Multiview video coding (MVC) plays an important role in three-dimensional video applications. Joint Video Team developed a joint multiview video model (JMVM) in which full-search algorithm is employed in macroblock mode selection to provide the best rate distortion performance for MVC. However, it results in a considerable increase in encoding complexity. We propose a hybrid fast macroblock mode selection algorithm after analyzing the full-search algorithm of JMVM. For nonanchor frames of the base view, the proposed algorithm halfway stops the macroblock mode search process by designing three dynamic thresholds. When nonanchor frames of the other views are being encoded, the macroblock modes can be predicted from the frames of the neighboring views due to the strong correlations of the macroblock modes. Experimental results show that the proposed hybrid fast macroblock mode selection algorithm promotes the encoding speed by 2.37 ~ 9.97 times without noticeable quality degradation compared with the JMVM.

Optimising inter-view prediction structure for multiview video coding with minimum spanning tree

A new systematic method to construct a near-optimal inter-view prediction structure for multiview video coding is proposed based on the directed minimum spanning tree of inter-view prediction cost graph. The method is applicable to arbitrary irregular and dynamic changing camera arrangements. Results show that the generated structure achieves better compression efficiency than the reference structure of the joint multiview video model.