Multi-view Distributed Video Coding Research Articles

Temporal–Spatial Symmetric Distributed Multi-View Video Coding Scheme

To improve the rate stability and make a balance for different viewpoints in distributed multi-view video coding (DMVC) system, a novel symmetric DMVC (SDMVC) scheme is proposed in this paper. In the proposed scheme, every frame from all views adopts the same encoding mode and stable output rates are achieved, which are significant to improve the transmission efficiency in the channel. Both temporal and spatial correlations are exploited, in addition, a novel side information (SI) generation algorithm aiming at better exploring the correlations of proposed scheme has been proposed to obtain better performance. The simulation results show that the proposed SDMVC scheme gets a much more stable rate than the asymmetric scheme, only with neglectable bit-rate increasing. Meanwhile, the proposed SI generation algorithm significantly improves the coding performance.

Toward enhancing the distributed video coder under a multiview video codec framework

The advance of video coding technology enables multiview video (MVV) or three-dimensional television (3-D TV) display for users with or without glasses. For mobile devices or wireless applications, a distributed video coder (DVC) can be utilized to shift the encoder complexity to decoder under the MVV coding framework, denoted as multiview distributed video coding (MDVC). We proposed to exploit both inter- and intraview video correlations to enhance side information (SI) and improve the MDVC performance: (1) based on the multiview motion estimation (MVME) framework, a categorized block matching prediction with fidelity weights (COMPETE) was proposed to yield a high quality SI frame for better DVC reconstructed images. (2) The block transform coefficient properties, i.e., DCs and ACs, were exploited to design the priority rate control for the turbo code, such that the DVC decoding can be carried out with fewest parity bits. In comparison, the proposed COMPETE method demonstrated lower time complexity, while presenting better reconstructed video quality. Simulations show that the proposed COMPETE can reduce the time complexity of MVME to 1.29 to 2.56 times smaller, as compared to previous hybrid MVME methods, while the image peak signal to noise ratios (PSNRs) of a decoded video can be improved 0.2 to 3.5 dB, as compared to H.264/AVC intracoding.

SIFT-Based Homographies for Efficient Multiview Distributed Visual Sensing

Considering the scarce resources in visual sensor networks, video coding solutions are necessary to address these constraints, namely the limited energy, memory, and bandwidth. The distributed video coding (DVC) paradigm is able to address these limitations by shifting most of the complexity to the decoder, typically a central location with a significant amount of resources. In a DVC context, the video codec rate-distortion (RD) performance is strongly dependent on the efficient generation of the so-called side information (SI). While in monoview DVC solutions the SI is generated by exploiting only the temporal correlation between frames, in multiview DVC (MV-DVC) solutions, the SI may be estimated by exploiting both the temporal and interview correlations. In this paper, a novel MV-DVC solution is proposed to improve the state-of-the-art MV-DVC RD performance with two key contributions in terms of an interview SI creation and SI fusion. According to the obtained experimental results, the proposed solution improves the RD performance associated to the current MV-DVC state-of-the-art solutions, exhibiting bitrate savings $\sim 10$ % for the same output video quality, without increasing the encoding complexity and eventually also the decoding complexity.

A robust fusion method for multiview distributed video coding

Abstract Distributed video coding (DVC) is a coding paradigm which exploits the redundancy of the source (video) at the decoder side, as opposed to predictive coding, where the encoder leverages the redundancy. To exploit the correlation between views, multiview predictive video codecs require the encoder to have the various views available simultaneously. However, in multiview DVC (M-DVC), the decoder can still exploit the redundancy between views, avoiding the need for inter-camera communication. The key element of every DVC decoder is the side information (SI), which can be generated by leveraging intra-view or inter-view redundancy for multiview video data. In this paper, a novel learning-based fusion technique is proposed, which is able to robustly fuse an inter-view SI and an intra-view (temporal) SI. An inter-view SI generation method capable of identifying occluded areas is proposed and is coupled with a robust fusion system able to improve the quality of the fused SI along the decoding process through a learning process using already decoded data. We shall here take the approach to fuse the estimated distributions of the SIs as opposed to a conventional fusion algorithm based on the fusion of pixel values. The proposed solution is able to achieve gains up to 0.9 dB in Bjøntegaard difference when compared with the best-performing (in a RD sense) single SI DVC decoder, chosen as the best of an inter-view and a temporal SI-based decoder one.

Depth-Based Multiview Distributed Video Coding

Multiview distributed video coding (DVC) has gained much attention in the last few years because of its potential in avoiding communication between cameras without decreasing the coding performance. However, the current results are not matching the expectations mainly due to the fact that some theoretical assumptions are not satisfied in the current implementations. For example, in distributed source coding the encoder must know the correlation between the sources, which cannot be achieved in the traditional DVC systems without having a communication between the cameras. In this work, we propose a novel multiview distributed video coding scheme in which the depth maps are used to estimate the way two views are correlated with no exchanges between the cameras. Only their relative positions are known. We design the complete scheme and further propose a rate allocation algorithm to efficiently share the bit budget between the different components of our scheme. Then, a rate allocation algorithm for depth maps is proposed in order to maximize the quality of synthesized virtual views. We show, through detailed experiments, that our scheme significantly outperforms the state-of-the-art DVC system.

Multi-View Distributed Video Coding Based on Discrete Cosine

To investigate the allocation scheme of the multi-view distributed video coding (DVC), the corresponding improvements are proposed correspondingly for traditional multi-view DVC. Traditional multi-view DVC (Wyner-Ziv DVC) encodes for all areas of Wyner-Ziv frame indiscriminately based on Turbo or LDPC. In this kind of encoding process, with regard to violent motor area, decoder can't decode violent motor area accurately and also send more solicited message to feedback channel, which lowers the code efficiency and decodes inaccurately for violent motor area, it causes a part of area distortion in the image. In this paper, a distributed video encryption algorithm is proposed which based on discrete cosine transform (DCT). The algorithm combines decision criteria of ROI to get violent motor area and non-violent motor area. For violent motor area, to extract low frequency coefficient of DCT as DCT-R algorithm to assist decoder end to decode, decoder utilizes low frequency coefficient of DCT which has already been decoded to carry on bi-directional movement evaluation. Simulation experiment tests and verifies the improved algorithm effectiveness of proposed multi-view DVC in this paper.

시점 간 비선형 움직임 블록 예측에 기초한 적응적 다시점 비디오 보상 보간 기법

최근에 무선 비디오 감사, 무선 비디오 센서 네트워크 그리고 무선 모바일 비디오와 같은 다시점 비디오 서비스에 대한 연구가 활발히 진행되고 있다. 다시점 비디오 신호처리에서 다수 개의 다른 카메라에서 획득되는 영상 사이에 존재하는 높은 상관성을 이용하는 것은 매우 핵심적인 기술이다. 본 논문에서는 카메라들 사이에 상호작용을 요구하지 않고, 다시점 분산 비디오 부호화에 효과적으로 사용할 수 있는 적응적인 다시점 보간 기법을 제안한다. 제안한 방법은 비선형적인 블록 예측, 시차 보상 시점 예측 그리고 비신뢰 블록에 대한 채우기 기법 등으로 구성된다. 모의실험을 통하여 제안한 방식은 기존의 방식에 비해 우수한 성능을 보인다. Recently, many researches have been focused on multi-view video applications and services such as wireless video surveillance networks, wireless video sensor networks and wireless mobile video. In multi-view video signal processing, to exploit the strong correlation between images acquired by different cameras plays great role in developing a core technique of multi-view video coding. This paper proposes an adaptive multi-view video interpolation technique which is applicable for multi-view distributed video coding without requiring any cooperation amongst the cameras. The proposed algorithm estimates the non-linear moving blocks and employs disparity compensated view prediction, and then fills in the unreliable blocks. Through computer simulations, it is shown that the proposed method outperforms the conventional methods.

Motion-Aware Mesh-Structured Trellis for Correlation Modelling Aided Distributed Multi-View Video Coding

A joint source-channel coding has attracted substantial attention with the aim of further exploiting the residual correlation residing in the encoded video signals for the sake of improving the reconstructed video quality. In our previous paper, a first-order Markov process model was utilized as an error concealment tool for exploiting the intra-frame correlation residing in the Wyner-Ziv (WZ) frame in the context of pixel-domain distributed video coding. In this contribution, we exploit the interview correlation with the aid of an interview motion search in distributed multi-view video coding (DMVC). Initially, we rely on the system architecture of WZ coding invoked for multiview video. Then, we construct a novel mesh-structured pixel-correlation model from the inter-view motion vectors and derive its decoding rules for joint source-channel decoding. Finally, we benchmark the attainable system performance against the existing pixel-domain WZ coding based DMVC scheme, where the classic turbo codec is employed. Our simulation results show that substantial bitrate reductions are achieved by employing the proposed motion-aware mesh-structured correlation modelling technique in a DMVC scheme.

A Coding and Postprocessing Framework of Multiview Distributed Video for Wireless Video Sensor Networks

In many surveillance application scenarios of wireless video sensor networks (WVSNs), a number of video sensors are deployed, and multidimension monitored the visual information in a region of interest, forming multiview videos. Since the power, computing capability, and bandwidth are very limited in WVSNs, the conventional multiview video coding method is no longer applicable. So multiview distributed video coding (MDVC) emerged and developed rapidly. In this paper, we propose a new multiview video coding and postprocessing framework for multiview videos. First, in coding scheme, motion intense regions (MIRs) and nonmotion intense regions (NMIRs) based on sum of absolute difference (SAD) criteria are distinguished. For the MIR, the side information (SI) is generated by fusion temporal SI and interview spatial SI at the pixel level. But for the NMIR, the temporal SI is directly use as the ultimate SI. Then, to further improve the quality of the decoded image, an image postprocessing scheme is designed by using deblocking and deringing artifact filters on decoded image. Finally, a set of experimental results show that the proposed fusion SI approach can bring improvements up to 0.2–0.5 dB when compares with only temporal SI used. The subsequent decoded videos postprocessing simulation proves that the proposed postprocessing scheme can provide an additional improvement of about 0.1 dB to the decoded video sequences.

Sprite-based generation of side information for multi-view Distributed Video Coding

Generation of side information for multi-view Distributed Video Coding in multi-camera environments (e.g., video surveillance) poses challenges in scenarios with temporary non-overlapping among views and consequently, no resources for generating the side information at some time instants. In this paper we extend our previous work (Ciobanu and Corte-Real, Multimed Tools Appl 48(3):411---436, 2010) (for scenarios with permanent complete-overlapping among views) and propose a solution to this problem by exploiting the past visual data associated with each view, gathered over time as a panoramic image (sprite). The entire collection of temporal data from all the cameras is subsequently used for generating the side information. We tackle several topics related to these scenarios and propose solutions for the encountered issues. Optimization techniques are also discussed, e.g., temporal tags and block alternatives associated with the sprite contents for an improved generation of side information. This paper also presents a post-processing technique for additional refinement of generated side information. Practical results show an overall significant enhancement of side information by over 2 dB.

Joint Coding/Routing Optimization for Distributed Video Sources in Wireless Visual Sensor Networks

This paper studies a joint coding/routing optimization between network lifetime and video distortion by applying information theory to wireless visual sensor networks for correlated sources. Arbitrary coding [distributed video coding and network coding (NC)] from both combinatorial optimization and information theory could make significant progress toward the performance limit and tractable. Also, multipath routing can spread energy utilization across nodes within the entire network to keep a potentially longer lifetime, and solve the wireless contention issues by the splitting traffic. The objective function not only keeps the total energy consumption of encoding power, transmission power, and reception power minimized, but ensures the information received by sink nodes to approximately reconstruct the visual field. Also, a generalized power consumption model for distributed video sources is developed, in which the coding complexity of Key frames and Wyner-Ziv frames is measured by translating specific coding behavior into energy consumption. On the basis of the distributed multiview video coding and NC-based multipath routing, the balance problem between lifetime (costs) and distortion (capacity) is modeled as an optimization formulation with a fully distributed solution. Through a primal decomposition, a two-level optimization is relaxed with Lagrangian dualization and solved by the gradient algorithm. The low-level optimization problem is further decomposed into a secondary master dual problem with four cross-layer subproblems: a rate control problem, a channel contention problem, a distortion control problem, and an energy conservation problem. The implementation of the distributed algorithm is discussed with regard to the communication overhead and dynamic network change. Simulation results validate the convergence and performance of the proposed algorithm.

Distributed multiview video coding with 3D-DCT transform domain Wyner-Ziv codec

The need for efficient multiview video coding schemes is expected to strongly increase in the near future. The distributed multiview video coding (DMVC) approach seems very promising since it can achieve good compression efficiency while keeping the complexity low. The main contribution of this paper is to investigate how to improve the classic DMVC framework based on transform domain WZ video coding (TDWZ) by means of the introduction of the 3D-DCT. The main advantage of this combination resides in the limited computational complexity of the overall framework, which however does not penalise the compression performance since both the spatial and the temporal domain correlation can be exploited due to the use of the 3D-DCT. The framework is designed in a flexible way so that it can handle both traditional and residual-frame-based WZ coding. The simulation results confirm the validity of the proposed framework in terms of video quality improvements, with gains up to 4.4 dB PSNR compared to a pixel domain WZ technique, and up to 0.6 dB PSNR compared to a 2D-DCT-based one.

Recent Advances in Multiview Video Coding

In the last years more and more research efforts have been strengthened on the Multiview Video Coding (MVC). From the point of coding complexity balance, MVC can be divided into two categories: the first is used for the applications with complex encoder and simple decoder, and the second is used for the scenarios demanding simple encoder but allowing complex decoder. In general, the former is referred as “MVC” directly and most existing algorithms and patents are related to it, in contrast the latter is called as “DMVC” (distributed MVC). This paper focuses on the first type of MVC schemes, which can also be divided into two types: H.264/AVC-based and wavelet-based schemes. After briefly describing the prediction structures, various coding techniques are introduced by dividing them coarsely into five fields: prediction mode selection, illumination compensation/color correction, view interpolation/synthesis prediction, reference frame management and asymmetric decoding. In addition, distributed MVC are briefly introduced. Finally, some future directions are concluded. Keywords: Multiview video coding, 4D wavelet multiview video coding, distributed multiview video coding.

Iterative filtering of SIFT keypoint matches for multi-view registration in Distributed Video Coding

Multi-view registration is an essential step in order to generate the side information for multi-view Distributed Video Coding. As stated in our previous work (Ciobanu and Corte-Real, Multimed Tools Appl 48(3):411---436, 2010) it can be achieved by SIFT (scale-invariant feature transform) generated keypoint matches. The registration accuracy is vital for the adequate generation of side information and it directly depends on the reliable match of possibly all the available point to point correlations between two complete-overlapped views. We propose a solution to this problem based on iterative filtering of SIFT-generated keypoint matches, using the Hough transform and block matching. It aims the generic, real-life and constraint-free scenarios having an arbitrarily close angle between the two views. Practical results show an overall significant reduction of the outliers while maintaining a high rate of correct matches.

Subgraphs Matching-Based Side Information Generation for Distributed Multiview Video Coding

We adopt constrained relaxation for distributed multiview video coding (DMVC). The novel framework integrates the graph-based segmentation and matching to generate interview correlated side information without knowing the camera parameters, inspired by subgraph semantics and sparse decomposition of high-dimensional scale invariant feature data. The sparse data as a good hypothesis space aim for a best matching optimization of interview side information with compact syndromes, from inferred relaxed coset. The plausible filling-in from a priori feature constraints between neighboring views could reinforce a promising compensation to interview side-information generation for joint multiview decoding. The graph-based representations of multiview images are adopted as constrained relaxation, which assists the interview correlation matching for subgraph semantics of the original Wyner-Ziv image by the graph-based image segmentation and the associated scale invariant feature detector MSER (maximally stable extremal regions) and descriptor SIFT (scale-invariant feature transform). In order to find a distinctive feature matching with a more stable approximation, linear (PCA-SIFT) and nonlinear projections (Locally linear embedding) are adopted to reduce the dimension SIFT descriptors, and TPS (thin plate spline) warping model is to catch a more accurate interview motion model. The experimental results validate the high-estimation precision and the rate-distortion improvements.

Successive refinement of side information for multi-view distributed video coding

Inter-camera registration in multi-view systems with overlapped views has a particularly long and sophisticated research history within the computer vision community. Moreover, when applied to Distributed Video Coding, in systems with at least one moving camera it represents a real challenge due to the necessary data at decoder for generating the side information without any a priori knowledge of each instant camera position. This paper proposes a solution to this problem based on successive multi-view registration and motion compensated extrapolation for on-the-fly re-correlation of two views at decoder. This novel technique for side information generation is codec-independent, robust and flexible with regard to any free motion of the cameras. Furthermore, it doesn’t require any additional information from encoders nor communication between cameras or offline training stage. We also propose a metric for an objective assessment of the multi-view correlation performance.

Side information estimation and new symmetric schemes for multi-view distributed video coding

This paper deals with distributed video coding (DVC) for multi-view sequences. DVC of multi-view sequences is a recent field of research, with huge potential impact in applications such as videosurveillance, real-time event streaming from multiple cameras, and, in general, immersive communications. It raises however several problems, and in this paper we tackle two of them. Based on the principles of Wyner–Ziv (WZ) coding, in multi-view DVC many estimations can be generated in order to create the side information (SI) at the decoder. It has been shown that the quality of the SI strongly influences the global coding performances. Therefore, this paper proposes to study the contribution of multiple SI estimations (in the temporal and view directions) to the global performances. Moreover, we propose new symmetric schemes for longer group of pictures (GOP) in multi-view DVC and show that we can further exploit the long-term correlations using a new kind of estimation, called diagonal. For such schemes, several decoding strategies may be envisaged. We perform a theoretical study of the temporal and inter-view dependencies, and confirm by experiments the conclusion about the best decoding strategy.