Side Information Frame Research Articles

Improving distributed video coding with deep learning

In the current state-of-the-art distributed video coding (DVC) solutions, side information (SI) frames are created by motion-compensated interpolation. This is the same technique used by frame rate upconversion operations. Recently, these operations are accomplished using deep learning (DL), with significantly improved outcomes. DL is also used to improve the decoded images of numerous compression methods. Consequently, it is natural to wonder what are the impacts of these enhancements on DVC. In this regard, we employed DL networks to create SI and enhance the quality of the DVC decoded frames. We show how these two operations were applied to DVC and their effects on rate-distortion (RD) performance. The comparison with state-of-the-art DVC systems shows an improvement in the quality of the SI frames and that of the decoded frames, and thus an improvement of the RD performance of DVC.

Adaptive Block-Based Compressed Video Sensing Based on Saliency Detection and Side Information.

The setting of the measurement number for each block is very important for a block-based compressed sensing system. However, in practical applications, we only have the initial measurement results of the original signal on the sampling side instead of the original signal itself, therefore, we cannot directly allocate the appropriate measurement number for each block without the sparsity of the original signal. To solve this problem, we propose an adaptive block-based compressed video sensing scheme based on saliency detection and side information. According to the Johnson–Lindenstrauss lemma, we can use the initial measurement results to perform saliency detection and then obtain the saliency value for each block. Meanwhile, a side information frame which is an estimate of the current frame is generated on the reconstruction side by the proposed probability fusion model, and the significant coefficient proportion of each block is estimated through the side information frame. Both the saliency value and significant coefficient proportion can reflect the sparsity of the block. Finally, these two estimates of block sparsity are fused, so that we can simultaneously use intra-frame and inter-frame correlation for block sparsity estimation. Then the measurement number of each block can be allocated according to the fusion sparsity. Besides, we propose a global recovery model based on weighting, which can reduce the block effect of reconstructed frames. The experimental results show that, compared with existing schemes, the proposed scheme can achieve a significant improvement in peak signal-to-noise ratio (PSNR) at the same sampling rate.

Side information hybrid generation based on improved motion vector field

The quality of side information has an important impact on the performance of a distributed video coding system. At present, the generation of side information is mainly based on the translational motion model using the inter-frame correlation for motion estimation. Considering that the generated side information is prone to block effect and ghosting, as well as the situation that the nonlinear motion is not fully considered and the intra-frame correlation is not fully utilized, a side information hybrid generation algorithm based on an improvement model for the motion vector field is proposed. The side information frame to be generated for the current Wyner-Ziv frame is divided into easy-to-estimate and difficult-to-estimate macroblocks. For difficult-to-estimate macroblocks, the Horn and Schunck dense optical flow method is used to generate reliable motion vectors, for easy-to-estimate macroblocks, the block matching method is used to generate unreliable motion vectors which are modified by the proposed scheme, and then, the improved motion vectors are used for motion compensation to produce the final side information frame. Experiment results show that the quality of side information obtained by using the improved motion vector field for motion compensation has been significantly improved, thus the overall performance of the distributed video coding system has been effectively improved.

Successive refinement of side information frames in distributed video coding

In distributed video coding, generation of high-quality side information is required for reliable soft-input information for decoding each DCT band of a Wyner-Ziv (WZ) frame in the decoder, which in turn leads to a more efficient decoding. Consequently, less error-correcting bits need to be transmitted from the encoder to the decoder to decode the bitplanes of each DCT band, leading to a better compression efficiency and rate-distortion performance. In this paper, we investigate the problem of successive improvements in the quality of side information frames in distributed video coding in order to improve the rate-distortion performance. A new algorithm for the successive refinement of the side information is proposed to refine the initial side information frame using the additional information obtained after decoding the previous DCT bands of a WZ frame. As more information about the WZ frame becomes available after the decoding of each DCT band of the WZ frame, the corresponding side information frame is refined and then employed to decode the next DCT band of the WZ frame. Simulations are carried out demonstrating that the proposed algorithm for refinement of side information frame results in a considerable improvement in the RD performance of distributed video coding.

Decoder side Wyner–Ziv frame estimation using Chebyshev polynomial-based FLANN technique for distributed video coding

In this paper, a Chebyshev polynomial-based functional link artificial neural network (CFLANN) technique for Wyner–Ziv (WZ) frame estimation in a distributed video coding framework is proposed. The estimated WZ frame at the decoder is also referred to as the side information (SI). The proposed scheme (CFLANN-SI) works in two phases, namely, training and testing. The network is trained offline, and to achieve better generalization, the training (input, target) patterns are created across several video sequences constituting varied motion behavior. It estimates the SI frame using adjacent key frames as inputs. The training convergence characteristics of CFLANN-SI is observed to be faster with reduced mean square error as compared to a multi-layer perceptron-based prediction scheme. It is also observed that once the model is trained, it is capable of estimating SI for rest of the incoming WZ frames of the video sequences as well as for the video sequences which are not considered during the learning phase. The proposed scheme is evaluated with respect to different parameters, namely, rate-distortion, peak-signal-to-noise-ratio, the number of parity requests made per estimated frame, decoding time requirement and so on. Comparative analysis shows that the present CFLANN-SI technique generates better SI in resemblance to the competent schemes, in terms of the subjective quality improvement as well as the objective quality gains. Further, to substantiate that the present scheme provides a significant improvement over that of the benchmark techniques, a statistical analysis tool is used with a significance level of 5%.

Multi-resolution extreme learning machine-based side information estimation in distributed video coding

Context: Encoding of video frames in a traditional video coding architecture involves exhaustive computations due to the motion estimation (ME) task. Hence, it requires a considerable amount of computing aid, battery power, and resource memory. These codecs are not effective and reliable for applications like surveillance systems, wireless sensor networks, wireless camcorders, having scarcity in the availability of resources and computing ability. Therefore, in such scenarios, distributed video coding (DVC) represents a viable solution for power-constrained hand-held devices. DVC empowers the adaptability in distributing the complexity between the encoder and the decoder. Objective: Like any other building block, the decoder driven side information (SI) generation module plays a key role in a DVC codec. The efficacy of a DVC codec firmly relies on the quality of the SI generated at the decoder. SI is considered to be the facsimile of the original Wyner-Ziv (WZ) frame. Hence, the superior the quality of SI, improved is the efficiency of the codec. The primary objective of the present work is to enhance the quality of the SI frame so that the overall performance of the DVC is improved. To achieve this objective, this article deals with a hybrid SI generation scheme utilizing the principles of discrete wavelet transform (DWT) and extreme learning machine (ELM) algorithm in a transform domain-based DVC framework. Results: Exhaustive simulations have been carried out for some standard video sequences with the proposed and benchmark schemes. The proposed scheme is evaluated with respect to different performance metrics such as rate-distortion (RD), SI peak-signal-to-noise-ratio (PSNR) vs frame number, number of parity requests per SI frame, and so on. Experimental results and its analyses corroborate that the performance of the proposed technique surpasses as that of the benchmark schemes.

Decoder driven side information generation using ensemble of MLP networks for distributed video coding

This paper proposes an ensemble of multi-layer perceptron (MLP) networks for side information (SI) generation in distributed video coding (DVC). In the proposed scheme, both three-layer and four-layer MLP structures are used to form the ensemble model. The proposed model includes four sub-modules. The first sub-module involves the training of the individual networks. The second sub-module selects ‘M’ number of trained MLPs based on the mean square error (MSE) performance metric. Next, the third sub-module involves the testing phase of each of the selected MLPs. Finally, in the last sub-module, the overall ensemble SI is generated using a dynamically averaging (DA) method. The primary goal of this work is to minimize the estimation error between the SI and the corresponding Wyner-Ziv (WZ) frame so that the overall efficiency of DVC codec can be increased. The proposed scheme is evaluated with respect to different parameters such as Rate-Distortion (RD), Peak Signal to Noise Ratio (PSNR), Structural Similarity Index (SSIM), and number of parity requests made per estimated frame. The evaluation indicates that the proposed ensemble model shows better generalization capabilities with improved PSNR (in dB) as compared to each of the individual selected networks. Additionally, the comparative analysis also exhibits that the proposed SI generation scheme generates better SI frames in comparison with the contemporary techniques. Further, using a statistical test, namely, ANOVA with significance level of 5%, it has been validated that the proposed technique yields a significant enhancement in the performance as compared to that of the benchmark schemes.

Toward enhancing the distributed video coder under a multiview video codec framework

The advance of video coding technology enables multiview video (MVV) or three-dimensional television (3-D TV) display for users with or without glasses. For mobile devices or wireless applications, a distributed video coder (DVC) can be utilized to shift the encoder complexity to decoder under the MVV coding framework, denoted as multiview distributed video coding (MDVC). We proposed to exploit both inter- and intraview video correlations to enhance side information (SI) and improve the MDVC performance: (1) based on the multiview motion estimation (MVME) framework, a categorized block matching prediction with fidelity weights (COMPETE) was proposed to yield a high quality SI frame for better DVC reconstructed images. (2) The block transform coefficient properties, i.e., DCs and ACs, were exploited to design the priority rate control for the turbo code, such that the DVC decoding can be carried out with fewest parity bits. In comparison, the proposed COMPETE method demonstrated lower time complexity, while presenting better reconstructed video quality. Simulations show that the proposed COMPETE can reduce the time complexity of MVME to 1.29 to 2.56 times smaller, as compared to previous hybrid MVME methods, while the image peak signal to noise ratios (PSNRs) of a decoded video can be improved 0.2 to 3.5 dB, as compared to H.264/AVC intracoding.

Perception-based adaptive quantization for transform-domain Wyner-Ziv video coding

Distributed video coding (DVC) is desirable for encoding systems with tight power or computational constraints, for which the popular practical solution is transform-domain Wyner-Ziv video coding (TD-WZVC). To achieve the similar coding performance with H.264/AVC, quantization is a key factor in TD-WZVC. Practically, the quantization matrix is trained offline and remains fixed value during coding. Optimal rate-distortion (RD) performance cannot be achieved due to the varying quality of side information (SI) frame. In this paper, a novel model of perceptual distortion probability is developed to estimate the perceptual distortion of SI frame and to derive the target perceptual distortion. With the two perceptual distortion probabilities, three components (i.e. quality of SI frame, perceptual features and RD optimization) are integrated to determine the optimal quantization matrix adaptively, which improves the coding performance. Extensive experiments demonstrate that the proposed scheme can adaptively determine proper quantization matrix online and achieve similar visual quality with less bit-rate, as compared to other adaptive quantization schemes in TD-WZVC.

Merge Frame Design for Video Stream Switching Using Piecewise Constant Functions.

The ability to efficiently switch from one pre-encoded video stream to another (e.g., for bitrate adaptation or view switching) is important for many interactive streaming applications. Recently, stream-switching mechanisms based on distributed source coding (DSC) have been proposed. In order to reduce the overall transmission rate, these approaches provide a merge mechanism, where information is sent to the decoder, such that the exact same frame can be reconstructed given that any one of a known set of side information (SI) frames is available at the decoder (e.g., each SI frame may correspond to a different stream from which we are switching). However, the use of bit-plane coding and channel coding in many DSC approaches leads to complex coding and decoding. In this paper, we propose an alternative approach for merging multiple SI frames, using a piecewise constant (PWC) function as the merge operator. In our approach, for each block to be reconstructed, a series of parameters of these PWC merge functions are transmitted in order to guarantee identical reconstruction given the known SI blocks. We consider two different scenarios. In the first case, a target frame is first given, and then merge parameters are chosen, so that this frame can be reconstructed exactly at the decoder. In contrast, in the second scenario, the reconstructed frame and the merge parameters are jointly optimized to meet a rate-distortion criteria. Experiments show that for both scenarios, our proposed merge techniques can outperform both a recent approach based on DSC and the SP-frame approach in H.264, in terms of compression efficiency and decoder complexity.

Distributed video coding supporting hierarchical GOP structures with transmitted motion vectors

Abstract In this paper, we propose a new distributed video coding (DVC) method, with hierarchical group of picture (GOP) structure. Coding gain of DVC can be significantly improved by enlarging GOP size for slow-moving frames. The proposed DVC decoder estimates a side information (SI) frame and transmits motion vectors (MVs) of the SI to the proposed encoder. Using the received MVs from the decoder, the proposed encoder can generate a predicted SI (PSI), which is the same as the SI in the decoder, and estimate the quality of PSI with minimal computational complexity. The proposed method decides the best coding mode among key, Wyner-Ziv (WZ), and skip modes, by estimating rate-distortion costs. Based on the selected best coding mode, the best GOP size can be automatically determined. As the GOP size is adaptively decided depending on the SI quality, entropy and parity bits can be effectively consumed. Experimental results show that the proposed algorithm is around 0.80 dB better in Bjøntegaard delta (BD) bitrate than an existing conventional DVC system.

Re-estimation of Motion and Reconstruction for Distributed Video Coding

Transform domain Wyner-Ziv (TDWZ) video coding is an efficient approach to distributed video coding (DVC), which provides low complexity encoding by exploiting the source statistics at the decoder side. The DVC coding efficiency depends mainly on side information and noise modeling. This paper proposes a motion re-estimation technique based on optical flow to improve side information and noise residual frames by taking partially decoded information into account. To improve noise modeling, a noise residual motion re-estimation technique is proposed. Residual motion compensation with motion updating is used to estimate a current residue based on previously decoded frames and correlation between estimated side information frames. In addition, a generalized reconstruction algorithm to optimize a multihypothesis reconstruction is proposed. The proposed techniques using motion and reconstruction re-estimation (MORE) are integrated in the SING TDWZ codec, which uses side information and noise learning. For Wyner-Ziv frames using GOP size 2, the MORE codec significantly improves the TDWZ coding efficiency with an average (Bjøntegaard) PSNR improvement of 2.5 dB and up to 6 dB improvement compared with DISCOVER.

Block boundary matching algorithm for generating side information in distributed video codec

Distributed video codec (DVC) has been developed to construct a simple encoder that utilizes information theory for distributed sources in the circumstance of mobile multimedia communication. In the DVC codec, an efficient algorithm to generate side information (SI) is one of the most important techniques to improve the coding performance. We propose a scheme to increase the quality of SI frame, where the proposed scheme consists of three steps. In the first step, SI frame is constructed by motion estimation and motion compensation in the DVC decoder. Then, in the second step, the blocks in the temporary SI frame are classified into reli- able or unreliable ones. The unreliable blocks are updated by block boun- dary matching algorithm in the third step. Simulation results show that the proposed algorithm outperforms the conventional methods significantly. In addition, the proposed scheme can be combined with the conventional schemes generating SI frame to increase the coding performance of the DVC codec. © The Authors. Published by SPIE under a Creative Commons Attribution 3.0 Unported License. Distribution or reproduction of this work in whole or in part requires full attribution of the original publication, including its DOI. (DOI: 10.1117/1.OE.52 .10.103111)

Side Information Refinement for Transform domain Distributed Video Coding

A new side information refinement method for transform domain distributed video coding (DVC) is proposed. In DVC, side information is the frame constructed from key frames which are intra coded. It is utilized as the initial estimate of source symbols of Wyner-Ziv coded frame which is not transmitted to the decoder. Errors in the estimated frames are to be corrected by the decoder using the parity bits which are the only information received from the encoder. Quality of decoder generated side information is one of the most important factors deciding the overall performance of the decoder. A new refinement method of improving the quality of side information of the DVC decoder is proposed. The side information frame is refined at the end of each transform band decoding stage followed by a conventional bi-directional motion estimation and compensation technique. Proposed side information refinement technique consequently improves the quality of final reconstructed pictures.

An improved side information generation for distributed video coding

This paper presents a side information (SI) scheme for distributed video coding based on multilayer perceptron. The suggested scheme predicts a Wyner–Ziv (WZ) frame from two decoded key frames. The network is trained offline using patterns from different standard video sequences with varied motion characteristics to achieve generalization. The proposed scheme is simulated along with other standard video coding schemes. Performance comparisons have been made with respect to training convergence, rate distortion (RD), peak signal to noise ratio (PSNR), number of requests per SI frame, decoding time requirement, etc. In general, it is observed that the proposed scheme has a superior SI frame generation capability as compared to its competent schemes.

Distributed Video Coding Based on Multiple-source Correlation Model

Distributed Video Coding (DVC) is a new video coding paradigm which has the ability to shift coding complexity from encoder to decoder. State-of-art DVC schemes are mainly based on two-source correlation model and only one correlation channel between side information and wyner-ziv frame is considered. In this paper, multiple-source correlation model is introduced into a famous DVC scheme, namely Transform Domain Wyner-Ziv video coder (TDWZ). A soft-input computing algorithm based on multiple-source correlation model is firstly proposed. The correlations a mong multiple side information are separately exploited. Then a new reconstruction function based on multiple source correlation model is also proposed to improve reconstructed image's quality. Experimental results show that the proposed algorithms can effectively improve the coding performance of systems.

CDV-DVC: Transform-domain distributed video coding with multiple channel division

This paper proposes an efficient distributed coding system based on multiple channel division. We develop the hierarchical motion refinement scheme using DC syndrome bits to generate high quality side information (SI) for Wyner–Ziv (WZ) frames. Moreover, we estimate local distortion characteristics of an SI frame and encode the SI frame in three coding modes: skip mode for the reliable channel, WZ mode for the medium channel, and intra mode for the unreliable channel. No bit is transmitted in the skip mode. Syndrome bits in the WZ mode are adaptively allocated based on the local distortion characteristics. The H.264 intra coding is performed in the intra mode to recover severely erroneous blocks. Experimental results show that the proposed algorithm provides significantly better rate-distortion performance than the state-of-the-art DISCOVER codec.

Side Information and Noise Learning for Distributed Video Coding Using Optical Flow and Clustering

Distributed video coding (DVC) is a coding paradigm that exploits the source statistics at the decoder side to reduce the complexity at the encoder. The coding efficiency of DVC critically depends on the quality of side information generation and accuracy of noise modeling. This paper considers transform domain Wyner-Ziv (TDWZ) coding and proposes using optical flow to improve side information generation and clustering to improve the noise modeling. The optical flow technique is exploited at the decoder side to compensate for weaknesses of block-based methods, when using motion-compensation to generate side information frames. Clustering is introduced to capture cross band correlation and increase local adaptivity in the noise modeling. This paper also proposes techniques to learn from previously decoded WZ frames. Different techniques are combined by calculating a number of candidate soft side information for low density parity check accumulate decoding. The proposed decoder side techniques for side information and noise learning (SING) are integrated in a TDWZ scheme. On test sequences, the proposed SING codec robustly improves the coding efficiency of TDWZ DVC. For WZ frames using a GOP size of 2, up to 4-dB improvement or an average (Bjøntegaard) bit-rate savings of 37% is achieved compared with DISCOVER.

Transform-Domain Adaptive Correlation Estimation (TRACE) for Wyner–Ziv Video Coding

Wyner-Ziv video coding (WZVC) is a newly emerged video coding scheme which compresses the input video frames with the side information (SI) frames only available at the decoder. WZVC exploits the statistics between the source frame and the SI frame at the decoder by utilizing their correlation information. This correlation information is important but also difficult to estimate due to the absence of the SI frame at the encoder, and the lack of the source frame at the decoder. In this paper, we focus on this problem and propose a novel transform-domain adaptive correlation estimation method called TRACE for WZVC. In TRACE, the correlation information is progressively learned during the decoding process of each frame. Within TRACE, we also propose a convex optimization based band-level correlation estimation method which is optimal in the sense of minimizing the theoretical bit rate. Experiments suggest that, when applied in motion compensated interpolation-based low complexity WZVC, TRACE yields competitive results against the state-of-the-art correlation estimation algorithms. More importantly, different from the existing coefficient-level correlation estimation algorithms, the proposed TRACE can be applied in many other WZVC schemes and can provide considerable gain over the popular band-level correlation estimation methods.

Iterative Side-Information Generation in a Mixed Resolution Wyner-Ziv Framework

We propose a mixed resolution framework based on full resolution key frames and spatial-reduction-based Wyner-Ziv coding of intermediate nonreference frames. Improved rate-distortion performance is achieved by enabling better side-information generation at the decoder side and better rate-allocation at the encoder side. The framework enables reduced encoding complexity by low resolution encoding of the nonreference frames, followed by Wyner-Ziv coding of the Laplacian residue. The quantized transform coefficients of the residual frame are mapped to cosets without the use of a feedback channel. A study to select optimal coding parameters in the creation of the memoryless cosets is made. Furthermore, a correlation estimation mechanism that guides the parameter choice process is proposed. The decoder first decodes the low resolution base layer and then generates a super-resolved side-information frame at full resolution using past and future key frames. Coset decoding is carried using side-information to obtain a higher quality version of the decoded frame. Implementation results are presented for the H.264/AVC codec.