Inter-frame Correlation Research Articles

A region-based intra-frame rate control scheme by jointing inter-frame dependency and inter-frame correlation

During I frame switching, the subjective quality between I frame and P frames usually have obvious fluctuation due to different coding methods. The periodic temporal visual fluctuation will cause video flicker. According to extensive experiments, we observe that I frame flicker possess a strong regional characteristics and different region have different degree of flicker. Based on this observation, a region-based I frame rate control scheme is proposed to suppress I frame flicker according to the different characteristics of the moving and non-moving regions. Firstly, by jointly considering the inter-frame dependency between I frame and subsequent un-encoded P frames and the inter-frame correlation between I frame and previous encoded P frame, an optimization model is proposed to achieve the optimal QPs for different regions. Secondly, a region-based inter-frame dependency model is proposed to separately describe the inter-frame dependency of different regions, which can accurately describe their description of the inter-frame dependency. The experimental results demonstrate that the proposed scheme can efficiently suppress I frame flicker and maintain the smoothness of subjective quality. Moreover, the proposed scheme can achieve a PSNR gain by 0.26 dB on average when compared with the rate control scheme adopted by the HEVC reference software HM15.0.

Intra frame compression and video restoration based on conditional markov processes theory

Digital images and video have significant statistical redundancy due to high intra and inter-frame correlation. Modeling images and video using two-dimensional Markov processes with discrete states maximizes the use of images and video redundancy while processing. Intra-frame compression algorithm proposed below allows 2---3 times increase in processing speed for comparable compression ratios. Nonlinear filtering algorithm of video (distorted during transmission on a radio channel in terms of interference) allows effective video restoring even for negative signal/noise ratio unlike the known methods. A distinctive feature of this algorithm is ease of implementation with minimal computing resources.

Compressive Parameter Estimation for Correlated Frames in MIMO Visible Light Communications

Estimation of the image location of a mobile LED array transmitter on the camera sensor plane of a digital camera is considered. A Bayesian framework is used to design an initial large measurement matrix over multiple image frames to exploit inter-frame temporal correlations. This large matrix is then factorized into two component matrices; one is used as the front-end measurement matrix with the potential for hardware implementation while the other is designed for software implementation. The factorization can be performed offline through fixed point iterations. The proposed measurement matrix design directly exploits possible transmitter mobility through a parametric framework. The findings are validated through analysis, computer simulation and hardware experiments. Sub-pixel localization accuracy with significant compression is achieved, and the proposed compressive parameter estimation outperforms results obtained using a Gaussian measurement matrix.

Object tracking based on local dynamic sparse model

Sparse representation has been widely applied in many objecting tracking methods. In this paper, we present a robust and effective object tracking approach based on the local dynamic sparse model, called Local Dynamic Sparse Tracking (LDST). In the proposed method, the local patches of a tracked object are linearly represented by their respective dictionary updated online, and the inter-frame correlation between sparse representations of corresponding patches are modeled in the time domain. To further improve its robustness, the dependency of sparse coefficients between patches in each frame is also characterized by the ℓ1,2 mixed norms. In addition, for each patch, different weights are exploited in calculating the likelihood probability, in order to eliminate the effect of occluded patches when updating templates. The evaluation experiments on the challenging sequences demonstrate that the proposed method has the better performance compared with some typical state-of-the-art methods.

Analysis of objective functions in the problem of estimating mutual geometric deformations of images

To estimate mutual geometric deformations of images, a comparative analysis is carried out of the efficiency of using the mean square interframe difference (MSID), the interframe correlation coefficient (ICC), and mutual information (MI) as an objective function. A comparative, largely experimental, analysis is carried out of these similarity measures for various brightness distortions of images, and the convergence of estimates of deformation parameters under the application of objective functions for stochastic pseudogradient procedures is investigated.

Estimation of measurements for block‐based compressed video sensing: study of correlation noise in measurement domain

Compressed video sensing (CVS) is an application of compressed sensing theory which samples a signal below the Shannon–Nyquist rate. However, previous research about CVS has largely ignored the inter-frame correlation analysis in the measurement domain, and then is not able to remove the time redundancy. In this study, the authors consider the estimation of the measurements of a block in any possible position in a frame by introducing a correlation noise (CN) between the actual and the estimated measurements. In this work, they first establish a correlation model (CM) in the pixel domain between a block which is in a random unknown position in a frame and the adjacent non-overlapping blocks that they already have. Then, a novel measurement domain CM is presented to approximate the measurements for the random block. Lastly, they employ the CN to characterise the accuracy of the CM in the measurement domain. The simulation results show that the proposed model can make an accurate estimation to the actual measurements of an arbitrary block in a frame and that by using the proposed CN to perform motion estimation, they can improve the peak signal-to-noise ratio of the video sequences by 0.1–1.7 dB compared with the existing methods.

Estimation of Subband Speech Correlations for Noise Reduction via MVDR Processing

Recently, it has been proposed to use the minimum-variance distortionless-response (MVDR) approach in single-channel speech enhancement in the short-time frequency domain. By applying optimal FIR filters to each subband signal, these filters reduce additive noise components with less speech distortion compared to conventional approaches. An important ingredient to these filters is the temporal correlation of the speech signals. We derive algorithms to provide a blind estimation of this quantity based on a maximum-likelihood and maximum a-posteriori estimation. To derive proper models for the inter-frame correlation of the speech and noise signals, we investigate their statistics on a large dataset. If the speech correlation is properly estimated, the previously derived subband filters discussed in this work show significantly less speech distortion compared to conventional noise reduction algorithms. Therefore, the focus of the experimental parts of this work lies on the quality and intelligibility of the processed signals. To evaluate the performance of the subband filters in combination with the clean speech inter-frame correlation estimators, we predict the speech quality and intelligibility by objective measures.

GOP Adaptation Coding of H.264/SVC Based on Precise Positions of Video Cuts

Hierarchical B-frame coding was introduced into H.264/SVC to provide temporal scalability and improve coding performance. A content analysis-based adaptive group of picture structure (AGS) can further improve the coding efficiency, but damages the inter-frame correlation and temporal scalability of hierarchical B-frame to different degrees. In this paper, we propose a group of pictures (GOP) adaptation coding method based on the positions of video cuts. First, the cut positions are accurately detected by the combination of motion coherence (MC) and mutual information (MI); then the GOP is adaptively and proportionately set by the analysis of MC in one scene. In addition, we propose a binary tree algorithm to achieve the temporal scalability of any size of GOP. The results for test sequences and real videos show that the proposed method reduces the bit rate by up to about 15%, achieves a performance gain of about 0.28?1.67 dB over a fixed GOP, and has the advantages of better transmission resilience and video summaries.

No reference quality assessment for MPEG video delivery over IP

Video delivering over Internet protocol (IP)-based communication networks is widely used in the actual information sharing scenario. As well known, the best-effort Internet architecture cannot guarantee an errorless data delivering. In this paper, an objective no-reference video quality metric for assessing the impact of the degradations introduced by video transmission over heterogeneous IP networks is presented. The proposed approach is based on the analysis of the inter-frame correlation measured at the output of the rendering application. It does not require any information on errors, delays, and latencies affecting the links and on the countermeasures introduced by decoders in order to face the potential quality loss. Experimental results show the effectiveness of the proposed algorithm in approximating the assessments obtained by using human visual system (HVS)-inspired full reference metrics.

Reversible video watermarking using motion-compensated frame interpolation error expansion

In this study, a reversible video watermarking method based on motion-compensated frame interpolation error expansion is developed. Interframe correlation is exploited more efficiently as a result of using motion-compensated frame interpolation error instead of motion-compensated prediction error that is used in the current reversible video watermarking methods. For this reason, on the one hand the proposed method allows high-capacity data insertion to video; on the other hand, it causes small distortion in the original video. In the proposed method, the original video and watermark can be obtained reversibly from the watermarked video, and the amount of side information in the watermarked video required for watermark decoding and video restoration is extremely low. The method is shown to be superior to the existing methods in terms of capacity and visual video quality by performing computer simulations carried out for various widely used test video sequences.

Efficient implementation techniques of an SVM-based speech/music classifier in SMV

For real-time speech and audio encoders used in various multimedia applications, low-complexity encoding algorithms are required. Indeed, accurate classification of input signals is the key prerequisite for variable bit rate encoding, which has been introduced in order to effectively utilize limited communication bandwidth. This paper investigates implementation issues with a support vector machine (SVM)-based speech/music classifier in the selectable mode vocoder (SMV) framework, which is a standard codec adopted by the Third-Generation Partnership Project 2 (3GPP2). While a support vector machine is well known for its superior classification capability, it is accompanied by a high computational cost. In order to achieve a more realizable system, we propose two techniques for the SVM-based speech/music classifier, aimed at reducing the number of classification requests to the classifier. The first technique introduces a simpler classifier that processes some of the input frames instead of the SVM-based classifier, and the second technique skips a portion of input frames based on strong inter-frame correlation in speech and music frames. Our experimental results show that the proposed techniques can reduce the computational cost of the SVM-based classifier by 95.4 % with negligible performance degradation, making it plausible for integration into the SMV codec.

Video Object Detection in Dynamic Scene using Inter-Frame Correlation based Histogram Approach

this paper the problem of video object detection under dynamic scene is considered. The dynamisms of the scene are assumed to be due to illumination variation and swaying of the trees and leaf. Many algorithms have been proposed to cope to this situation. But the major drawback in most of them is misclassified object and background area. Stochastic approaches including MRF based algorithms exist in literature but the practical implementation of such complex models remains still a challenge for the VLSI architecture designers. Thereby real-time object recognition and tracking process largely depends on simple and deterministic approaches and their accuracy. But many a times due to failure of the detection algorithms the efficacy of the hardware remains poor. In previous version of this work a supervised approach to improvised the correct classification of the object and background regions has been proposed. Although the results obtained were as per expectation but the model parameters estimation; such as the threshold selection process was manually done. In order to make it adaptive to the scene, in this paper a classification algorithm has been used which takes the histogram of correlation matrix into account and classify the object. The segmentation of the correlation plane is done by a threshold. This threshold selection is made adaptive to the video sequence considered. This segmented plane along with the moving edge image is then taken into consideration to improvise the correct classification of the moving object in the video. It is observed that the proposed algorithm yields quite manageable results in terms of correct classification.

Principal component analysis algorithm in video compressed sensing

An algorithm of principal component analysis in video compressed sensing is proposed in the paper. Aiming at the compressed sensing problems of video sequences, the inter-frame correlation among the images is analyzed and the transform coefficients with lower value are removed according to the energy concentration characteristics of principal component analysis. Therefore, the sparse realization of video signals in the form of principal component analysis is accomplished and the possibility of the transformation being used in compressed sensing algorithm is verified. Finally, simulation results show that, with the comparison of the traditional algorithm based on wavelet transform, the proposed algorithm can not only improve the reconstructed quality and the visual effects of the video sequence, but also save the sampling resources. Moreover, it is more suitable for stream transmission of multimedia.

Kinect-Like Depth Data Compression

Unlike traditional RGB video, Kinect-like depth is characterized by its large variation range and instability. As a result, traditional video compression algorithms cannot be directly applied to Kinect-like depth compression with respect to coding efficiency. In this paper, we propose a lossy Kinect-like depth compression framework based on the existing codecs, aiming to enhance the coding efficiency while preserving the depth features for further applications. In the proposed framework, the Kinect-like depth is reformed first by divisive normalized bilateral filter (DNBL) to suppress the depth noises caused by disparity normalization, and then block-level depth padding is implemented for invalid depth region compensation in collaboration with mask coding to eliminate the sharp variation caused by depth measurement failures. Before the traditional video coding, the inter-frame correlation of reformed depth is explored by proposed 2D+T prediction, in which depth volume is developed to simulate 3D volume to generate pseudo 3D prediction reference for depth uniqueness detection. The unique depth region, called active region is fed into the video encoder for traditional intra and inter prediction with residual coding, while the inactive region is skipped during depth coding. The experimental results demonstrate that our compression scheme can save 55%-85% in terms of bit cost and reduce coding complexity by 20%-65% in comparison with the traditional video compression algorithms. The visual quality of the 3D reconstruction is also improved after employing our compression scheme.

Compressed Sensing with Generalized Hebbian Algorithm in Video Frame Prediction

Compressed Sensing with Generalized Hebbian Algorithm (GHA) in Video Frame Prediction is proposed in the paper. After analyzing the inter-frame correlation among the images of video sequences, GHA, as neural network algorithm of PCA, is adopted to remove the transform coefficients with lower value according in order to implement video compressed sensing. Furthermore, for statistics of the adjacent frames are similar enough, the algorithm processes superiority in video frame prediction. Simulation results show that, the proposed algorithm can not only improve the reconstructed quality and the visual effects of the video sequence, but also save the sampling resources. Moreover, video frames can be predicted capitally through the application of the algorithm.

Spatio-Temporal Iterative Source–Channel Decoding Aided Video Transmission

Low-complexity uncompressed video transmission meets the requirements of home networking and quality/delay-sensitive medical applications. Hence, it has attracted research attention in recent years. The redundancy inherent in the uncompressed video signals may be exploited by joint source–channel decoding to improve the attainable error resilience. Hence, in this treatise, we study the application of iterative joint source–channel decoding aided uncompressed video transmission, where correlation inherent in the video signals is modeled by a first-order Markov process. First, we propose a spatiotemporal joint source–channel decoding system using a recursive systematic convolutional codec, where both the horizontal and vertical intraframe correlations, as well as the interframe correlations, are exploited by the receiver, hence relying on 3-D information exchange. This scheme may be combined with arbitrary channel codecs. Then, we analyze the three-stage decoder's convergence behavior using 3-D extrinsic information transfer (EXIT) charts. Finally, we benchmark the attainable system performance against a couple of video communication systems, including our previously proposed 2-D scheme, where only intraframe correlations were exploited without invoking a channel codec. Our simulation results show that substantial $E_{b}/N_{0}$ improvements are attainable by the proposed technique.

Adaptive Multihypothesis Prediction Algorithm for Distributed Compressive Video Sensing

A novel adaptive multihypothesis (MH) prediction algorithm for distributed compressive video sensing (DCVS) is proposed in this paper. In the proposed framework, consistent block-based random measurement for each video frame is adopted at the encoder independently. Meanwhile, a mode decision algorithm is applied in CS-blocks via block-based correlation measurements at the decoder. The inter-frame MH mode is selected for the current block wherein the interframe correlation coefficient value exceeds a predetermined threshold. Otherwise, the intraframe MH mode is worthwhile to be selected. Moreover, the adaptive search window and cross-diamond search algorithms on measurement domain are also incorporated to form the dictionary for MH prediction. Both the temporal and spatial correlations in video signals are exploited to enhance CS recovery to satisfy the best linear combination of hypotheses. The simulation results show that the proposed framework can provide better reconstruction quality than the framework using original MH prediction algorithm, and for sequences with slow motion and relatively simple scene composition, the proposed method shows significant performance gains at low measurement subrate.

Color Video Denoising Based on Combined Interframe and Intercolor Prediction

An advanced color video denoising scheme which we call CIFIC based on combined interframe and intercolor prediction is proposed in this paper. CIFIC performs the denoising filtering in the RGB color space, and exploits both the interframe and intercolor correlation in color video signal directly by forming multiple predictors for each color component using all three color components in the current frame as well as the motion-compensated neighboring reference frames. The temporal correspondence is established through the joint-RGB motion estimation (ME) which acquires a single motion trajectory for the red, green, and blue components. Then the current noisy observation as well as the interframe and intercolor predictors are combined by a linear minimum mean squared error (LMMSE) filter to obtain the denoised estimate for every color component. The ill condition in the weight determination of the LMMSE filter is detected and remedied by gradually removing the “least contributing” predictor. Furthermore, our previous work on the LMMSE filter applied in the adaptive luminance-chrominance space (LAYUV for short) is revisited. By reformulating LAYUV and comparing it with CIFIC, we deduce that LAYUV is a restricted version of CIFIC, and thus CIFIC can theoretically achieve lower denoising error. Experimental results verify the improvement brought by the joint-RGB ME and the integration of the intercolor prediction, as well as the superiority of CIFIC over LAYUV. Meanwhile, when compared with other state-of-the-art algorithms, CIFIC provides competitive performance both in terms of the color peak signal-to-noise ratio and in perceptual quality.

Distributed Compressed Video Sensing in Camera Sensor Networks

With the booming of video devices ranging from low-power visual sensors to mobile phones, the video sequences captured by these simple devices must be compressed easily and reconstructed by relatively more powerful servers. In such scenarios, distributed compressed video sensing (DCVS), combining distributed video coding (DVC) and compressed sensing (CS), is developed as a novel and powerful signal-sensing and compression algorithm for video signals. In DCVS, video frames can be compressed to a few measurements in a separate manner, while the interframe correlation is explored by the joint recovery algorithm. In this paper, a new DCVS joint recovery scheme using side-information-based belief propagation (SI-BP) is proposed to exploit both the intraframe and interframe correlations, which is particularly efficient over error-prone channels. The DCVS scheme using SI-BP is designed over two frame signal models, the mixture Gaussian (MG) model and the wavelet hidden Markov tree (WHMT) model. Simulation results evaluated on two video sequences illustrate that the SI-BP-based DCVS scheme is error resilient when the measurements are transmitted through the noisy wireless channels.

Spectral envelope quantization based on conditional inter-frame prediction

The human vocal tract system is commonly modeled by a linear predictive coding (LPC) filter whose coefficients are transformed into a line spectral frequency (LSF) vector for quantization. Predictive split-vector quantization (PSVQ) based on an auto-regressive model (AR-PSVQ), which exploits the inter-frame correlation of the LSF vectors, provides a better rate-distortion performance compared with quantization methods that only consider the intra-frame correlation. In the proposed conditional PSVQ (C-PSVQ), the conditional distribution of the current-frame LSF given the previous-frame LSF is taken into account. Compared with AR-PSVQ, C-PSVQ gains 1 bit in terms of average spectral distortion and 2 bits in terms of the number of outlier frames. Memory requirements and computational complexity of C-PSVQ are similar to those of AR-PSVQ.