Motion-compensated Temporal Filtering Research Articles

Hardware Efficient Hybrid Encoder for Video Surveillance Application

Extensive use of digital multimedia has led to the development of advance video processing techniques for development of multimedia applications. Application such as video surveillance requires 247 recording and streaming. So, the bandwidth and storage costs become significant. With introduction of video streaming over internet, where different kinds of end users request same content with different available bandwidth, it requires scalable video coding (SVC). These challenges can be overcome by developing new techniques to reduce redundancy in subsequent frames and to improve the coding efficiency. In this paper, overlapping weighted linear sum (OWLS) pre-processing method and its hardware architecture are proposed. It is implemented using field progrmmable gate array (FPGA) and the application specific integrated circuit (ASIC) is also developed using TSMC180nm technology standard cell library. Results show improvement in terms of power and area as compared to the existing work. In motion compensated temporal filtering (MCTF), wavelet transform is implemented by temporal filters. Architecture for 5/3 Lifting MCTF is also implemented and compared with baseline H.264 video codec. Simulation results show that the average peak signal to noise ratio (PSNR) improvement is 2.36[Formula: see text]dB. The MCTF design using 5/3 Lifting filter is synthesized for Virtex-5 FPGA and compared with the existing close-loop architecture with better performance.

EEG Compression Using Motion Compensated Temporal Filtering and Wavelet Based Subband Coding

Electroencephalography (EEG) signals are commonly used in medical applications for prevention, diagnosis, and detection of neurological diseases. These EEG signals have also been used in designing brain computer interfaces for assistive technologies. For densely placed electrodes and long EEG recordings, a large amount of data needs to be stored, preferably in compressed form. This EEG signal compression is particularly required in an out-of-the-lab environment so that these signals are efficiently transmitted over wired/wireless communication channels. To this end, we propose a novel compression scheme for EEG signals, which exploits the intra-channel redundancy using motion compensated temporal filtering (MCTF) and discrete wavelet transform (DWT) based sub-band coding. In the pre-processing stage, multi-frame data is constructed such that each group of picture (GOP) contains information from a single channel of EEG data. This helps in removing the intra-channel redundancy. We apply MCTF on each GOP to exploit temporal redundancy, following which the DWT is applied on temporally decomposed frames to exploit spatial redundancy. Each spatio-temporal decomposed frame is assigned a bit budget for minimum distortion. For this purpose, we assign more bit budget to temporally decomposed low pass frames as compared to high pass frames. Spatio-temporal frames are then encoded at the assigned bit rate by using set partitioning in hierarchical tree (SPIHT) algorithm to create the bit stream. Our experimental results showed 4.5% and 2.4% reduction in distortion at the same data rate for BCI-3 and BCI-4 datasets, respectively. These results improve upon the reduction in data size achieved using state-of-the-art compression methods such as SPIHT and SPIHT with independent component analysis.

Layer Selection in Progressive Transmission of Motion-Compensated JPEG2000 Video

MCJ2K (Motion-Compensated JPEG2000) is a video codec based on MCTF (Motion- Compensated Temporal Filtering) and J2K (JPEG2000). MCTF analyzes a sequence of images, generating a collection of temporal sub-bands, which are compressed with J2K. The R/D (Rate-Distortion) performance in MCJ2K is better than the MJ2K (Motion JPEG2000) extension, especially if there is a high level of temporal redundancy. MCJ2K codestreams can be served by standard JPIP (J2K Interactive Protocol) servers, thanks to the use of only J2K standard file formats. In bandwidth-constrained scenarios, an important issue in MCJ2K is determining the amount of data of each temporal sub-band that must be transmitted to maximize the quality of the reconstructions at the client side. To solve this problem, we have proposed two rate-allocation algorithms which provide reconstructions that are progressive in quality. The first, OSLA (Optimized Sub-band Layers Allocation), determines the best progression of quality layers, but is computationally expensive. The second, ESLA (Estimated-Slope sub-band Layers Allocation), is sub-optimal in most cases, but much faster and more convenient for real-time streaming scenarios. An experimental comparison shows that even when a straightforward motion compensation scheme is used, the R/D performance of MCJ2K competitive is compared not only to MJ2K, but also with respect to other standard scalable video codecs.

Content Adaptive Lagrange Multiplier Selection for Rate-Distortion Optimization in 3-D Wavelet-Based Scalable Video Coding.

Rate-distortion optimization (RDO) plays an essential role in substantially enhancing the coding efficiency. Currently, rate-distortion optimized mode decision is widely used in scalable video coding (SVC). Among all the possible coding modes, it aims to select the one which has the best trade-off between bitrate and compression distortion. Specifically, this tradeoff is tuned through the choice of the Lagrange multiplier. Despite the prevalence of conventional method for Lagrange multiplier selection in hybrid video coding, the underlying formulation is not applicable to 3-D wavelet-based SVC where the explicit values of the quantization step are not available, with on consideration of the content features of input signal. In this paper, an efficient content adaptive Lagrange multiplier selection algorithm is proposed in the context of RDO for 3-D wavelet-based SVC targeting quality scalability. Our contributions are two-fold. First, we introduce a novel weighting method, which takes account of the mutual information, gradient per pixel, and texture homogeneity to measure the temporal subband characteristics after applying the motion-compensated temporal filtering (MCTF) technique. Second, based on the proposed subband weighting factor model, we derive the optimal Lagrange multiplier. Experimental results demonstrate that the proposed algorithm enables more satisfactory video quality with negligible additional computational complexity.

Denoising Algorithm for CFA Image Sensors Considering Inter-Channel Correlation.

In this paper, a spatio-spectral-temporal filter considering an inter-channel correlation is proposed for the denoising of a color filter array (CFA) sequence acquired by CCD/CMOS image sensors. Owing to the alternating under-sampled grid of the CFA pattern, the inter-channel correlation must be considered in the direct denoising process. The proposed filter is applied in the spatial, spectral, and temporal domain, considering the spatio-tempo-spectral correlation. First, nonlocal means (NLM) spatial filtering with patch-based difference (PBD) refinement is performed by considering both the intra-channel correlation and inter-channel correlation to overcome the spatial resolution degradation occurring with the alternating under-sampled pattern. Second, a motion-compensated temporal filter that employs inter-channel correlated motion estimation and compensation is proposed to remove the noise in the temporal domain. Then, a motion adaptive detection value controls the ratio of the spatial filter and the temporal filter. The denoised CFA sequence can thus be obtained without motion artifacts. Experimental results for both simulated and real CFA sequences are presented with visual and numerical comparisons to several state-of-the-art denoising methods combined with a demosaicing method. Experimental results confirmed that the proposed frameworks outperformed the other techniques in terms of the objective criteria and subjective visual perception in CFA sequences.

Video quality assessment using motion-compensated temporal filtering and manifold feature similarity.

Well-performed Video quality assessment (VQA) method should be consistent with human visual systems for better prediction accuracy. In this paper, we propose a VQA method using motion-compensated temporal filtering (MCTF) and manifold feature similarity. To be more specific, a group of frames (GoF) is first decomposed into a temporal high-pass component (HPC) and a temporal low-pass component (LPC) by MCTF. Following this, manifold feature learning (MFL) and phase congruency (PC) are used to predict the quality of temporal LPC and temporal HPC respectively. The quality measures of the LPC and the HPC are then combined as GoF quality. A temporal pooling strategy is subsequently used to integrate GoF qualities into an overall video quality. The proposed VQA method appropriately processes temporal information in video by MCTF and temporal pooling strategy, and simulate human visual perception by MFL. Experiments on publicly available video quality database showed that in comparison with several state-of-the-art VQA methods, the proposed VQA method achieves better consistency with subjective video quality and can predict video quality more accurately.

Adaptive Hybrid Digital–Analog Video Transmission in Wireless Fading Channel

We propose an adaptive hybrid digital–analog video transmission (A-HDAVT) scheme for robust video streaming in mobile networks with realistic fading channels. This scheme is fundamentally different from recent research in hybrid digital–analog video transmission in which wireless channels are unrealistically assumed to be Gaussian. For fading channels, it is critical to take full advantage of diversity in both video contents and multiuser channels. Like all hybrid approaches, A-HDAVT is designed to exploit the benefits from both digital and analog systems. To achieve this goal, each group of pictures is first transformed into one low-pass frame and several high-pass frames with motion-compensated temporal filtering. The critical low-pass frame is reliably transmitted as base layer in a digital mode, while high-pass frames are transmitted as enhancement layers in an analog mode to achieve desired graceful degradation performance. In analog transmission, we introduce a channel prediction-based adaptive power-distortion optimization (P-APDO) scheme to combat channel fading in mobile networks. The basic idea behind P-APDO is to perform power allocation based on the video content as well as the predicted channel status. Furthermore, we also investigate the multiuser scenarios in which the content diversity and channel diversity among users are appropriately exploited. Extensive simulations have been carried out to evaluate the performance of A-HDAVT under various degrees of channel fading. The results show that A-HDAVT achieves significant performance gains over competing schemes Robust Uncoded Video Transmission, Parcast, and Softcast in both single-user and multiuser scenarios.

An improved 3D wavelet-based scalable video coding codec for MC-EZBC

With the rapid growth of modern multimedia applications, 3D wavelet-based scalable video coding (SVC) codec has received considerable attention lately because of its high coding performance and flexibility in bitstream scalability. It combines the motion-compensated temporal filtering (MCTF) together with the spatial decomposition to produce an embedded bitstream offering various levels of video quality over the heterogeneous networks. However, in the existing 3D wavelet-based SVC schemes, where the block types for block matching algorithms are limited, weighting matrices for block-wise motion compensation are fixed, and variations in activities of temporal subbands are not considered in the selection of the Lagrange multiplier for mode decision. In this paper, our major contribution is to provide some recent extensions to the well-known scalable subband/wavelet video codec Motion-Compensated Embedded Zero Block Coding (MC-EZBC) using three novel and content adaptive algorithms. Firstly, the enhanced hierarchical variable size block matching (Enhanced HVSBM) algorithm is proposed for the variable block size motion estimation. Then, the rate-distortion optimization (RDO) based adaptive Lagrange multiplier selection model for mode decision is presented. Finally, we introduce the adaptive weighting matrices design for overlapped block motion compensation (OBMC). Experimental results show that all the three proposed algorithms significantly improve the overall coding performance of MC-EZBC. Comparisons with other popular wavelet-based SVC codecs demonstrate the effectiveness of our improved codec in terms of both video quality assessment and computational complexity.

Denoising and Resource Allocation in Uncoded Video Transmission

The recently revitalized uncoded transmission has shown great potential in handling channel variations and user heterogeneities for wireless video communications. In this paper, we propose to adopt motion-compensated temporal filtering (MCTF) at the sender and denoising techniques at the receiver to fully exploit the temporal and spatial redundancy in video signals and therefore significantly improve the efficiency of uncoded video transmission. Although there are readily applicable MCTF and denoising techniques, integrating them into the uncoded video transmission framework is nontrivial, especially in the case of denoising. To address these challenges, we first propose spatial-domain transmission and cascaded denoising to harness the potential of denoising techniques. Second, we develop novel resource allocation algorithms based on variable-size L-shaped chunk division, which are not only compatible with spatial-domain transmission, but also achieves better energy efficiency than existing schemes based on fixed-size rectangular chunks. Our experimental results show that when transmitting 720p videos, our scheme achieves up to a 3.3 dB gain in video PSNR over the state-of-the-art uncoded transmission scheme SoftCast and up to a 5.3 dB gain over a digital scheme based on robust rate adaptation and H.264 scalable video coding (SVC).

Motion Compensated Temporal Filtering in Scalable Video Coding Algorithm

the temporal correlation exists between the frame and the frame in a video message, in order to solve temporal redundancy.This paper adopts motion-compensated temporal filtering method for removing temporal correlation, motion-compensated temporal filtering and Haar wavelet lifting techniques are studied and discussed; finally presented the improved motion compensated temporal filtering methods. Results show the new method improves coding efficiency and scalability.

움직임 벡터 필드의 상관도 향상을 통한 효과적인 MCTF 방법

본 논문에서는 웨이블릿 변환 기반의 스케일러블 비디오 부호화의 핵심 요소인 움직임 보상 시간적 필터링 (motion compensated temporal filtering; MCTF)의 성능향상을 위한 효과적인 움직임 예측 기법을 제안한다. MCTF의 성능향상과 밀접한 관련이 있는 픽셀 연결성을 개선하기 위하여 변형 중간값 연산 및 주변 블록의 움직임 벡터를 이용한 움직임 예측 방법으로 움직임 벡터 필드를 평탄화한다. 또한 영상의 복잡도 등의 특성에 따라 가변 블록크기로 움직임 예측을 수행하여 움직임 벡터 필드의 상관도를 더욱 향상시킨다. 제안하는 방법은 시간적 필터링 과정에서 고주파 프레임으로 분해되는 에너지를 줄여 MCTF 및 전체 스케일러블 부호기의 성능을 향상시킨다. 실험결과 기존의 고정 블록크기 전역탐색 방법과 비교하여 제안하는 방법이 시간적 고주파 부대역 프레임에 포함된 에너지를 최대 30.33%까지 감소시키는 것을 확인하였다. This paper presents an effective motion estimation to improve the performance of the motion compensated temporal filtering (MCTF) which is a core part of the wavelet-based scalable video coding. The proposed scheme makes the motion vector field uniform by the modified median operation and the search strategies using adjacent motion vectors, in order to enhance the pixel connectivity which is significantly relevant to the performance of the MCTF. Moreover, the motion estimation with variable block sizes that reflects the features of frames is introduced for further correlation improvement of the motion vector field. Experimental results illustrate that the proposed method reduces the decomposed energy on the temporal high frequency subband frame up to 30.33% in terms of variance compared to the case of the full search with fixed block sizes.

Scalable video coding using motion-compensated temporal filtering and intra-band wavelet based compression

Scalable video compression is a crucial task that allows for high flexibility of video streams to different networks in various applications. Current video coding techniques exploit temporal correlation using motion-compensated predictive or filtering approaches. Particularly, motion-compensated temporal filtering (MCTF) is a useful framework for scalable video compression schemes. In this paper, we propose a new scalable video coding method that combines open-loop motion-compensated prediction with an embedded intra-band wavelet based compression. Our major objective is to provide a wavelet based video coding system that circumvents the drawbacks of conventional closed-loop prediction systems, without sacrificing compression performance. To improve the coding efficiency, we adaptively weight the target bitrate according to the temporal frame position in the temporal pyramid. Comparisons with state-of-the-art scalable video coding solutions confirm an overall coding efficiency gain of the proposed method specially at high bitrates.

Adaptive Temporal Trajectory Filtering for Video Compression

Most in-loop filters currently being employed in video compression algorithms use spatial information from a single frame of the video sequence only. In this paper, a new filter is introduced and investigated that combines both spatial and temporal information to provide subjective and objective quality improvement. The filter only requires a small overhead on slice level while using the temporal information conveyed in the bit stream to reconstruct the individual motion trajectory of every pixel in a frame at both encoder and decoder. This information is then used to perform pixel-wise adaptive motion-compensated temporal filtering. It is shown that the filter performs better than the state-of-the-art codec H.264/AVC over a large range of sequences and bit rates. Additionally, the filter is compared with another, Wiener-based in-loop filtering approach and a complexity analysis of both algorithms is conducted.

Redundancy controllable scalable unbalanced multiple description bitstream generation for peer-to-peer video streaming

In peer-to-peer video streaming, the robustness to peer and packet losses is regarded very important in order to enjoy a good quality of experience. Multiple description coding (MDC) schemes are known to provide high robustness to packet losses. In peer-to-peer streaming in addition to high robustness, efficient controllability of data rate and redundancy among the data streams from peers is also vital. In this paper, a novel framework for scalable multiple description coding scheme for peer-to-peer video streaming is presented by addressing these requirements. The proposed MDC solution is based on the multiple description scalar quantization (MDSQ) by addressing the joint decoding of unbalanced descriptions and addition of jointly decodable successive refinement layers to side descriptions. Firstly, the design conditions for MDSQ with constrained successive refinement to obtain scalable multiple description streams where the bit streams can be truncated at any point to obtain lower quality spatial temporal descriptions are firstly proposed. Then the design conditions for joint decoding of two or more side descriptions from different multiple description scalar quantizers, originating from various bin spread factors leading to controllable redundancy are proposed. These design conditions enable joint redundancy and data rate control for streams coming from different peers, thereby, enabling high robustness to packet losses as well as peer losses with the adaptability of redundancy levels and truncation of scalable streams. The proposed design constraints are used within the motion compensated temporal filtering (MCTF) framework to demonstrate its advancement in robust peer-to-peer video streaming. The results show significant improvements over conventional MDC based on simple MSDQ and over single description scalable video.

2D+t Wavelet Domain Video Watermarking

A novel watermarking framework for scalable coded video that improves the robustness against quality scalable compression is presented in this paper. Unlike the conventional spatial-domain (t + 2D) water-marking scheme where the motion compensated temporal filtering (MCTF) is performed on the spatial frame-wise video data to decompose the video, the proposed framework applies the MCTF in the wavelet domain (2D + t) to generate the coefficients to embed the watermark. Robustness performances against scalable content adaptation, such as Motion JPEG 2000, MC-EZBC, or H.264-SVC, are reviewed for various combinations of motion compensated 2D + t + 2D using the proposed framework. The MCTF is improved by modifying the update step to follow the motion trajectory in the hierarchical temporal decomposition by using direct motion vector fields in the update step and implied motion vectors in the prediction step. The results show smaller embedding distortion in terms of both peak signal to noise ratio and flickering metrics compared to frame-by-frame video watermarking while the robustness against scalable compression is improved by using 2D + t over the conventional t + 2D domain video watermarking, particularly for blind watermarking schemes where the motion is estimated from the watermarked video.

Fast Implementation of the Subband Weighting for 3D Wavelet Coding

Performing optimal bit-allocation with 3D wavelet coding methods is difficult, because energy is not conserved after applying the motion-compensated temporal filtering (MCTF) process and the spatial wavelet transform. The problem has been solved by extending the 2D wavelet coefficients weighting method with the consideration of the complicated pixel connectivity resulting from the lifting-based MCTF process. However, the computation of weighting factors demands much computing time and vast amount of memory. In this paper, we propose a method, which facilitates the property of sparse matrices, to compute the subband weighing factors efficiently. The proposed method is employed for the 5–3 temporal filter and the 9–7 temporal filter in the lifting-based MCTF process. Experiments on various video sequences show that the computing time and memory needed for the computation of weighting factors are dramatically reduced by applying the proposed approach.

AND-OR tree-based mode selection for video coding

The conventional video coding approach is pragmatic in that researchers design different methods and compare their results to state-of-the-art methods. Although the H.264/AVC baseline is effective, there is a need to develop an abstract view of video coding, in the hope that new insights can be derived from the abstraction. In this paper, we propose a preliminary approach based on an AND-OR tree representation of video coding. We show that the H.264/AVC baseline can be represented as an AND-OR tree structure. Based on the AND-OR tree representation, we propose two video coding systems: one is a T+2D wavelet codec based on a motion-compensated temporal filtering (MCTF) lifting structure, and the other is the AND-OR tree implementation of the H.264/AVC baseline. We also compare the proposed systems’ coding performance in terms of the PSNR with that of H.264/AVC JM 16.2.

Arbitrary ROI-based wavelet video coding

This paper presents a novel arbitrary shape region of interest (ROI) coding for scalable wavelet video codec. The motion information of the ROIs is estimated by macroblock padding and polygon matching. The derived motion vectors are set as the motion trajectory of the samples to generate a one-dimensional temporal signal. This signal is then filtered to reduce the temporal redundancy using motion compensated temporal filtering for arbitrary shape ROI. Compared to traditional non-ROI coding, the reconstructed quality of the ROI coding can be significantly improved at low bit-rates. The efficiency of motion compensated temporal filtering (MCTF) based on arbitrary ROI is also compared with that of the video object coding in MPEG-4. Based on large number of experiments, the ability of the MCTF to reduce the temporal redundancy is demonstrated as better than (or at least comparable to) that of MPEG-4.

Digital hologram compression technique by eliminating spatial correlations based on MCTF

This paper proposed a new algorithm which compresses holographic signals captured or generated as digital data. A captured digital hologram is partitioned into multiple subregions with the same size. Next, if the Discrete Cosine Transform (DCT) is applied to these subregions, information that is visually similar to the original object is derived. Then the correlated subregions are organized into a single video stream and applied with Motion-Compensated Temporal Filtering (MCTF). Lastly, the streams are compressed using a lossy coding algorithm. This paper executed compression from 10:1 to 200:1 of digital holograms and observed the result after the reconstruction process. The numerical results represented about 11% increased compression efficiency compared to the previous researches.

A Rate Control Algorithm for Scalable Video Coding

The joint scalable video model(JSVM)as a scalable extension of H.264/AVC provides a high efficient solution for scalable video coding(SVC).However,there is not efficient rate control scheme is specified.This paper presents a new rate control scheme for the JSVM based on the special structure of hierarchical B decomposition(or motion compensation temporal filtering(MCTF)).In a sense of bit allocation,a novel hierarchical bit allocation strategy is proposed.Meanwhile,the different rate-distortion(R-D)model is provided for each kind picture with different picture type or temporal decomposition level.Experiment results show that the mismatch of target bit rate and real bit rate generated for coding doesn't exceed 2%;meanwhile,the PSNR improvement can be up to 1dB in low bit rate;furthermore,comparing to the original rate control method which achieves the target bit rate through regulating quantization parameter(QP)step by step,the computation complexity of the proposed rate control algorithm base on experimental model is reduced drastically.