Motion-compensated Temporal Filtering Research Articles

Motion-compensated temporal filtering with optimized motion estimation

The lifting-based discrete wavelet transforms applied temporally along the motion trajectory is a powerful tool in scalable video compression. The longer filter is used for motion-compensated temporal filtering (MCTF) for better de-correlation. The bidirectional motion estimation (ME) used for MCTF is the major speed bottleneck. This paper propose two stages motion predictive technique to optimize the ME. In the first stage, zero motion block prediction is carried out using the content of high frequency frame, obtained after TF without motion compensation. In the second stage, the inter-frame and inter-layer prediction technique is applied to skip the coarse search of any fast ME, except for the first pair of frames in a group of pictures. The proposed algorithm is implemented in MC-EZBC. The variations observed in objective or subjective quality of the decoded video is negligible compared to the speed gained.

Motion-compensated DCT temporal filters for efficient spatio-temporal scalable video coding

Although most of the proposals for implementing motion-compensated temporal filtering (MCTF) schemes are based on the wavelet transform, in this paper, we propose an MCTF framework based on the discrete cosine transform (DCT). Using DCT decimation and interpolation, several temporal decomposition structures named motion-compensated DCT temporal filters (MCDCT-TF) are introduced. These structures are able to employ filters of any length with particular emphasis on 5/3 DCT and 7/4 DCT. The proposed MCDCT-TF and the two-dimensional (2D) DCT decimation technique are incorporated into H.264/AVC to provide spatio-temporal scalability. Compared with the current MCTF-based lifting schemes such as Haar, and 5/3 wavelet filters, simulation results show that the proposed MCDCT-TF utilizing longer tap DCT filters achieves a significant improvement in coding gain. The impact of odd/even group of frames, the decimation/interpolation ratios, and motion-compensated connectivity on the MCDCT-TF performance are also analyzed. Moreover, simulation results show that the performance of the presented scalable video coding is close to the single layer H.264/AVC and is slightly inferior to the temporal scalability supported in JSVM, the state-of-the-art scalable video coding standard, that gets its gain from Hierarchical B-pictures. However, our spatio-temporal coding scheme outperforms the spatio-temporal supported in JSVM even if it uses hierarchical B-pictures to improve its gain.

Flexible three-band motion-compensated temporal filtering for scalable video coding

A novel scheme for scalable video coding using three-band lifting-based motion-compensated transform is presented in this article. A series of flexible three-band motion-compensated lifting steps are used to implement the temporal wavelet transform, which provide improved compression performance by selecting specific motion model according to real video sequences, and offer higher temporal scalability flexibility by using three-band lifting steps. The experimental results compared with motion picture expert group (MPEG)-4 codec concerning standard video sequences demonstrate the effectiveness of the method.

画像処理・符号化とアプリケーション スケーラブル映像符号化のための動き補償時空間非分離フィルタリング

The use of a novel motion compensated spatio-temporal nonseparable filter (MCSTF) is proposed for scalable video coding. This filter is an alternative to the present motion-compensated temporal filtering (MCTF), a fundamental component for scalable video coding in the next generation. MCTF improve the coding efficiency and provide the temporal scalability. We classify MCSTF as a non-separable sub-sampling version of MCTF, and it provides interlaced pictures as intermediate video sequence by using a spatio-temporal split process. Furthermore, its 1/3-transform structure, which excludes the lifting-update-step, significantly suppresses the PSNR fluctuation that occurs when the existing MCTF technique is used. Our proposed system suppresses PSNR fluctuation with a higher average PSNR than the existing MCTF. Two types of sub-sampling lattices were investigated: the vertical-temporal (VT) quincunx and the face-centered orthorhombic (FCO) ones. Experimental results with entropy coded scalar quantization show the signifaicance of our proposed technique.

Subband Weighting With Pixel Connectivity for 3-D Wavelet Coding

Performing optimal bit-allocation with 3-D wavelet coding methods is difficult because energy is not conserved after applying the motion-compensated temporal filtering (MCTF) process and the spatial wavelet transform. The problem cannot be solved by extending the 2-D wavelet coefficients weighting method directly and then applying the result to 3-D wavelet coefficients, since this approach does not consider the complicated pixel connectivity that results from the lifting-based MCTF process. In this paper, we propose a novel weighting method, which takes account of the pixel connectivity, to solve the problem and derive the effect of the quantization error of a subband on the reconstruction error of a group of pictures. We employ the proposed method on a 2-D + t structure with different temporal filters, namely the 5-3 filter and the 9-7 filter. Experiments on various coding parameters and sequences show that the proposed approach improves the bit-allocation performance over that obtained by using the weightings derived without considering the pixel connectivity in the MCTF process.

Enhanced MC-EZBC Scalable Video Coder

In this paper, we provide some recent extension to the scalable subband/wavelet video coder MC-EZBC. The enhanced MC-EZBC employs an adaptive motion-compensated temporal filter (MCTF) framework. Directional I-BLOCKs and overlapped block motion compensation (OBMC) further improve MCTF efficiency. A scalable motion vector coder based on CABAC is shown to improve the overall performance at low bitrates/resolution. Frequency rolloff is incorporated to reduce spatial aliasing at low resolution without PSNR loss at full resolution. Experimental results show that the new features significantly improve the performance of MC-EZBC. We provide several comparisons to other recent coders.

A Cross-Resolution Leaky Prediction Scheme for In-Band Wavelet Video Coding With Spatial Scalability

In most existing in-band wavelet video coding schemes, over-complete wavelet transform is used for the motion-compensated temporal filtering (MCTF) of each spatial subband. It can overcome the shift-variance of critical sampling wavelet transform and improve the coding efficiency of the in-band scheme. However, a dilemma exists in the current implementations of in-band MCTF (IBMCTF), which is whether or not to exploit the spatial highpass subbands in motion compensation of the spatial lowpass subband. The absence of the spatial highpass subbands will result in significant quality loss in the reconstructed full-resolution video, whereas the presence of the spatial highpass subbands may bring serious mismatch error in the decoded low-resolution video since the corresponding highpass subbands may be unavailable at the decoder. In this paper, we first analyze the mismatch error propagation in decoding the low-resolution video. Based on our analysis, we then propose a frame-based cross-resolution leaky prediction scheme for IBMCTF. It can make a good tradeoff between alleviating the low-resolution mismatch and improving the full-resolution coding efficiency. Experimental results show that the proposed scheme can dramatically reduce the mismatch error by 0.3-2.5 dB for low resolution, while the performance loss is marginal for high resolution.

Efficient Architecture Design of Motion-Compensated Temporal Filtering/Motion Compensated Prediction Engine

Since motion-compensated temporal filtering (MCTF) becomes an important temporal prediction scheme in video coding algorithms, this paper presents an efficient temporal prediction engine which not only is the first MCTF hardware work but also supports traditional motion-compensated prediction (MCP) scheme to provide computation scalability. For the prediction stage of MCTF and MCP schemes, modified extended double current Frames is adopted to reduce the system memory bandwidth, and a frame-interleaved macroblock pipelining scheme is proposed to eliminate the induced data buffer overhead. In addition, the proposed update stage architecture with pipelined scheduling and motion estimation (ME)-like motion compensation (MC) with level C+ scheme can also save about half external memory bandwidth and eliminate irregular memory access for MC. Moreover, 76.4% hardware area of the update stage is saved by reusing the hardware resources of the prediction stage. This MCTF chip can process CIF 30 fps in real-time, and the searching range is [-32, 32) for 5/3 MCTF with four-decomposition level and also support 1/3 MCTF, hierarchical B-frames, and MCP coding schemes in JSVM and H.264/AVC. The gate count is 352-K gates with 16.8 KBytes internal memory, and the maximum operating frequency is 60 MHz.

Subband Coupling Aware Rate Allocation for Spatial Scalability in 3-D Wavelet Video Coding

The motion compensated temporal filtering (MCTF) technique, which is extensively used in 3-D wavelet video coding schemes nowadays, leads to signal coupling among various spatial subbands because motion alignment is introduced in the temporal filtering. Using all spatial subbands as a reference enables MCTF to fully take advantage of temporal correlation across frames but inevitably brings drifting problem in supporting spatial scalability. This paper first analyzes the signal coupling phenomenon and then proposes a quantitative model to describe signal propagation across spatial subbands during the MCTF process. The signal propagation is modeled for a single MC step based on the shifting effect of wavelet synthesis filters and then it is extended to multilevel MCTF. This model is called subband coupling aware signal propagation (SCASP) model in this paper. Based on the model, we further propose a subband coupling aware rate allocation scheme as one possible solution to the above dilemma in supporting spatial scalability. To find the optimal rate allocation among all subbands for a specified reconstruction resolution, the SCASP model is used to approximate the reconstruction process and derive the synthesis gain of each subband with regard to that reconstruction. Experimental results have fully demonstrated the advantages of our proposed rate allocation scheme in improving both objective and subjective qualities of reconstructed low-resolution video, especially at middle bit rates and high bit rates.

Improved Update Step for Scalable Video Coding in Video Surveillance

Many evolving video services and applications for intelligent security systems require reliable transmission of high quality video to diverse clients over heterogeneous networks using available system resources. Scalable video coding (SVC) is one of the emerging video compression technologies with such potential capabilities. Advances in lifting-based motion-compensated temporal filtering (MCTF) have enabled highly efficient and flexible spatial, temporal, signal-to-noise ratio (SNR), and complexity scalability to be realized over a wide range of bit rates. In this paper, we present an algorithm to improve the update step of MCTF, which serves as an important informative step for the coding performance of SVC. A novel update-step algorithm, which takes advantage of the chrominance information of the video sequence and the correlation of the motion vectors (MVs) of the neighboring blocks as well as the correlation of the derived update MVs in the low-pass frames, is proposed to improve update step of MCTF by (1) computing correct update motion information, (2) generating correct amount of energy contained in the high-pass frames. Experimental results show that the proposed algorithm can significantly improve the quality of the reconstructed video sequence in visual quality.

Distributed Temporal Multiple Description Coding for Robust Video Transmission

The problem of multimedia communications over best-effort networks is addressed here with multiple description coding (MDC) in a distributed framework. In this paper, we first compare four video MDC schemes based on different time splitting patterns and temporal two- or three-band motion-compensated temporal filtering (MCTF). Then, the latter schemes are extended with systematic lossy description coding where the original sequence is separated into two subsequences, one being coded as in the latter schemes, and the other being coded with a Wyner-Ziv (WZ) encoder. This amounts to having a systematic lossy Wyner-Ziv coding of every other frame of each description. This error control approach can be used as an alternative to automatic repeat request (ARQ) or forward error correction (FEC), that is, the additional bitstream can be systematically sent to the decoder or can be requested, as in ARQ. When used as an FEC mechanism, the amount of redundancy is mostly controlled by the quantization of the Wyner-Ziv data. In this context, this approach leads to satisfactory rate-distortion performance at the side decoders, however it suffers from high redundancy which penalizes the central description. To cope with this problem, the approach is then extended to the use of MCTF for the Wyner-Ziv frames, in which case only the low-frequency subbands are WZ-coded and sent in the descriptions.

On the Coding Efficiency of MCTF Update Steps for Scalable Video Coding

It has been widely believed that motion-compensated update steps performed in the motion-compensated temporal filtering (MCTF) framework improve coding efficiency in comparison to conventional hybrid coding. However, the coding efficiency of MCTF update steps for scalable video coding has not been examined sufficiently in the past. In this paper, we investigate the efficiency of the decoder side update steps by analyzing the effect of performing the inverse update steps at the decoder side. We demonstrate, by theoretical approach, that the MCTF update step at the decoder side does not contribute significantly to the coding efficiency, except at high spatial-temporal resolutions with high bit rates. The results in our study provide a theoretical justification for not including MCTF update steps in the decoding process for scalable video coding

Motion-Compensated Temporal Filtering and Motion Vector Coding Using Biorthogonal Filters

This paper investigates 3-D subband coding via motion-compensated temporal filtering (MCTF) using biorthogonal filters. While nonoverlapping Haar filter-based MCTF has a fixed size group of pictures (GOP) structure, for longer filters we need to use extensions at the GOP ends resulting in coding efficiency loss and/or significant peak signal-to-noise (PSNR) variation. We solve this problem by introducing a "sliding window" approach to MCTF. While we find the longer filters have higher coding gain and significant PSNR improvement at high bit rates, a necessary doubling of the number of motion vectors (MVs) causes a drop in PSNR at lower bit rates. The paper then concentrates on improving the efficiency of both the MV estimation and compression. We employ the MV estimate at higher temporal resolutions as the starting point for local MV refinement at the present temporal resolution thereby reducing the complexity of motion search and obtaining a more uniform motion field. We improve MV coding performance by adapting the context-based binary arithmetic coder (CABAC) from H.264. Instead of encoding MV residuals along a quadtree scanning path, we reduce the MV bit rate by prediction from both neighboring MV blocks and blocks in the previous or the next lower temporal level. Our proposed motion estimation and coding schemes can reduce the MV bit rate by 10%-20% for Haar and 15%-30% for 5/3 MCTF

Two-Dimensional Channel Coding Scheme for MCTF-Based Scalable Video Coding

The motion-compensated temporal filtering (MCTF)-based scalable video coding (SVC) provides a full scalability including spatial, temporal and signal-to-noise ratio (SNR) scalability with fine granularity, each of which may result in different visual effect. This paper addresses a novel approach of two-dimensional unequal error protection (2D UEP) for the scalable video with a combined temporal and quality (SNR) scalability over packet-erasure channel. The bit-stream is divided into scalable subbitstreams based on the structure of MCTF. Each subbitstream is further divided into several quality layers. Unequal quantities of bits are allocated to protect different layers to obtain acceptable quality video with smooth degradation under different transmission error conditions. Experimental results are presented to show the advantage of the proposed 2D UEP scheme over the traditional one-dimensional unequal error protection (1D UEP) scheme. Comparing the proposed method with the 1D UEP scheme on SNR layers, our method gives up to 0.81-dB improvement for some video sequences

Theory of bit allocation analysis

Bit allocation analysis is concerned with the study of efficient combinations of quantizer-based allocations and bit consumption by a model capable of numerical application. The modus operandi of bit-allocation analysis is the use of set theory and the fundamental theorems of mathematical optimization. The most important concept in this analysis is the efficient allocation process, which represents a combination of quantizer-based allocations and bit consumption such that no bit allocation of a quantizer can be increased without decreasing another quantizer's allocation or increasing consumption. The main result of this paper allows one to characterize the concept of efficient allocation process by profit maximization with respect to any allocation-consumption combination among competing quantizers. In bit allocation analysis, the system makes a choice from the set of efficient allocations at any given time by using the appropriate strategy for computing the profit vector. It may allow attending to different parameters of interest at different bit rates within the same spatial locations. It is a typical linear programming problem, for which computational methods are well known and widely used in practice. The comparative performance of the 3-D set partitioning in hierarchical trees with motion-compensated temporal filtering and the proposed coder (without motion filtering) using bit allocation analysis is here evaluated on a set of sequences of moving targets.

4-D Wavelet-Based Multiview Video Coding

The conventional multiview video coding (MVC) schemes, utilizing both neighboring temporal frames and view frames as possible references, have only shown a slight gain over those using temporal frames alone in terms of coding efficiency. The reason for this is that the neighboring temporal frames exhibit stronger correlation with the current frame and the view frames often fail to be selected as references. This paper proposes an elegant MVC framework using high dimensional wavelet, which rightly matches the inherent high dimension property of multiview video. It also makes a better usage of both temporal and view correlations thanks to the hierarchical decomposition. Besides the proposed framework, this paper also investigates MVC coding from the following aspects. First, a disparity-compensated view filter (DCVF) with pixel alignment is proposed, which can accommodate both global and local view disparities among view frames. The proposed DCVF and the existing motion-compensated temporal filter (MCTF) unify the view and temporal decompositions as a generic lifting transform. Second, an adaptive decomposition structure based on the analysis of the temporal and view correlations is proposed. A Lagrangian cost function is derived to determine the optimum decomposition structure. Third, the major components of the proposed MVC coding are figured out, including macroblock type design, subband coefficient coding, and rate allocation. Extensive experiments are carried out on the MPEG 3DAV test sequences and the superior performance of the proposed MVC coding is demonstrated. In addition, the proposed MVC framework can easily support temporal, spatial, SNR, as well as view scalabilities

System Analysis of VLSI Architecture for 5/3 and 1/3 Motion-Compensated Temporal Filtering

Motion-compensated temporal filtering (MCTF) is an innovative prediction scheme for video coding and it has become the core technology of the coming video coding standard, Scalable Video Coding. Since MCTF is important, this paper provides the system analysis of MCTF for hardware architecture design, including computational complexity, external memory bandwidth, and external memory size. The one-level MCTF is analyzed first, in which several frame-level data reuse schemes are proposed and the tradeoffs between external memory usages and on-chip memory size in these frame-level data reuse schemes are also discussed. Next, the analysis is extended to multilevel MCTF. The computational complexity of multilevel MCTF is close to that of traditional MC prediction with two reference frames. The memory bandwidth of multilevel MCTF depends on the frame-level data reuse scheme and performing the update stage or not. The external memory size is linearly proportional to the number of decomposition levels. Finally, a real-life test case is given to compare the system requirements between MCTF with various frame-level data reuse schemes and the prediction scheme of H.264/AVC.

3-band motion-compensated temporal structures for scalable video coding

Recent breakthroughs in motion-compensated temporal wavelet filtering have finally enabled implementation of highly efficient scalable and error-resilient video codecs. These new wavelet codecs provide numerous advantages over nonscalable conventional solutions techniques based on motion-compensated prediction, such as no recursive predictive loop, separation of noise and sampling artifacts from the content through use of longer temporal filters, removal of long range as well as short range temporal redundancies, etc. Moreover, these wavelet video coding schemes can provide flexible spatial, temporal, signal-to-noise ratio and complexity scalability with fine granularity over a large range of bit rates, while maintaining a very high coding efficiency. However, most motion-compensated wavelet video schemes are based on classical two-band decompositions that offer only dyadic factors of temporal scalability. In this paper, we propose a three-band temporal structure that extends the concept of motion-compensated temporal filtering (MCTF) that was introduced in the classical lifting framework. These newly introduced structures provide higher temporal scalability flexibility, as well as improved compression performance compared with dyadic Haar MCTF.

Multiple description scalable video coding based on 3D lifted wavelet transform

In this work, a new method to deal with the unconnected pixels in motion compensated temporal filtering (MCTF) is presented, which is designed to improve the performance of 3D lifted wavelet coding. Furthermore, multiple description scalable coding (MDSC) is investigated, and novel MDSC schemes based on 3D wavelet coding are proposed, using the lifting imple- mentation of temporal filtering. The proposed MDSC schemes can avoid the mismatch problem in multiple description video coding, and have high scalability and robustness of video transmission. Experimental results showed that the proposed schemes are feasible and adequately effective.

Improved update operators for lifting-based motion-compensated temporal filtering

Motion-compensated temporal filtering (MCTF) is an essential ingredient of the recently developed wavelet-based scalable video coding schemes. Lifting implementation of these decompositions represents a versatile tool for spatiotemporal optimizations, and several improvements have already been proposed in this framework. In this paper, we propose new spatiotemporal update operators based either on weighted averages of multiple connected pixels or on nonlinear filtering of these pixels. We demonstrate the improved performance of the proposed operators, both theoretically and experimentally.