Temporal Scalability Research Articles

Delivering scalable video with QoS to the home

User satisfaction is a key factor in the success of novel multimedia services. Yet, to enable service providers and network operators to control and maximize the quality (QoS, QoE) of delivered video streams, quite some challenges remain. In this paper, we particularly focus on three of them. First of all, objectively measuring video quality requires appropriate quality metrics and methods of assessing them in a real-time fashion. Secondly, the recent Scalable Video Coding (SVC) format opens opportunities for adapting video to the available (network) resources, yet the appropriate configuration of video encoding as well as real-time streaming adaptation are largely unaddressed research areas. Thirdly, while bandwidth reservation mechanisms in access/core networks do exist, service providers lack a means for guaranteeing QoS in the increasingly complex home networks (which they are not in full control of). In this paper we offer a broad view on these interrelated issues, by presenting the developments originating in a Flemish research project (including proof-of-concept demonstrations). From a developmental perspective, we propose an architecture combining a real-time video quality monitoring platform, on-the-fly adaptation (optimizing the video quality) and QoS reservation in a heterogeneous home network based on UPnP QoS v3. From a research perspective, we propose a new subjective test procedure that revealed user preference for temporal scalability over quality scalability. In addition, an extensive study on optimizing HD SVC encoding in IPTV scenarios with fluctuating bandwidth showed that under certain bandwidth constraints (prohibiting sufficient fidelity) spatial scalability is a better option than quality scalability.

Fast Context-Adaptive Mode Decision Algorithm for Scalable Video Coding With Combined Coarse-Grain Quality Scalability (CGS) and Temporal Scalability

To speed up the H.264/MPEG scalable video coding (SVC) encoder, we propose a layer-adaptive intra/inter mode decision algorithm and a motion search scheme for the hierarchical B-frames in SVC with combined coarse-grain quality scalability (CGS) and temporal scalability. To reduce computation but maintain the same level of coding efficiency, we examine the rate-distortion (R-D) performance contributed by different coding modes at the enhancement layers (EL) and the mode conditional probabilities at different temporal layers. For the intra prediction on inter frames, we can reduce the number of Intra4×4/Intra 8×8 prediction modes by 50% or more, based on the reference/base layer intra prediction directions. For the EL inter prediction, the look-up tables containing inter prediction candidate modes are designed to use the macroblock (MB) coding mode dependence and the reference/base layer quantization parameters (Qp). In addition, to avoid checking all motion estimation (ME) reference frames, the base layer (BL) reference frame index is selectively reused. And according to the EL MB partition, the BL motion vector can be used as the initial search point for the EL ME. Compared with Joint Scalable Video Model 9.11, our proposed algorithm provides a 20× speedup on encoding the EL and an 85% time saving on the entire encoding process with negligible loss in coding efficiency. Moreover, compared with other fast mode decision algorithms, our scheme can demonstrate a 7-41% complexity reduction on the overall encoding process.

SPC: Fast and Efficient Scalable Predictive Coding of Animated Meshes

Abstract Animated meshes are often represented by a sequence of static meshes with constant connectivity. Due to their frame‐based representation they usually occupy a vast amount of bandwidth or disk space. We present a fast and efficient scalable predictive coding (SPC) scheme for frame‐based representations of animated meshes. SPC decomposes animated meshes in spatial and temporal layers which are efficiently encoded in one pass through the animation. Coding is performed in a streamable and scalable fashion. Dependencies between neighbouring spatial and temporal layers are predictively exploited using the already encoded spatio‐temporal neighbourhood. Prediction is performed in the space of rotation‐invariant coordinates compensating local rigid motion. SPC supports spatial and temporal scalability, and it enables efficient compression as well as fast encoding and decoding. Parts of SPC were adopted in the MPEG‐4 FAMC standard. However, SPC significantly outperforms the streaming mode of FAMC with coding gains of over 33%, while in comparison to the scalable FAMC, SPC achieves coding gains of up to 15%. SPC has the additional advantage over FAMC of achieving real‐time encoding and decoding rates while having only low memory requirements. Compared to some other non‐scalable state‐of‐the‐art approaches, SPC shows superior compression performance with gains of over 16% in bit‐rate.

Interactive region of interest scalability for wavelet based scalable video coder

The current wavelet-based scalable video coder supports quality, spatial and temporal scalabilities required for recent multimedia applications. One of the key scalability required by the Joint Video Team (JVT) is Interactive Region of Interest (IROI) scalability. This paper proposes a technique to interactively extract ROI from scalable sub bit-streams. At the encoder, after sub-band decomposition the 3D wavelet tree is divided into N sub tree, each of these has spatial relevance in the original video, which can be decoded interactively. A generalized model for optimal value of N is proposed, that satisfies minimum requirement of the JVT about IROI support at reduced cost of the encoder. The ROI request from decoder along with resolution and bandwidth are mapped to that in the sub-band domain and respective sub-bit-streams are identified. The extractor assigns more bandwidth to ROI and remaining to non-ROI regions. Hence the decoder reconstructs the video with highest quality at the ROI. The proposed IROI functionality enhances the scalability of video coding by additional dimension.

패킷 손실에 대한 스케일러블 비디오(SVC) 적응기법 및 성능분석

SVC (Scalable Video Coding) is a new video coding standard to provide convergence media service in heterogeneous environments with different networks and diverse terminals through spatial-temporal-quality combined flexible scalabilities. This paper presents the performance analysis on packet loss in the delivery of SVC over IP networks and an efficient adaptation method to packet loss caused by buffer overflow. In particular, SVC with MGS (Medium Grained Scalability) as well as spatial and temporal scalabilities is addressed in the consideration of packet-based adaptation since finer adaptation is possible with a sufficient numbers of quality layers in MGS. The effect on spatio-temporal quality due to the packet loss of SVC with MGS is evaluated. In order to minimize quality degradation resulted by packet loss, the proposed adaptation of MGS based SVC first sets adaptation unit of AU (Access Unit) or GOP corresponding to allowed delay and then selectively discards packets in order of importance in terms of layer dependency. In the experiment, the effects of packet loss on quantitative qualities are analyzed and the effectiveness of the proposed adaptation to packet loss is shown. Keyword : Video adaptation, Scalable Video Coding (SVC), Packet loss

Graceful Degradation in 3GPP MBMS Mobile TV Services Using H.264/AVC Temporal Scalability

These days, there is an increasing interest in Mobile TV broadcast services shown by customers as well as service providers. One general problem of Mobile TV broadcast services is to maximize the coverage of users receiving an acceptable service quality, which is mainly influenced by the user's position and mobility within the cell. In this paper, graceful degradation is considered as an approach for improved service availability and coverage. We present a layered transmission approach for 3GPP's Release 6 Multimedia and Broadcast Service (MBMS) based on temporal scalability using H.264/AVC Baseline Profile. A differentiation in robustness between temporal quality layers is achieved by unequal error protection approach based on either application layer Forward Error Correction (FEC) or unequal transmit power for the layers or even a combination of both. We discuss the corresponding MBMS service as well as network settings and define measures allowing for evaluating the amount of users reached with a certain mobile terminal play-out quality while considering the network cell capacity usage. Using simulated 3GPP Rel. 6 network conditions, we show that if the service and network settings are chosen carefully, a noticeable extension of the coverage of the MBMS service can be achieved.

Scheduling and Resource Allocation for SVC Streaming Over OFDM Downlink Systems

We consider the problem of scheduling and resource allocation for multiuser video streaming over downlink orthogonal frequency division multiplexing (OFDM) channels. The video streams are precoded using the scalable video coding (SVC) scheme that offers both quality and temporal scalabilities. The OFDM technology provides the flexibility of resource allocation in terms of time, frequency, and power. We propose a gradient-based scheduling and resource allocation algorithm, which prioritizes the transmissions of different users by considering video contents, deadline requirements, and transmission history. Simulation results show that the proposed algorithm outperforms the content-blind and deadline-blind algorithms with a gain of as much as 6 dB in terms of average PSNR when the network is congested.

On the use of hierarchical prediction structures for efficient summary generation of H.264/AVC bitstreams

Video summarization refers to an important set of abstraction techniques aimed to provide a compact representation of the video essential to effectively browse and retrieve video content from multimedia repositories. Most of these video summarization techniques, such as image storyboards, video skims and fast previews, are based on selecting some frames or segments. H.264/AVC has become a widely accepted coding standard and is expected that many of the content will be available in this format soon. This paper proposes a generic model of video summarization especially suitable for generating summaries of H.264/AVC bitstreams in a highly efficient manner, using the concept of temporal scalability via hierarchical prediction structures. Along with the model, specific examples of summarization techniques are given to prove the utility of the model.

SNR and temporal scalable coding of 3-D mesh sequences using singular value decomposition

A signal-to-noise ratio (SNR) and temporal scalable coding algorithm for 3-D mesh sequences using singular value decomposition (SVD) is proposed in this work. The proposed algorithm employs SVD to represent a mesh sequence with a small number of basis vectors, and encodes those basis vectors with a bit plane coder. We analytically derive the contribution of each bit plane to the reconstructed mesh quality, and transmit the bit planes in the decreasing order of their amounts of contribution. As the decoder receives more bit planes, it reconstructs higher quality mesh sequences progressively. Moreover, we develop a temporal prediction mode to improve the rate–distortion (R–D) performance further, which also supports temporal scalability. Simulation results demonstrate that the proposed algorithm yields significantly better R–D performance than conventional SVD-based coders.

Motion-compensated DCT temporal filters for efficient spatio-temporal scalable video coding

Although most of the proposals for implementing motion-compensated temporal filtering (MCTF) schemes are based on the wavelet transform, in this paper, we propose an MCTF framework based on the discrete cosine transform (DCT). Using DCT decimation and interpolation, several temporal decomposition structures named motion-compensated DCT temporal filters (MCDCT-TF) are introduced. These structures are able to employ filters of any length with particular emphasis on 5/3 DCT and 7/4 DCT. The proposed MCDCT-TF and the two-dimensional (2D) DCT decimation technique are incorporated into H.264/AVC to provide spatio-temporal scalability. Compared with the current MCTF-based lifting schemes such as Haar, and 5/3 wavelet filters, simulation results show that the proposed MCDCT-TF utilizing longer tap DCT filters achieves a significant improvement in coding gain. The impact of odd/even group of frames, the decimation/interpolation ratios, and motion-compensated connectivity on the MCDCT-TF performance are also analyzed. Moreover, simulation results show that the performance of the presented scalable video coding is close to the single layer H.264/AVC and is slightly inferior to the temporal scalability supported in JSVM, the state-of-the-art scalable video coding standard, that gets its gain from Hierarchical B-pictures. However, our spatio-temporal coding scheme outperforms the spatio-temporal supported in JSVM even if it uses hierarchical B-pictures to improve its gain.

Error Resilient Coding and Error Concealment in Scalable Video Coding

Scalable video coding (SVC), which is the scalable extension of the H.264/AVC standard, was developed by the Joint Video Team (JVT) of ISO/IEC MPEG (Moving Picture Experts Group) and ITU-T VCEG (Video Coding Experts Group). SVC is designed to provide adaptation capability for heterogeneous network structures and different receiving devices with the help of temporal, spatial, and quality scalabilities. It is challenging to achieve graceful quality degradation in an error-prone environment, since channel errors can drastically deteriorate the quality of the video. Error resilient coding and error concealment techniques have been introduced into SVC to reduce the quality degradation impact of transmission errors. Some of the techniques are inherited from or applicable also to H.264/AVC, while some of them take advantage of the SVC coding structure and coding tools. In this paper, the error resilient coding and error concealment tools in SVC are first reviewed. Then, several important tools such as loss-aware rate-distortion optimized macroblock mode decision algorithm and error concealment methods in SVC are discussed and experimental results are provided to show the benefits from them. The results demonstrate that PSNR gains can be achieved for the conventional inter prediction (IPPP) coding structure or the hierarchical bi-predictive (B) picture coding structure with large group of pictures size, for all the tested sequences and under various combinations of packet loss rates, compared with the basic joint scalable video model (JSVM) design applying no error resilient tools at the encoder and only picture copy error concealment method at the decoder.

Perceptual Optimization for Scalable Video Compression Based on Visual Masking Principles

This paper describes a visual optimization strategy for scalable video compression. The challenge scalable coding presents is that truncation of an embedded codestream may induce variable and highly visible distortion. To overcome the deficiencies of visually lossless coding schemes, we propose using an adaptive masking slope to model the perceptual impact of suprathreshold distortion arising from resolution and bit-rate scaling. This allows important scene structures to be better preserved. Following visual masking principles, local sensitivity to distortion is assessed within each frame. To keep the perceptual response uniform against spatiotemporal errors, we mitigate errors compounded by the motion field during temporal synthesis. Visual sensitivity weights are projected into the subband domain along motion trajectories via a process called perceptual mapping. This uses error propagation paths to capture some of the noise-shaping effects attributed to the motion-compensated transform. A key observation is that low contrast regions in the video are generally more susceptible to unmasking of quantization errors. The proposed approach raises the distortion-length slope associated with these critical regions, altering the bitstream embedding order so that visually sensitive sites may be encoded with higher fidelity. Subjective evaluation demonstrates perceptual improvement with respect to bit-rate, spatial and temporal scalability.

Flexible three-band motion-compensated temporal filtering for scalable video coding

A novel scheme for scalable video coding using three-band lifting-based motion-compensated transform is presented in this article. A series of flexible three-band motion-compensated lifting steps are used to implement the temporal wavelet transform, which provide improved compression performance by selecting specific motion model according to real video sequences, and offer higher temporal scalability flexibility by using three-band lifting steps. The experimental results compared with motion picture expert group (MPEG)-4 codec concerning standard video sequences demonstrate the effectiveness of the method.

Accurate Distortion Estimation and Optimal Bandwidth Allocation for Scalable H.264 Video Transmission Over MIMO Systems

In this paper, we propose an optimal strategy for the transmission of scalable video over packet-based multiple-input multiple-output (MIMO) systems. The scalable extension of H.264/AVC that provides a combined temporal, quality and spatial scalability is used. For given channel conditions, we develop a method for the estimation of the distortion of the received video and propose different error concealment schemes. We show the accuracy of our distortion estimation algorithm in comparison with simulated wireless video transmission with packet errors. In the proposed MIMO system, we employ orthogonal space-time block codes (O-STBC) that guarantee independent transmission of different symbols within the block code. In the proposed constrained bandwidth allocation framework, we use the estimated end-to-end decoder distortion to optimally select the application layer parameters, i.e., quantization parameter (QP) and group of pictures (GOP) size, and physical layer parameters, i.e., rate-compatible turbo (RCPT) code rate and symbol constellation. Results show the substantial performance gain by using different symbol constellations across the scalable layers as compared to a fixed constellation.

Concealment of Whole-Picture Loss in Hierarchical B-Picture Scalable Video Coding

H.264/AVC scalable video coding (H.264/AVC SVC), as the scalable extension of H.264/AVC, offers the flexible adaptivity in terms of spatial, temporal and SNR scalabilities for the generated bitstream. However, such compressed video still suffers from the bad playback quality when packet loss occurs over unreliable networks. In this paper, we present an error concealment algorithm to tackle the whole-picture loss problem in H.264/AVC SVC when hierarchical B-picture coding is used to support temporal scalability. In the proposed algorithm, by taking advantage of the temporal relationship among the adjacent video pictures, the motion information of the lost picture is derived simply and efficiently based on the principle of temporal direct mode. Utilizing the derived motion information, the lost picture is concealed by performing motion compensation on the correctly received temporally previous and future video pictures. The experimental results demonstrate that as a post-processing tool, the proposed error concealment algorithm is able to significantly improve both the objective and subjective qualities of the decoded video pictures in the presence of packet losses when compared to the error concealment algorithm used in H.264/AVC SVC reference software. The proposed method can also be applied to H.264/AVC with hierarchical B-picture coding for error concealment.

画像処理・符号化とアプリケーション スケーラブル映像符号化のための動き補償時空間非分離フィルタリング

The use of a novel motion compensated spatio-temporal nonseparable filter (MCSTF) is proposed for scalable video coding. This filter is an alternative to the present motion-compensated temporal filtering (MCTF), a fundamental component for scalable video coding in the next generation. MCTF improve the coding efficiency and provide the temporal scalability. We classify MCSTF as a non-separable sub-sampling version of MCTF, and it provides interlaced pictures as intermediate video sequence by using a spatio-temporal split process. Furthermore, its 1/3-transform structure, which excludes the lifting-update-step, significantly suppresses the PSNR fluctuation that occurs when the existing MCTF technique is used. Our proposed system suppresses PSNR fluctuation with a higher average PSNR than the existing MCTF. Two types of sub-sampling lattices were investigated: the vertical-temporal (VT) quincunx and the face-centered orthorhombic (FCO) ones. Experimental results with entropy coded scalar quantization show the signifaicance of our proposed technique.

Multiple Distortion Measures for Packetized Scalable Media

As the diversity in end-user devices and networks grows, it becomes important to be able to efficiently and adaptively serve media content to different types of users. A key question surrounding adaptive media is how to do Rate-Distortion optimized scheduling. Typically, distortion is measured with a single distortion measure, such as the Mean-Squared Error compared to the original high resolution image or video sequence. Due to the growing diversity of users with varying capabilities such as different display sizes and resolutions, we introduce <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Multiple</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Distortion</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Measures</i> (MDM) to account for a diverse range of users and target devices. MDM gives a clear framework with which to evaluate the performance of media systems which serve a variety of users. Scalable coders, such as JPEG2000 and H.264/MPEG-4 SVC, allow for adaptation to be performed with relatively low computational cost. We show that accounting for MDM can significantly improve system performance; furthermore, by combining this with scalable coding, this can be done efficiently. Given these MDM, we propose an algorithm to generate <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">embedded</i> schedules, which enables low-complexity, adaptive streaming of scalable media packets to minimize distortion across multiple users. We show that using MDM achieves up to 4 dB gains for spatial scalability applied to images and 12 dB gains for temporal scalability applied to video.

Bit-depth scalability compatible to H.264/AVC-scalable extension

While eight-bit playback and display devices will be dominating the marketplaces in the near future, superior visual quality by high bit-depth videos is desirable for applications such as high standard entertainment and healthcare. Hence, conventional eight-bit and high bit-depth digital imaging systems will coexist in marketplaces. Content distributors supporting both of them need to provide different contents for different users, e.g., simulcastly code the different representations for the same video content. This requires more storage or bandwidth for video content delivery. Bit-depth scalability is an efficient tool to solve this problem. However, video coding techniques can allow flexible usage of various versions of the same visual content that may have spatial resolutions and even alterations in color. In this paper, we propose a bit-depth scalable coding solution that is compatible to the scalable extension of H.264/AVC, also known as Scalable Video Coding (SVC). The proposed bit-depth scalable coding is capable of providing an 8-bit AVC main profile or high profile compliant base layer which is multiplexed with a high bit depth (e.g., 10-, 12-, or up to 14-bit) enhancement layer through macroblock level inter-layer bit-depth prediction. New decoding processes for inter-layer bit-depth prediction are introduced to enable bit-depth scalability. Combination with other types of scalability: temporal, spatial and SNR scalability, as well as single-loop decoding is also supported since our algorithm is implemented based on the up-to-date SVC reference software. Furthermore, the proposed solution supports adaptive inter-layer prediction to determine whether or not the inter-layer bit-depth prediction shall be invoked for each Macroblock. The coding efficiency of the proposed bit-depth scalable coding can be further improved by incorporating advanced inter-layer bit-depth prediction algorithms. Experimental results are presented for 8-bit to high bit (10 or 12) scalability and also the combined bit-depth and spatial scalability.

Scalable and Media Aware Adaptive Video Streaming over Wireless Networks

This paper proposes an advanced video streaming system based on scalable video coding in order to optimize resource utilization in wireless networks with retransmission mechanisms at radio protocol level. The key component of this system is a packet scheduling algorithm which operates on the different substreams of a main scalable video stream and which is implemented in a so-called media aware network element. The concerned type of transport channel is a dedicated channel subject to parameters (bitrate, loss rate) variations on the long run. Moreover, we propose a combined scalability approach in which common temporal and SNR scalability features can be used jointly with a partitioning of the image into regions of interest. Simulation results show that our approach provides substantial quality gain compared to classical packet transmission methods and they demonstrate how ROI coding combined with SNR scalability allows to improve again the visual quality.

Traffic and Quality Characterization of the H.264/AVC Scalable Video Coding Extension

The recent scalable video coding (SVC) extension to the H.264/AVC video coding standard has unprecedented compression efficiency while supporting a wide range of scalability modes, including temporal, spatial, and quality (SNR) scalability, as well as combined spatiotemporal SNR scalability. The traffic characteristics, especially the bit rate variabilities, of the individual layer streams critically affect their network transport. We study the SVC traffic statistics, including the bit rate distortion and bit rate variability distortion, with long CIF resolution video sequences and compare them with the corresponding MPEG‐4 Part 2 traffic statistics. We consider (i) temporal scalability with three temporal layers, (ii) spatial scalability with a QCIF base layer and a CIF enhancement layer, as well as (iii) quality scalability modes FGS and MGS. We find that the significant improvement in RD efficiency of SVC is accompanied by substantially higher traffic variabilities as compared to the equivalent MPEG‐4 Part 2 streams. We find that separately analyzing the traffic of temporal‐scalability only encodings gives reasonable estimates of the traffic statistics of the temporal layers embedded in combined spatiotemporal encodings and in the base layer of combined FGS‐temporal encodings. Overall, we find that SVC achieves significantly higher compression ratios than MPEG‐4 Part 2, but produces unprecedented levels of traffic variability, thus presenting new challenges for the network transport of scalable video.