Modern Video Codecs Research Articles

Error Resilience of H.264/avc Coding Structures for Delivery over Wireless Networks

Video content represents a large portion of global network traffic, and it is expected that this trend will continue as 4K video streaming becomes more popular and more applications such as entertainment and surveillance become available. Wireless networks are becoming the most important technologies for delivering cost-effective content, including video. The unpredictability of wireless channels, as well as the sensitivity of video content to packet losses, errors, and delays makes video delivery over wireless network difficult. Furthermore, when compared to wire links, congestion and complex traffic patterns make wireless channels much worse. Packet losses and network errors can be mitigated with the help of modern codecs. This study looks at the modern video codec H.264/Advanced Video Coding (AVC) and how to use the built-in error resilience tools to improve end-to-end video quality over simulated wireless networks. Network Simulator 3 (NS-3) was used and a framework for video quality assessment based on EvalVid 2.7 tools was employed. We tested the performance of several coding structures in the presence of different intra coding techniques. Results for less active video sequences showed that the IBP coding structure with intra period of 18 achieved best performance at lower loss rates. The addition of an intra MB line or extra random intra MBs did not seem to offer extra protection against errors for these sequences. For more active video sequences, IBBBP coding structure without or with an intra MB line performed better than all other configurations at all loss rates

Edge-Based Video Compression Texture Synthesis Using Generative Adversarial Network

It has been recognized that texture patterns with abundant high-frequency components, such as grass and water, produce visual masking effects, and the distortion in textures is hard to be perceived by human eyes than structure regions. However, modern video codecs in a rate-distortion optimized manner usually consume a lot of bits to encode textures, leading to the insufficiency in perceptual coding performance. Nowadays, with the rapid development of deep learning, learning based texture synthesis methods have been proposed to replace the coding process of prediction residuals to reduce the rate cost. In this paper, we present a deep texture synthesizer named edge-based texture synthesis framework (ETSF). At encoder side, the framework detects texture regions by semantic and fidelity classification criteria, and the detected regions are quantized coarsely by the hybrid coding framework. In texture characterization, ETSF extracts low-level edge features representing pixel intensity variation. Feature processing tools are developed to remove the spatiotemporal redundancy of edges. The processed edge information is compressed and transmitted. To effectively recover textures, we design an edge-based texture synthesis generative adversarial network (ETSGAN) at the decoder of ETSF, which can incorporate edge information into convolutional layers and generate realistic textures. Experimental results on a collected texture dataset show that the proposed ETSF can achieve an average of -12.8%, -14.2% and -9.6% MOS BD-rate under lowdelay_B, lowdelay_P and random_access configurations of VVC coding, respectively.

Perceptual Video Quality Prediction Emphasizing Chroma Distortions

Measuring the quality of digital videos viewed by human observers has become a common practice in numerous multimedia applications, such as adaptive video streaming, quality monitoring, and other digital TV applications. Here we explore a significant, yet relatively unexplored problem: measuring perceptual quality on videos arising from both luma and chroma distortions from compression. Toward investigating this problem, it is important to understand the kinds of chroma distortions that arise, how they relate to luma compression distortions, and how they can affect perceived quality. We designed and carried out a subjective experiment to measure subjective video quality on both luma and chroma distortions, introduced both in isolation as well as together. Specifically, the new subjective dataset comprises a total of 210 videos afflicted by distortions caused by varying levels of luma quantization commingled with different amounts of chroma quantization. The subjective scores were evaluated by 34 subjects in a controlled environmental setting. Using the newly collected subjective data, we were able to demonstrate important shortcomings of existing video quality models, especially in regards to chroma distortions. Further, we designed an objective video quality model which builds on existing video quality algorithms, by considering the fidelity of chroma channels in a principled way. We also found that this quality analysis implies that there is room for reducing bitrate consumption in modern video codecs by creatively increasing the compression factor on chroma channels. We believe that this work will both encourage further research in this direction, as well as advance progress on the ultimate goal of jointly optimizing luma and chroma compression in modern video encoders.

Mononizing binocular videos

This paper presents the idea of mono-nizing binocular videos and a framework to effectively realize it. Mono-nize means we purposely convert a binocular video into a regular monocular video with the stereo information implicitly encoded in a visual but nearly-imperceptible form. Hence, we can impartially distribute and show the mononized video as an ordinary monocular video. Unlike ordinary monocular videos, we can restore from it the original binocular video and show it on a stereoscopic display. To start, we formulate an encoding-and-decoding framework with the pyramidal deformable fusion module to exploit long-range correspondences between the left and right views, a quantization layer to suppress the restoring artifacts, and the compression noise simulation module to resist the compression noise introduced by modern video codecs. Our framework is self-supervised, as we articulate our objective function with loss terms defined on the input: a monocular term for creating the mononized video, an invertibility term for restoring the original video, and a temporal term for frame-to-frame coherence. Further, we conducted extensive experiments to evaluate our generated mononized videos and restored binocular videos for diverse types of images and 3D movies. Quantitative results on both standard metrics and user perception studies show the effectiveness of our method.

A Novel Hybrid Cryptosystem for Secure Streaming of High Efficiency H.265 Compressed Videos in IoT Multimedia Applications

In this modernistic age of innovative technologies like big data processing, cloud computing, and Internet of things, the utilization of multimedia information is growing daily. In contrast to other forms of multimedia, videos are extensively utilized and streamed over the Internet and communication networks in numerous Internet of Multimedia Things (IoMT) applications. Consequently, there is an immense necessity to achieve secure video transmission over modern communication networks due to the third-party exploitation and falsification of transmitted and stored digital multimedia data. The present methods for secure communication of multimedia content between clouds and mobile devices have constraints in terms of processing load, memory support, data size, and battery power. These methods are not the optimum solutions for large-sized multimedia content and are not appropriate for the restricted resources of mobile devices and clouds. The High-Efficiency Video Coding (HEVC) is the latest and modern video codec standard introduced for efficiently storing and streaming of high-resolution videos with suitable size and higher quality. In this paper, a novel hybrid cryptosystem combining DNA (Deoxyribonucleic Acid) sequences, Arnold chaotic map, and Mandelbrot sets is suggested for secure streaming of compressed HEVC streams. Firstly, the high-resolution videos are encoded using the H.265/HEVC codec to achieve efficient compression performance. Subsequently, the suggested Arnold chaotic map ciphering process is employed individually on three channels (Y, U, and V) of the compressed HEVC frame. Then, the DNA encoding sequences are established on the primary encrypted frames resulted from the previous chaotic ciphering process. After that, a modified Mandelbrot set-based conditional shift process is presented to effectively introduce confusion features on the Y, U, and V channels of the resulted ciphered frames. Massive simulation results and security analysis are performed to substantiate that the suggested HEVC cryptosystem reveals astonishing robustness and security accomplishment in contrast to the literature cryptosystems.

Progressive Spatial Recurrent Neural Network for Intra Prediction

Intra prediction is an important component of modern video codecs, which is able to efficiently squeeze out the spatial redundancy in video frames. With preceding pixels as the context, traditional intra prediction schemes generate linear predictions based on several predefined directions (i.e. modes) for blocks to be encoded. However, these modes are relatively simple and their predictions may fail when facing blocks with complex textures, which leads to additional bits encoding the residue. In this paper, we design a Progressive Spatial Recurrent Neural Network (PS-RNN) that learns to conduct intra prediction. Specifically, our PS-RNN consists of three spatial recurrent units and progressively generates predictions by passing information along from preceding contents to blocks to be encoded. To make our network generate predictions considering both distortion and bit-rate, we propose to use Sum of Absolute Transformed Difference (SATD) as the loss function to train PS-RNN since SATD is able to measure rate-distortion cost of encoding a residue block. Moreover, our method supports variable-block-size for intra prediction, which is more practical in real coding conditions. The proposed intra prediction scheme achieves on average 2.5% bit-rate reduction on variable-block-size settings under the same reconstruction quality compared with HEVC.

Effects of Video Encoding on Camera-Based Heart Rate Estimation.

Public databases are important for evaluating and comparing different methods and algorithms for camera-based heart rate estimation. Because uncompressed video requires huge file sizes, a need for compression algorithms exists to store and share video data. Due to the optimization of modern video codecs for human perception, video compression can influence heart rate estimation negatively by reducing or eliminating small color changes of the skin (PPG) that are needed for camera based heart rate estimation. In this paper, we contribute a comprehensive analysis to answer the question of how to compress video without compromising PPG information. To analyze the influence of video compression, we compare the effect of several encoding parameters: two modern encoders (H264, H265), compression rate, resolution changes using different scaling algorithms, color subsampling, and file size on two publicly available datasets. We show that increasing the compression rate decreases the accuracy of heart rate estimation, but that resolution can be reduced (up to a cutoff point) and color subsampling can be applied for reducing file size without a big impact on heart rate estimation. From the results, we derive and propose guidelines for the recording and encoding of video data for camera-based heart rate estimation. The paper can sensitize the research community toward the problems of video encoding, and the proposed recommended practices can help with conducting future experiments and creating valuable datasets that can be shared publicly. Such datasets would improve comparability and reproducibility in the research field.

Video Classification With CNNs: Using the Codec as a Spatio-Temporal Activity Sensor

We investigate video classification via a two-stream convolutional neural network (CNN) design that directly ingests information extracted from compressed video bitstreams. Our approach begins with the observation that all modern video codecs divide the input frames into macroblocks (MBs). We demonstrate that selective access to MB motion vector (MV) information within compressed video bitstreams can also provide for selective, motion-adaptive, MB pixel decoding (a.k.a., MB texture decoding). This in turn allows for the derivation of spatio-temporal video activity regions at extremely high speed in comparison to conventional full-frame decoding followed by optical flow estimation. In order to evaluate the accuracy of a video classification framework based on such activity data, we independently train two CNN architectures on MB texture and MV correspondences and then fuse their scores to derive the final classification of each test video. Evaluation on two standard data sets shows that the proposed approach is competitive with the best two-stream video classification approaches found in the literature. At the same time: 1) a CPU-based realization of our MV extraction is over 977 times faster than GPU-based optical flow methods; 2) selective decoding is up to 12 times faster than full-frame decoding; and 3) our proposed spatial and temporal CNNs perform inference at 5 to 49 times lower cloud computing cost than the fastest methods from the literature.

Texture Segmentation Based Video Compression Using Convolutional Neural Networks

There has been a growing interest in using different approaches to improve the coding efficiency of modern video codec in recent years as demand for web-based video consumption increases. In this paper, we propose a model-based approach that uses texture analysis/synthesis to reconstruct blocks in texture regions of a video to achieve potential coding gains using the AV1 codec developed by the Alliance for Open Media (AOM). The proposed method uses convolutional neural networks to extract texture regions in a frame, which are then reconstructed using a global motion model. Our preliminary results show an increase in coding efficiency while maintaining satisfactory visual quality.

Motion compensation in video compression using hexagonal blocks

Modern video codecs map the frame area with square blocks to analyse inter-frame movement. This paper proposes a motion compensation algorithm based on a hexagonal form of blocks for video compression. The set partitioning in hierarchical trees algorithm based on the discrete wavelet transform has been used for residual image compression obtained after the motion compensation procedure. Our experiments showed that the usage of a hexagonal form of blocks for motion compensation has an average improvement of about 0.2 dB in terms of peak signal-to-noise ratio of the processed video sequence, the final productivity being even higher for the proposed algorithm as compared with the conventional square form of blocks.

Color-Aware Packet Marking Based on H.264AVC Coder Extensions for Improved Video Quality

Abstract In recent years there is a noticeable trend to implement the video transmission systems based on shared IP networks. At the same time new generations of video codecs such as H.264 are used in industrial installations. This situation forces the need for consideration of methods for efficient video transmission in industrial networks such as surveillance, identification and control systems. The first part of the article discusses the features of modern video codecs, relevant to the streaming applications. Attention is focused on the extensions of the H.264 standard that increase the error-resilience, particularly Data Partitioning (DP) and Flexible Macroblock Ordering (FMO). Next, the principles of prioritization of the video traffic based on the DiffServ architecture is discussed. In this context, separated section presents in detail the rules for packets marking which enable appropriate forwarding the video data. This information is referenced to current recommendations and technical standards. Next the performance of several classical packet marking algorithms and their possible modifications using FMO- and DP-based errorresilience configurations of H.264 are verified in simulations.

Tampering Detection in Compressed Digital Video Using Watermarking

This paper presents a method to detect video tampering and distinguish it from common video processing operations, such as recompression, noise, and brightness increase, using a practical watermarking scheme for real-time authentication of digital video. In our method, the watermark signals represent the macroblock's and frame's indices, and are embedded into the nonzero quantized discrete cosine transform value of blocks, mostly the last nonzero values, enabling our method to detect spatial, temporal, and spatiotemporal tampering. Our method can be easily configured to adjust transparency, robustness, and capacity of the system according to the specific application at hand. In addition, our method takes advantage of content-based cryptography and increases the security of the system. While our method can be applied to any modern video codec, including the recently released high-efficiency video coding standard, we have implemented and evaluated it using the H.264/AVC codec, and we have shown that compared with the existing similar methods, which also embed extra bits inside video frames, our method causes significantly smaller video distortion, leading to a PSNR degradation of about 0.88 dB and structural similarity index decrease of 0.0090 with only 0.05% increase in bitrate, and with the bit correct rate of 0.71 to 0.88 after H.264/AVC recompression.

LOW COMPLEXITY VIDEO CODING FOR SENSOR NETWORK

Modern video codecs such as H.264/AVC give state-of-the-art compression performance. However, extensive use of optimization tools makes them highly complex and hence not suitable for wireless video sensor network. In this paper an efficient video codec with substantially reduced complexity is proposed. Simulation result shows that the proposed video codec gives comparable compression performance compared to H.264/AVC but at substantially reduced computational complexity.

On Rate Distortion Optimization Using SSIM

In this paper, we present a method for performing rate-distortion optimization (RDO) using a perceptual visual quality metric, the structural similarity index (SSIM), as the target of optimization. Rate-distortion optimization is widely used in modern video codecs to make various encoder decisions to optimize the rate-distortion tradeoff. Typically, the distortion measure used is either sum-of-square error or sum-of-absolute distance, both of which are convenient when used in the RDO framework but not always reflective of a perceptual visual quality. We show that SSIM can be used as the distortion metric in the RDO framework in a simple, yet effective, manner by scaling the Lagrange multiplier used in RDO based on the local variance in that region. The experimental results on the H.264/AVC reference software show that compared to traditional RDO approaches, for the same SSIM score, the proposed approach can achieve an average rate reduction of about 9% and 14% for random access and low-delay encoding configurations. At the same time, there is no significant change in the encoding runtime.

Low Cost Design of a Hybrid Architecture of Integer Inverse DCT for H.264, VC‐1, AVS, and HEVC

The paper presents a unified hybrid architecture to compute the 8 × 8 integer inverse discrete cosine transform (IDCT) of multiple modern video codecs—AVS, H.264/AVC, VC‐1, and HEVC (under development). Based on the symmetric structure of the matrices and the similarity in matrix operation, we develop a generalized “decompose and share” algorithm to compute the 8 × 8 IDCT. The algorithm is later applied to four video standards. The hardware‐share approach ensures the maximum circuit reuse during the computation. The architecture is designed with only adders and shifters to reduce the hardware cost significantly. The design is implemented on FPGA and later synthesized in CMOS 0.18 um technology. The results meet the requirements of advanced video coding applications.

Sub-pixel motion estimation using kernel methods

Modern video codecs such as MPEG2, MPEG4-ASP and H.264 depend on sub-pixel motion estimation to optimise rate-distortion efficiency. Sub-pixel motion estimation is implemented within these standards using interpolated values at 1/2 or 1/4 pixel accuracy. By using these interpolated values, the residual energy for each predicted macroblock is reduced. However this leads to a significant increase in complexity at the encoder, especially for H.264, where the cost of an exhaustive set of macroblock segmentations needs to be estimated for optimal mode selection. This paper presents a novel scheme for sub-pixel motion estimation based on the whole-pixel SAD distribution. Both half-pixel and quarter-pixel searches are guided by a model-free estimation of the SAD surface using a two dimensional kernel method. While giving an equivalent rate distortion performance, this approach approximately halves the number of quarter-pixel search positions giving an overall speed up of approximately 10% compared to the EPZS quarter-pixel method (the state of the art H.264 optimised sub-pixel motion estimator).

Video quality and system resources: Scheduling two opponents

In this article we present three key ideas which together form a flexible framework for maximizing user-perceived quality under given resources with modern video codecs (H.264). First, we present a method to predict resource usage for video decoding online. For this, we develop and discuss a video decoder model using key metadata from the video stream. Second, we explain a light-weight method for providing replacement content for a given region of a frame. We use this method for online adaptation. Third, we select a metric modeled after human image perception which we extend to quantify the consequences of available online adaptation decisions. Together, these three parts allow us, to the best of our knowledge for the first time, to maximize user-perceived quality in video playback under given resource constraints.

Interpolation Free Subpixel Accuracy Motion Estimation

Subpixel motion estimation plays an important role in compression efficiency within modern video codecs such as MPEG2, MPEG4, and H.264. Subpixel motion estimation is implemented within these standards using interpolated values at 1/2 or 1/4 pixel accuracy. Such interpolation gives a good reduction in residual energy for each predicted macroblock and, therefore, improves compression. However, this leads to a significant increase in computational complexity at the encoder. This is especially true for H.264 where the cost of an exhaustive set of macroblock segmentations need to be estimated in order to obtain an optimal mode for prediction. This paper presents a novel interpolation-free scheme for subpixel motion estimation using the result of the full pixel sum of absolute difference distribution of each motion compensated block applied to an H.264 encoder. This system produces reduced complexity motion estimation with a controllable tradeoff between compression performance and encoder speed. These methods facilitate the generation of a real time software H.264 encoder