Improved Video Compression Research Articles

A Novel Information Hiding Approach using Selective Quantization Technique in Video Coding

This research examines the different areas of information hiding in current and emerging video compression standards. In the subsequent sections, we provide a detailed comparison of these techniques based on partition modes, prediction units, transform coding, and syntax elements. It shows the engineer and the reader that none of the methods are perfect but are the best for selected applications. We also consider the new video coding standards that have recently appeared, H.266/Versatile Video Coding (VVC) and H.265/High-Efficiency Video Coding (HEVC) and stress the fact that information hiding is critical in attaining such high compression efficacy. To facilitate the reader's understanding of all the relative information, the table that provides the analysis of each technique is presented in the form of a simple listing containing information about each technique's advantages, disadvantages, impacts, and practical applications. The current resource is intended to assist researchers and practitioners in optimizing information hiding for improved video compression. The study's outcome can contribute to enhancing knowledge of information hiding and the new developments of information hiding in video compression beyond what current research offers now, as well as provide a foundation for fresh advances in the field. Further, it is introduced to selective quantization techniques as the approach to information hiding. This method also minimizes this distortion while putting the information into the compressed stream. Finally, we evaluate the performance of this introduced approach towards information hiding capacity and maintaining video quality, with the potential to inspire further research and development in the field.

A fast and efficient data reuse scheme for HEVC Integer Motion Estimation hardware architecture

ABSTRACT High-Efficiency Video Coding (HEVC) is a video compression standard designed to improve video compression efficiency compared to its predecessor, Advanced Video Coding (AVC). HEVC provides approximately twice the compression efficiency compared to the AVC. In the HEVC system, the most computational complexity module is the Motion Estimation (ME). ME module helps reduce redundancy by compensating for motion during video compression. However, the computational complexity makes it a bottleneck in the design of high-resolution video encoders. This paper proposes an efficient architecture for the Integer ME (IME) module of the HEVC encoder. The proposed architecture introduces an efficient memory usage scheme to support the Full Search motion search algorithm. The full pipeline architecture includes 4096 Processing Elements (PE) and an adder tree to compute the Sum of Absolute Difference (SAD) for every Prediction Unit (PU) partition. The proposed architecture was designed and implemented on Xilinx Virtex-7 XC7VX550T FPGA. Our design achieved up to 275 FPS and approximately 10 FPS at 4 K video for the Search Regions of 32 × 16 and 128 × 128 pixels, respectively.

Fast CU Partition Algorithm for Intra Frame Coding Based on Joint Texture Classification and CNN.

High-efficiency video coding (HEVC/H.265) is one of the most widely used video coding standards. HEVC introduces a quad-tree coding unit (CU) partition structure to improve video compression efficiency. The determination of the optimal CU partition is achieved through the brute-force search rate-distortion optimization method, which may result in high encoding complexity and hardware implementation challenges. To address this problem, this paper proposes a method that combines convolutional neural networks (CNN) with joint texture recognition to reduce encoding complexity. First, a classification decision method based on the global and local texture features of the CU is proposed, efficiently dividing the CU into smooth and complex texture regions. Second, for the CUs in smooth texture regions, the partition is determined by terminating early. For the CUs in complex texture regions, a proposed CNN is used for predictive partitioning, thus avoiding the traditional recursive approach. Finally, combined with texture classification, the proposed CNN achieves a good balance between the coding complexity and the coding performance. The experimental results demonstrate that the proposed algorithm reduces computational complexity by 61.23%, while only increasing BD-BR by 1.86% and decreasing BD-PSNR by just 0.09 dB.

Txt2Vid: Ultra-Low Bitrate Compression of Talking-Head Videos via Text

Video represents the majority of internet traffic today, driving a continual race between the generation of higher quality content, transmission of larger file sizes, and the development of network infrastructure. In addition, the recent COVID-19 pandemic fueled a surge in the use of video conferencing tools. Since videos take up considerable bandwidth ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\sim 100$ </tex-math></inline-formula> Kbps to a few Mbps), improved video compression can have a substantial impact on network performance for live and pre-recorded content, providing broader access to multimedia content worldwide. We present a novel video compression pipeline, called Txt2Vid, which dramatically reduces data transmission rates by compressing webcam videos (“talking-head videos”) to a text transcript. The text is transmitted and decoded into a realistic reconstruction of the original video using recent advances in deep learning based voice cloning and lip syncing models. Our generative pipeline achieves two to three orders of magnitude reduction in the bitrate as compared to the standard audio-video codecs (encoders-decoders), while maintaining equivalent Quality-of-Experience based on a subjective evaluation by users ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n=242$ </tex-math></inline-formula> ) in an online study. The Txt2Vid framework opens up the potential for creating novel applications such as enabling audio-video communication during poor internet connectivity, or in remote terrains with limited bandwidth. The code for this work is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/tpulkit/txt2vid.git</uri> .

Fast CU Decision Algorithm Based on Texture Complexity and CNN for VVC

Versatile Video Coding (VVC) introduces many advanced video coding techniques. These advanced video coding techniques not only improve video compression efficiency and video quality, but also greatly increase coding time and computational complexity. Therefore, this paper uses a fast CU partitioning decision algorithm based on texture complexity and convolutional neural networks (CNN). First, we performed statistics and analysis of CU segmentation patterns for videos in the standard training set, then processed large blocks of CU based on texture complexity. To realize the purpose of fast partitioning, we design different CNN models for CU with different segmentation modes. In the design of CNN model, we use the symmetric convolutional kernel and asymmetric convolutional kernel to extract features in different directions effectively. In the loss function, we used the cross-entropy function to train the CNN model to improve the accuracy of the model. Finally, a double threshold is set in the candidate list to achieve a compromise between coding performance and coding complexity. Experimental results show that, compared to the VTM10.0 anchoring algorithm, our fast scheme, in terms of encoding time, decreases by 55.90% and BDBR increases by 1.79%; our moderate scheme, in terms of encoding time, decreases by 47.90%. BDBR increases by only 1.29%.

Fast CU Partition for VVC Using Texture Complexity Classification Convolutional Neural Network

Versatile video coding (H.266/VVC), which was newly released by the Joint Video Exploration Team (JVET), introduces quad-tree plus multi-type tree (QTMT) partition structure on the basis of quad-tree (QT) partition structure in High Efficiency Video Coding (H.265/HEVC). More complicated coding unit (CU) partitioning processes in H.266/VVC significantly improve video compression efficiency, but greatly increase the computational complexity compared. The ultra-high encoding complexity has obstructed its real-time applications. In order to solve this problem, a CU partition algorithm using convolutional neural network (CNN) is proposed in this paper to speed up the H.266/VVC CU partition process. Firstly, 64 × 64 CU is divided into smooth texture CU, mildly complex texture CU and complex texture CU according to the CU texture characteristics. Second, CU texture complexity classification convolutional neural network (CUTCC-CNN) is proposed to classify CUs. Finally, according to the classification results, the encoder is guided to skip different RDO search process. And optimal CU partition results will be determined. Experimental results show that the proposed method reduces the average coding time by 32.2% with only 0.55% BD-BR loss compared with VTM 10.2.

Load Balancing Strategies for Slice-Based Parallel Versions of JEM Video Encoder

The proportion of video traffic on the internet is expected to reach 82% by 2022, mainly due to the increasing number of consumers and the emergence of new video formats with more demanding features (depth, resolution, multiview, 360, etc.). Efforts are therefore being made to constantly improve video compression standards to minimize the necessary bandwidth while retaining high video quality levels. In this context, the Joint Collaborative Team on Video Coding has been analyzing new video coding technologies to improve the compression efficiency with respect to the HEVC video coding standard. A software package known as the Joint Exploration Test Model has been proposed to implement and evaluate new video coding tools. In this work, we present parallel versions of the JEM encoder that are particularly suited for shared memory platforms, and can significantly reduce its huge computational complexity. The proposed parallel algorithms are shown to achieve high levels of parallel efficiency. In particular, in the All Intra coding mode, the best of our proposed parallel versions achieves an average efficiency value of 93.4%. They also had high levels of scalability, as shown by the inclusion of an automatic load balancing mechanism.

Overview of the Versatile Video Coding (VVC) Standard and its Applications

Versatile Video Coding (VVC) was finalized in July 2020 as the most recent international video coding standard. It was developed by the Joint Video Experts Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG) to serve an ever-growing need for improved video compression as well as to support a wider variety of today’s media content and emerging applications. This paper provides an overview of the novel technical features for new applications and the core compression technologies for achieving significant bit rate reductions in the neighborhood of 50% over its predecessor for equal video quality, the High Efficiency Video Coding (HEVC) standard, and 75% over the currently most-used format, the Advanced Video Coding (AVC) standard. It is explained how these new features in VVC provide greater versatility for applications. Highlighted applications include video with resolutions beyond standard- and high-definition, video with high dynamic range and wide color gamut, adaptive streaming with resolution changes, computer-generated and screen-captured video, ultralow-delay streaming, 360° immersive video, and multilayer coding e.g., for scalability. Furthermore, early implementations are presented to show that the new VVC standard is implementable and ready for real-world deployment.

Fast CU Decision-Making Algorithm Based on DenseNet Network for VVC

The joint video expert team (JVET) is currently developing a new video coding standard called H.266/Versatile Video Coding (VVC). Compared with High Efficiency Video Coding (HEVC), VVC has added a variety of coding tools. These tools have greatly improved video compression efficiency and maintained a high level video quality. However, due to the increase in computational complexity, the encoding time is much longer than HEVC. We propose a prediction tool based on DenseNet (a convolutional neural network) to decrease the VVC coding complexity. We predict the probability that the edge of $4 \times 4$ blocks in each $64 \times 64$ block is the division boundary by Convolutional Neural Networks (CNN). Then, we skip the unnecessary rate distortion optimization (RDO) and speed up the coding by probability vectors in advance. The proposed method can reduce the coding complexity of 46.10% in VTM10.0 intra coding, while Bjontegaard delta bit rate (BDBR) only increases by 1.86%. In the sequence with a resolution greater than 1080P, the acceleration efficiency can be at 64.81%, the BDBR loss only increased by 1.92%.

Object-based digital hologram segmentation and motion compensation.

Digital video holography faces two main problems: 1) computer-generation of holograms is computationally very costly, even more when dynamic content is considered; 2) the transmission of many high-resolution holograms requires large bandwidths. Motion compensation algorithms leverage temporal redundancies and can be used to address both issues by predicting future frames from preceding ones. Unfortunately, existing holographic motion compensation methods can only model uniform motions of entire 3D scenes. We address this limitation by proposing both a segmentation scheme for multi-object holograms based on Gabor masks and derive a Gabor mask-based multi-object motion compensation (GMMC) method for the compensation of independently moving objects within a single hologram. The utilized Gabor masks are defined in 4D space-frequency domain (also known as time-frequency domain or optical phase-space). GMMC can segment holograms containing an arbitrary number of mutually occluding objects by means of a coarse triangulation of the scene as side information. We demonstrate high segmentation quality (down to ≤ 0.01% normalized mean-squared error) with Gabor masks for scenes with spatial occlusions. The support of holographic motion compensation for arbitrary multi-object scenes can enable faster generation or improved video compression rates for dynamic digital holography.

Large-scale video compression: recent advances and challenges

The evolution of social network and multimedia technologies encourage more and more people to generate and upload visual information, which leads to the generation of large-scale video data. Therefore, preeminent compression technologies are highly desired to facilitate the storage and transmission of these tremendous video data for a wide variety of applications. In this paper, a systematic review of the recent advances for large-scale video compression (LSVC) is presented. Specifically, fast video coding algorithms and effective models to improve video compression efficiency are introduced in detail, since coding complexity and compression efficiency are two important factors to evaluate video coding approaches. Finally, the challenges and future research trends for LSVC are discussed.

Context-aware block-based motion estimation algorithm for multimedia internet of things (IoT) platform

Shaping video data into fast-responding transmission and high resolution output video using cost-effective video processing is desirable in many applications including Internet of Things (IoT) applications. In association with rapid development of IoT smart sensor applications, real-time processing of huge-amount of data for a video signal has become necessary leading to video compression technology. Motion estimation (ME) is necessary for improving the quality, but it has high computational complexity in video compression system. The present article, therefore, proposes a context-aware adaptive pattern-based ME algorithm for multimedia IoT platform to improve video compression. In the proposed algorithm, the motions are classified into large or small based on distortion value. Accordingly, the search pattern is chosen either small diamond search pattern (SDSP) or large diamond search pattern (LDSP) in each and every step of ME; allowing adaptive processing of large and small abstract information. Compared to conventional fast algorithms, the experimental results demonstrate up to 40 and 36% reduction in encoding time for low-delay main (LB-main) and random access main (RA-main) profiles respectively in HEVC test model 16.10 encoder with bit-rate loss of 0.071 and 0.246% for both the profiles, ensuring quality video and searching precision.

Reduced Complexity Superresolution for Low-Bitrate Video Compression

Evolving video applications impose requirements for high image quality, low bitrate, and/or small computational cost. This paper combines state-of-the-art coding and superresolution (SR) techniques to improve video compression both in terms of coding efficiency and complexity. The proposed approach improves a generic decimation–quantization compression scheme by introducing low complexity single-image SR techniques for rescaling the data at the decoder side and by jointly exploring/optimizing the downsampling/upsampling processes. The enhanced scheme achieves improvement of the quality and system’s complexity compared with conventional codecs and can be easily modified to meet various diverse requirements, such as effectively supporting any off-the-shelf video codec, for instance H.264/Advanced Video Coding or High Efficiency Video Coding. Our approach builds on studying the generic scheme’s parameterization with common rescaling techniques to achieve 2.4-dB peak signal-to-noise ratio (PSNR) quality improvement at low-bitrates compared with the conventional codecs and proposes a novel SR algorithm to advance the critical bitrate at the level of 10 Mb/s. The evaluation of the SR algorithm includes the comparison of its performance to other image rescaling solutions of the literature. The results show quality improvement by 5-dB PSNR over straightforward interpolation techniques and computational time reduction by three orders of magnitude when compared with the highly involved methods of the field. Therefore, our algorithm proves to be most suitable for use in reduced complexity downsampled compression schemes.

Video Quality Evaluation Methodology and Verification Testing of HEVC Compression Performance

The High Efficiency Video Coding (HEVC) standard (ITU-T H.265 and ISO/IEC 23008-2) has been developed with the main goal of providing significantly improved video compression compared with its predecessors. In order to evaluate this goal, verification tests were conducted by the Joint Collaborative Team on Video Coding of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29. This paper presents the subjective and objective results of a verification test in which the performance of the new standard is compared with its highly successful predecessor, the Advanced Video Coding (AVC) video compression standard (ITU-T H.264 and ISO/IEC 14496-10). The test used video sequences with resolutions ranging from 480p up to ultra-high definition, encoded at various quality levels using the HEVC Main profile and the AVC High profile. In order to provide a clear evaluation, this paper also discusses various aspects for the analysis of the test results. The tests showed that bit rate savings of 59% on average can be achieved by HEVC for the same perceived video quality, which is higher than a bit rate saving of 44% demonstrated with the PSNR objective quality metric. However, it has been shown that the bit rates required to achieve good quality of compressed content, as well as the bit rate savings relative to AVC, are highly dependent on the characteristics of the tested content.

Improved Video Compression Efficiency Through Flexible Unit Representation and Corresponding Extension of Coding Tools

This paper proposes a novel video compression scheme based on a highly flexible hierarchy of unit representation which includes three block concepts: coding unit (CU), prediction unit (PU), and transform unit (TU). This separation of the block structure into three different concepts allows each to be optimized according to its role; the CU is a macroblock-like unit which supports region splitting in a manner similar to a conventional quadtree, the PU supports nonsquare motion partition shapes for motion compensation, while the TU allows the transform size to be defined independently from the PU. Several other coding tools are extended to arbitrary unit size to maintain consistency with the proposed design, e.g., transform size is extended up to 64 × 64 and intraprediction is designed to support an arbitrary number of angles for variable block sizes. Other novel techniques such as a new noncascading interpolation Alter design allowing arbitrary motion accuracy and a leaky prediction technique using both open-loop and closed-loop predictors are also introduced. The video codec described in this paper was a candidate in the competitive phase of the high-efficiency video coding (HEVC) standardization work. Compared to H.264/AVC, it demonstrated bit rate reductions of around 40% based on objective measures and around 60% based on subjective testing with 1080 p sequences. It has been partially adopted into the first standardization model of the collaborative phase of the HEVC effort.

Linear Rate Control and Optimum Statistical Multiplexing for H.264 Video Broadcast

The H.264 video coding standard achieves significantly improved video compression efficiency and finds important applications in digital video broadcast. To enable H.264 video encoding for digital TV broadcast and maximize its broadcast efficiency, there are two important issues that need to be adequately addressed. First, we need to understand the complex coding mechanism of an H.264 video encoder and develop a model to analyze and control its rate-distortion (R-D) behavior in an accurate and robust manner. Second, the R-D behaviors of individual channels in the broadcast system should be jointly controlled and optimized under bandwidth and buffer constraints so as to maximize the overall broadcast quality. In this paper, we develop a linear rate model and a linear rate control scheme for H.264 video coding. We develop an optimum statistical multiplexing system to allocate bits across video programs (each being encoded by an H.264 encoder) and video frames so that the overall video broadcast quality is maximized. We study the bandwidth and buffer constraints in video broadcast and formulate the optimum statistical multiplexing into a constrained mathematical optimization problem. Realizing that it is impossible to find a close-form solution for global optima, we propose a simple yet efficient algorithm to find a near-optimum solution for joint rate allocation under buffer constraints. Our extensive simulation results demonstrate that the proposed statistical multiplexing system achieves about 40-50% saving in bandwidth, provides a smooth video quality change across programs and frames, and maintains robust decoder buffer control.

Optimal Post-Process/In-Loop Filtering for Improved Video Compression Performance

This paper presents a novel optimal filtering approach that can be integrated into video compression schemes to improve video compression performance. In the proposed approach the encoder computes the coefficients of a linear filter in an optimal way, so as to minimize the squared error between the original frame and the filtered reconstructed frame. The encoder then multiplexes the filter coefficients into the bit-stream for decoder access. Reconstructed image frames are filtered at the decoder using optimal filter coefficients to obtain improved image frames. The optimal filtering approach is evaluated in the form of a post-process filter as well as in the form of an in-loop filter. It is shown using H.263+ and H.264/AVC that the proposed approach improves video compression performance of standard compression schemes.

Real time repeated video sequence identification

In this paper, we present a real time system for detecting repeated video clips from a live video source such as news broadcasts. Our system utilizes customized temporal video segmentation techniques to automatically partition the digital video signal into semantically sensible shots and scenes. As each frame of the video source is processed, we extract auxiliary information to facilitate repeated sequence detection. When the video transition marking the end of the shot/scene is detected, we are able to rapidly locate all previous occurrences of the video clip. Our objective is to use repeated sequence information in our multimedia content analysis application to deduce semantic relationships among shots/scenes in the input video. Our real time video processing techniques are independent of source and domain and can be applied to other applications such as commercial detection and improved video compression.