Video Coding Research Articles

Performance analysis of multiview video compression based on MIV and VVC multilayer

AbstractTo represent immersive media providing six degree‐of‐freedom experience, moving picture experts group (MPEG) immersive video (MIV) was developed to compress multiview videos. Meanwhile, the state‐of‐the‐art versatile video coding (VVC) also supports multilayer (ML) functionality, enabling the coding of multiview videos. In this study, we designed experimental conditions to assess the performance of these two state‐of‐the‐art standards in terms of objective and subjective quality. We observe that their performances are highly dependent on the conditions of the input source, such as the camera arrangement and the ratio of input views to all views. VVC‐ML is efficient when the input source is captured by a planar camera arrangement and many input views are used. Conversely, MIV outperforms VVC‐ML when the camera arrangement is non‐planar and the ratio of input views to all views is low. In terms of the subjective quality of the synthesized view, VVC‐ML causes severe rendering artifacts such as holes when occluded regions exist among the input views, whereas MIV reconstructs the occluded regions correctly but induces rendering artifacts with rectangular shapes at low bitrates.

Pelican Whale Optimization Enabled Deep Learning Framework for Video Steganography Using Arnold Transform-Based Embedding

Steganography refers to hiding a secret message from various sources, such as images, videos, audio and so on. The advantage of steganography is to avoid data hacking in transmission medium during the transmission of information sources. Video steganography is superior to image steganography since the videos can hide a substantial quantity of secret messages more than the image. Hence, this research introduced the video stereography technique, Arnold Transform with SqueezeNet-based Pelican Whale Optimization Algorithm (AT[Formula: see text]SqueezeNet_PWOA), for concealing the secret image on the video. To hide the secret image on the video, the proposed method follows three steps: key frame and feature extraction, pixel prediction and embedding. The extraction of the key frame process is carried out by the Structural Similarity Index Measure (SSIM), and then the neighborhood features and convolutional neural network (CNN) features are extracted from the frame to improve the robustness of the embedding process. Moreover, the pixel prediction is completed by the SqueezeNet model, wherein the learning factors are tuned by the PWOA. In addition, the embedding process is completed by applying the Arnold transform on the predicted pixel, and the transformed regions are combined with the secret image using the embedding function. Likewise, the extraction process extracts the secret image from the embedded video by substituting the predicted pixel and Arnold transform on the embedded video. The proposed method is used to hide chunks of secret data in the form of video sequences and it improves the performance. The Arnold transform used in this work provides security by encrypting the data. The use of SqueezeNet makes the proposed model a simple design and this reduces the computational time. Thus, the AT[Formula: see text]SqeezeNet_PWOA attained better correlation coefficient (CC), peak signal-to-noise ratio (PSNR) and mean square error (MSE) of 0.908, 48.66 and 0.001 dB with the Gaussian noise.

A Rate Control Scheme for VVC Intercoding Using a Linear Model

Versatile video coding (VVC) aims to achieve high compression but also issues like varying content/network conditions. Existing rate control (RC) methods struggle to achieve optimal quality under these complex scenarios. This paper proposes a novel RC scheme for VVC based on a linear model. The Lagrange minimization multiplier is introduced under bit budget constraints, allowing optimized bit allocation. RC optimization is formulated as a convex solution, and is derived into the optimal quantization parameter (QP) for RC. Experimental analysis demonstrates the proposed linear model-based RC algorithm performances are better compared to other state-of-the-art methods due to their use of a linear model and optimal QP determination.

An efficient versatile video coding motion estimation hardware

Versatile Video Coding (VVC) is the latest video coding standard. It provides higher compression efficiency than the previous video coding standards at the cost of significant increase in computational complexity. Motion estimation (ME) is the most time consuming and memory intensive module in VVC encoder. Therefore, in this paper, we propose an efficient VVC ME hardware. It is the first VVC ME hardware in the literature. It has real time performance with small hardware area. This efficiency is achieved by using a 64 × 64 systolic processing element array to support maximum coding tree unit (CTU) size of 128 × 128 and by using a novel memory-based sum of absolute differences (SAD) adder tree to calculate SADs of 128 × 128 CTUs. The proposed VVC ME hardware reduces memory accesses significantly by using an efficient data reuse method. It can process up to 30 4 K (3840 × 2160) video frames per second.

Retracted: A Review of Motion Vector-Based Video Steganography

Retracted: A Review of Motion Vector-Based Video Steganography

Two-Stage Perceptual Quality Oriented Rate Control Algorithm for HEVC

As a practical technique in mainstream video coding applications, rate control dominates important to ensure compression quality with limited bitrates constraints. However, most rate control methods mainly focus on objective quality while ignoring the perceptual quality improvement for human eyes. In this paper, we propose a two-stage rate control algorithm to optimize the perceptual quality at the frame encoding stage and the coding tree unit (CTU) encoding stage for high efficiency video coding (HEVC), respectively. Firstly, for the frame encoding stage, with inter-frame distortion dependency consideration, a frame-level rate control method is presented by adjusting the frame-level Lagrange multiplier adaptively with a preprocessing method. Secondly, for the CTU encoding stage, we propose a saliency-based CTU-level perceptual quality rate control algorithm, which employs CTU-level saliency weight to adjust the perceptual rate-distortion (R-D) model. We conduct the CTU-level rate control by an optimized Lagrange multiplier and quantization parameter (QP) to achieve perceptual quality optimization. Extensive experimental results reveal that, compared with state-of-the-art rate control methods on HEVC, our algorithm achieves significant perceptual coding performance with improved subjective visual quality.

A convolutional neural network-based rate control algorithm for VVC intra coding

The Versatile Video Coding (VVC) has shown significant improvements in Rate-Distortion (R-D) performance compared to its predecessor, High Efficiency Video Coding (HEVC). However, it still encounters several challenges. One of these challenges is the efficient allocation of bits among all Coding Tree Units (CTUs). Additionally, there is a lack of prior information for intra-frame coding, particularly for the first frame. After CTU-level bit allocation, only fixed parameters can be used to determine the λ for CTUs, which does not result in optimal rate–distortion performance. To tackle above challenges, we propose a rate control solution based on Convolutional Neural Network (CNN). This approach utilizes CNN to predict the key parameters α and β in the R-D model, addressing the problem of lacking prior information in intra-frame coding. Subsequently, the predicted α and β values are used to adaptively allocate bits for each CTU. Our proposed algorithm is implemented in VTM-16.0 under Common Test Conditions (CTC). Experimental results show that, compared to the default rate control algorithm in VTM-16.0, our proposed algorithm enhances R-D performance by 0.96% while maintaining rate control accuracy.

QoE-Based Performance Comparison of AVC, HEVC, and VP9 on Mobile Devices with Additional Influencing Factors

While current video quality assessment research predominantly revolves around resolutions of 4 K and beyond, targeted at ultra high-definition (UHD) displays, effective video quality for mobile video streaming remains primarily within the range of 480 p to 1080 p. In this study, we conducted a comparative analysis of the quality of experience (QoE) for widely implemented video codecs on mobile devices, specifically Advanced Video Coding (AVC), its successor High-Efficiency Video Coding (HEVC), and Google’s VP9. Our choice of 720 p video sequences from a newly developed database, all with identical bitrates, aimed to maintain a manageable subjective assessment duration, capped at 35–40 min. To mimic real-time network conditions, we generated stimuli by streaming original video clips over a controlled emulated setup, subjecting them to eight different packet-loss scenarios. We evaluated the quality and structural similarity of the distorted video clips using objective metrics, including the Video Quality Metric (VQM), Peak Signal-to-Noise Ratio (PSNR), Video Multi-Method Assessment Fusion (VMAF), and Multi-Scale Structural Similarity Index (MS-SSIM). Subsequently, we collected subjective ratings through a custom mobile application developed for Android devices. Our findings revealed that VMAF accurately represented the degradation in video quality compared to other metrics. Moreover, in most cases, HEVC exhibited an advantage over both AVC and VP9 under low packet-loss scenarios. However, it is noteworthy that in our test cases, AVC outperformed HEVC and VP9 in scenarios with high packet loss, based on both subjective and objective assessments. Our observations further indicate that user preferences for the presented content contributed to video quality ratings, emphasizing the importance of additional factors that influence the perceived video quality of end users.

Divide-and-conquer-based RDO-free CU Partitioning for 8K Video Compression

8K (7689×4320) ultra-high definition (UHD) videos are growing popular with the improvement of human visual experience demand. Therefore, the compression of 8K UHD videos has become a top priority in the third-generation audio video coding standard (AVS3). However, as an important part of the coding standard promotion, the real-time hardware implementation for AVS3-based 8K UHD video intra coding is severely hindered, especially in the coding unit (CU) partition stage. To break through the limitation, this article proposes a divide-and-conquer-based rate-distortion-optimization-free (RDO-free) CU partitioning algorithm for efficient hardware implementation. Aimed at the complex CU partition in AVS3, we separately design a lightweight optimization for original partitioning rules to improve division efficiency and a decision tree-based RDO-free CU decision framework to eliminate the latency caused by the waiting for rate-distortion cost calculation in RDO strategy. Afterward, a divide-and-conquer-based hardware-friendly gradient difference calculating approach is devised to accelerate the learning feature extracting speed. To ensure that the proposed algorithm is sufficient to support the real-time CU partition for 8K videos, we also develop a hardware architecture based on FPGA. Experimental results illustrate that the software coding performance of our algorithm is significantly ahead of the efficient implementation uAVS3e for AVS3 and the reference software HM-16.20 for High Efficiency Video Coding (HEVC), even though there is 9.96% loss on BD-Rate Y. Considering its importance for the hardware implementation of AVS3-based 8K real-time encoder, the coding loss is acceptable. Moreover, the hardware simulation results on VU440 FPGA with Vivado 2019 show that our algorithm can support 61.12 frames per second (fps) CU partition for 8K UHD videos with only 0.00%, 0.00%, 1.01%, and 7.78% consumption of BRAM_18K, DSP48E, FF, and LUT, respectively. Additionally, with dual-path parallelism, 122.24 fps also can be implemented with controllable resource utilization, which achieves the state-of-the-art performance.

Performance evaluation of mpeg-5 part 2 (lcevc): impact of packet loss

This paper presents an evaluation of the latest MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC) for video streaming applications using best effort protocols. LCEVC is a new video standard by MPEG, which enhances any base codec through an additional low bitrate stream, improving both video compression efficiency and and transmission. However, there is an interplay between packetization, packet loss visibility, choice of codec and video quality, which implies that prior studies with other codecs may be not as relevant. The contributions of this paper is, therefore in twofold: It evaluates the compression performance of LCEVC and then the impact of packet loss on its video quality when compared to H.264 and HEVC.The results from this evaluation suggest that, regarding compression, LCEVC outperformed its base codecs, overall in terms average encoding bitrate savings when using the constant rate factor (CRF) rate control. For example at a CRF of 19, the average encoding bitrate was reduced by 18.7% and 15.8% when compared with the base H.264 and HEVC codecs respectively. Furthermore, LCEVC produced better visual quality across the packet loss range compared to its base codecs and the quality only started to decrease once packet loss exceeded 0.8-1%, and decreases at a slower pace compared to its equivalent base codecs. This suggests that the LCEVC enhancement layer also provides error concealment. The results presented in this paper will be of interest to those considering the LCEVC standard and expected video quality in error-prone environments

Fast HEVC inter-frame coding based on LSTM neural network technology

High Efficiency Video Coding (HEVC) is the most commonly used video coding standard. However, its high coding complexity is a heavy burden for real-time video applications. But, coding tools designed based on traditional coding frameworks have reached limits. Furthermore, existing low-complexity video coding methods have not thoroughly analyzed the characteristics of compressed video, making it impossible to develop targeted models to reduce coding complexity. Therefore, in this research, a fast HEVC inter-frame coding technique is proposed. Firstly, we perform a characteristics analysis of HEVC inter-frame coding to explore the correlation between video frames. Secondly, we develop an Inter-Frame Feature Transfer-Long Short-Term Memory (IFFT-LSTM) model to obtain the optimal coding tree unit (CTU) partition structure. Thirdly, we embed the IFFT-LSTM model into the HEVC test platform. The experimental results show that suggested method can effectively reduce the HEVC inter-frame coding complexity with a small amount of rate-distortion (RD) performance loss while maintaining acceptable subjective video quality.

A multi-embedding domain video steganography algorithm based on TU partitioning and intra prediction mode

With High Efficiency Video Coding (HEVC) becoming a popular video coding standard in the world, combing the HEVC with data hiding methods is a complex task. In this paper, a multi-embedded video steganography algorithm based on Intra Prediction Mode (IPM) and Transform Unit (TU) partition is proposed. By analyzing the coding characteristics of the TU partition and the IPM, we select embedded regions that are relatively independent as much as possible. Then, suitable distortion functions are designed for different embedding domains. Using the minimum distortion framework of the STC model to embed messages. The experimental results indicate that the proposed method achieves a maximum embedding capacity 1.5 times larger than that of traditional single domain embedding methods, while maintaining good visual quality.

Scalable and resilient 360-degree-video adaptive streaming over HTTP/2 against sudden network drops

Realistic Virtual Reality is supported by 360° video, which provides viewers with an immersive watching experience. However, 360°video is bulky in size, while the transmission system has limited ability to provide bandwidth. As such, intelligent adaptive delivery solutions play a crucial role in enabling users to stream high-quality 360°video. In this research, we propose a novel approach for 360-degree video streaming over HTTP/2 that can provide consumers with a good watching experience (QoE) even in varying network circumstances and head-eye movements over time. The proposed method deploys the so-called BBAG algorithm (Buffer and Bandwidth Allocation Algorithm) using Scalable Video Coding to choose appropriate tile layers to resolve the trade-off between network and user adaptivity. With the support of HTTP/2’s stream termination capability, the delivery of late tile layers is terminated to handle abrupt interruptions. By employing multiple buffer thresholds, BBAG is able to adapt bitrates to changes in users’ perspective while watching a 360-degree video. BBAG is proven to improve QoE up to about 90% by maintaining high and stable buffer level, while enhancing average viewport bitrates by about 80% compared to state-of-the-art methods in different scenarios of network bandwidths.

Research on VVC Intra-Frame Bit Allocation Scheme Based on Significance Detection

This research is based on an intra-frame rate control algorithm based on the Versatile Video Coding (VVC) standard, considering that there is the phenomenon of over-allocating the bitrate of the end coding tree units (CTUs) in the bit allocation process, while the front CTUs are not effectively compressed. Fusing a Canny-based edge detection algorithm, a color contrast-based saliency detection algorithm, a Sum of Absolute Transformed Differences (SATD) based CTU coding complexity measure, and a Partial Least Squares (PLS) regression model, this paper proposes a CTU-level bit allocation improvement scheme for intra-mode code rate control of the VVC standard. First, natural images are selected to produce a lightweight dataset. Second, different metrics are utilized to obtain the significance and complexity values of each coding unit, the relatively important coding units in the whole frame are selected, which are adjusted with different weights, and the optimal adjustment multiplicity is supplemented into the dataset. Finally, the PLS regression model was used to obtain regression equations to refine the weights for adjusting the bit allocation. The proposed bit allocation scheme improves the average rate control accuracy by 0.453%, Y-PSNR by 0.05 dB, BD-rate savings by 0.33%, and BD-PSNR by 0.03 dB compared to the VVC standard rate control algorithm.

An End-to-End Video Coding Method via Adaptive Vision Transformer

Deep learning-based video coding methods have demonstrated superior performance compared to classical video coding standards in recent years. The vast majority of the existing deep video coding (DVC) networks are based on convolutional neural networks (CNNs), and their main drawback is that since CNNs are affected by the size of the receptive field, they cannot effectively handle long-range dependencies and local detail recovery. Therefore, how to better capture and process the overall structure as well as local texture information in the video coding task is the core issue. Notably, the transformer employs a self-attention mechanism that captures dependencies between any two positions in the input sequence without being constrained by distance limitations. This is an effective solution to the problem described above. In this paper, we propose end-to-end transformer-based adaptive video coding (TAVC). First, we compress the motion vector and residuals through a compression network built on the vision transformer (ViT) and design the motion compensation network based on ViT. Second, based on the requirement of video coding to adapt to different resolution inputs, we introduce a position encoding generator (PEG) as adaptive position encoding (APE) to maintain its translation invariance across different resolution video coding tasks. The experiment shows that for multiscale structural similarity index measurement (MS-SSIM) metrics, this method exhibits significant performance gaps compared to conventional engineering codecs, such as [Formula: see text], [Formula: see text], and VTM-15.2. We also achieved a good performance improvement compared to the CNN-based DVC methods. In the case of peak signal-to-noise ratio (PSNR) evaluation metrics, TAVC also achieves good performance.

Adaptive HEVC Video Steganography With High Performance Based on Attention-Net and PU Partition Modes

With the increasing popularity of digital video, video steganography has become a hot research topic in the field of covert communication and privacy protection. The existing prediction unit (PU) based video steganography often tends to result in large bit rate increase, which is also easily noticeable to the steganography analyst. To solve this problem, an adaptive steganography for HEVC video based on attention-net and PU partition modes is proposed. First, the distortion of modified PUs is analyzed from the perspective of rate distortion optimization at the group of pictures (GOP) level, and we find that modifying PU will lead to distortion accumulation and abnormal bitrate increase. Therefore, an adaptive distortion function based on the improved rate distortion cost is designed, and the embedding distortion is minimized by using Syndrome-Trellis Code (STC) steganography coding. Meanwhile, a super-resolution convolutional neural network with non-local sparse attention-net filter is proposed to replace the in-loop filter in HEVC to reconstruct the reference frame, thereby reducing the bitrate cost and improving the visual quality of stego-video. Experimental results show that the proposed algorithm can achieve superior perceptual quality and bitrate performance comparing with the sate-of-the-art works.

Chroma Intra Prediction With Lightweight Attention-Based Neural Networks

Neural networks can be successfully used for cross-component prediction in video coding. In particular, attention-based architectures are suitable for chroma intra prediction using luma information because of their capability to model relations between different channels. However, the complexity of such methods is still very high and should be further reduced, especially for decoding. In this paper, a cost-effective attention-based neural network is designed for chroma intra prediction. Moreover, with the goal of further improving coding performance, a novel approach is introduced to utilize more boundary information effectively. In addition to improving prediction, a simplification methodology is also proposed to reduce inference complexity by simplifying convolutions. The proposed schemes are integrated into H.266/Versatile Video Coding (VVC) pipeline, and only one additional binary block-level syntax flag is introduced to indicate whether a given block makes use of the proposed method. Experimental results demonstrate that the proposed scheme achieves up to -0.46%/-2.29%/-2.17% BD-rate reduction on Y/Cb/Cr components, respectively, compared with H.266/VVC anchor. Reductions in the encoding and decoding complexity of up to 22% and 61%, respectively, are achieved by the proposed scheme with respect to the previous attention-based chroma intra prediction method while maintaining coding performance.

Recent Advances in Rate Control: From Optimization to Implementation and Beyond

Video coding is a video compression technique that compresses the original video sequence to produce a smaller archive file or reduce the transmission bandwidth under constraints on the visual quality loss. Rate control (RC) plays a critical role in video coding. It can achieve stable stream output in practical applications, especially real-time video applications such as video conferencing or game live streaming. Most RC algorithms either directly or indirectly characterise the relationship between the bit rate (R) and quantisation (Q) and then allocate bits to every coding unit so as to guarantee the global bit rate and video quality level. This paper comprehensively reviews the classic RC technologies used in international video standards of past generations, analyses the mathematical models and implementation mechanisms of various schemes, and compares the performance of recent state-of-the-art RC algorithms. Finally, we discuss future directions and new application areas for RC methods. We hope that this review can help support the development, implementation, and application of RC for new video coding standards.

Video coding using closed-loop texture analysis and synthesis

The compression of video content containing high-frequency textures using the Mean Squared Error (MSE) criterion or other similar distortion measures typically leads to very large bit rate assignments to these areas. However, we assume that the exact reproduction (in an MSE sense) is subjectively not required and that other measures and coding methods can be used leading to substantially lower bit rate assignments. One such measure is a global similarity measure and we demonstrate its suitability for assessing the distortion of such textures in the context of video coding. The above-mentioned concept is adapted to a new, generic, and fully automated video coding scheme that provides a texture analyzer at the encoder and a texture synthesizer at the decoder. The texture analyzer thereby identifies the target textures, where it is assumed that the viewer perceives the semantic meaning of the displayed texture rather than the specific details therein. The texture synthesizer regenerates an approximate version of the original texture based on corresponding meta data transmitted by the encoder. The texture itself is not coded in the sense of minimizing MSE or other similar distortion measures. The new approacch is integrated into an H.264/AVC video codec and yields bit rate reductions of up to 41% when compared at the same visual quality to a reference H.264/AVC codec.

Cross-Component Prediction Boosted With Local and Non-Local Information in Video Coding.

In the versatile video coding (VVC) standard, cross-component prediction (CCP) is introduced to utilize the correlation between luma and chroma components. To further exploit the cross-component redundancy, a bundle of novel CCP methods with more sophisticated models are studied in enhanced compression model (ECM), which is a platform targeting at the next generation video coding standard, developed by JVET. This paper presents two methods, known as local-boosting CCP (LB-CCP) and non-local CCP (NL-CCP) to improve CCP with local and non-local information. With LB-CCP, prediction samples of CCP can be filtered with neighbouring samples. Besides, neighbouring template costs are calculated to determine the range of training samples, as well as the cross-component prediction method used in the chroma fusion mode. With NL-CCP, a CCP model can be derived with samples non-adjacent to the current block. As an equivalent but simpler implementation, the CCP model can be inherited from a CCP-coded neighbouring block as a spatial CCP candidate. Moreover, the CCP model of a previous CCP-coded block can be stored in a history-based table, which can be fetched and used by the current block as a history-based CCP candidate. A NL-CCP candidate list is built with the two kinds of candidates. The encoder can select the optimal candidate and send an index to the decoder. Experimental results show that LB-CCP together with NL-CCP can provide an average Bjontegaard delta rate (BD-rate) reduction of 0.27%, 2.31%, 2.44% on Y, Cb, Cr components, respectively, with a negligible change in the running time, compared with ECM-8.0 in all intra configurations. Both LB-CCP and NL-CCP have been adopted into ECM.