Video Coding Research Articles

Video Steganography with Deep Learning

Video Steganography with Deep Learning

Exploring an Alternative to Record Motor Competence Assessment: Interrater and Intrarater Audio–Video Reliability

The Canadian Agility and Movement Skill Assessment (CAMSA) is a valid and reliable circuit-based test of motor competence which can be used to assess children’s skills in a live or recorded performance and then coded. We aimed to analyze the intrarater reliability of the CAMSA scores (total, time, and skill score) and time measured, by comparing the live audio with the video assessment method. We also aimed to assess the interrater reliability using both audio- and video coding on a sample of 177 Spanish children. We found moderate-to-excellent inter- and intrarater video–audio intraclass correlation coefficients for the CAMSA score, time measured, time score, and skill score. Nonsignificant differences were found between video and audio recordings in the CAMSA score, time measured, and time score. Our findings support the rationale that different raters and scoring methods can accurately assess the participants’ motor competence level using the CAMSA Spanish version.

λ-Domain Rate Control via Wavelet-Based Residual Neural Network for VVC HDR Intra Coding.

High dynamic range (HDR) video offers a more realistic visual experience than standard dynamic range (SDR) video, while introducing new challenges to both compression and transmission. Rate control is an effective technology to overcome these challenges, and ensure optimal HDR video delivery. However, the rate control algorithm in the latest video coding standard, versatile video coding (VVC), is tailored to SDR videos, and does not produce well coding results when encoding HDR videos. To address this problem, a data-driven λ-domain rate control algorithm is proposed for VVC HDR intra frames in this paper. First, the coding characteristics of HDR intra coding are analyzed, and a piecewise R-λ model is proposed to accurately determine the correlation between the rate (R) and the Lagrange parameter λ for HDR intra frames. Then, to optimize bit allocation at the coding tree unit (CTU)-level, a wavelet-based residual neural network (WRNN) is developed to accurately predict the parameters of the piecewise R-λ model for each CTU. Third, a large-scale HDR dataset is established for training WRNN, which facilitates the applications of deep learning in HDR intra coding. Extensive experimental results show that our proposed HDR intra frame rate control algorithm achieves superior coding results than the state-of-the-art algorithms. The source code of this work will be released at https://github.com/TJU-Videocoding/WRNN.git.

Spatio-Temporal Convolutional Neural Network for Enhanced Inter Prediction in Video Coding.

This paper presents a convolutional neural network (CNN)-based enhancement to inter prediction in Versatile Video Coding (VVC). Our approach aims at improving the prediction signal of inter blocks with a residual CNN that incorporates spatial and temporal reference samples. It is motivated by the theoretical consideration that neural network-based methods have a higher degree of signal adaptivity than conventional signal processing methods and that spatially neighboring reference samples have the potential to improve the prediction signal by adapting it to the reconstructed signal in its immediate vicinity. We show that adding a polyphase decomposition stage to the CNN results in a significantly better trade-off between computational complexity and coding performance. Incorporating spatial reference samples in the inter prediction process is challenging: The fact that the input of the CNN for one block may depend on the output of the CNN for preceding blocks prohibits parallel processing. We solve this by introducing a novel signal plane that contains specifically constrained reference samples, enabling parallel decoding while maintaining a high compression efficiency. Overall, experimental results show average bit rate savings of 4.07% and 3.47% for the random access (RA) and low-delay B (LB) configurations of the JVET common test conditions, respectively.

Perception-oriented Video Coding based on Image Analysis and Completion: A Review

Perception-oriented video coding based on image analysis and completion (IAC) has received increasing attention over recent years. Several approaches have been proposed that follow diverging strategies: from full compatibility to hybrid block transform coding to alternative codec designs. In this paper, we review the most significant issues in IAC coding and their relevance for future codec designs. The most promising pathways will be analyzed along with limitations and open issues. The challenges in IACbased video coding will be summarized using methods developed by the authors and others for block-based and region-based IAC coding.

A Flexible Algorithm Design of Spatial Scalability for Real-time Surveillance Applications

The Surveillance Video and Audio Coding )SVAC( working group is currently developing the third-generation video compression standard, SVAC 3.0. To extend this standard, this paper proposes a spatial scalable coding Scalable Surveillance Video Coding )SSVC( framework, so that the SVAC3.0 video stream can gracefully adapt to different transmission bandwidth limitations and the requirements of decoding hardware, while keep the quality of the reconstructed image without degradation. In order to achieve the scalability of hardware implementation, SSVC designs a flexible reference frame marking and usage scheme, so that the enhanced layer coding does not directly depend on the basic layer, and effectively reduces the coding coupling between layers. SSVC improves the motion vector prediction method, effectively utilizes the encoding information of the base layer, and is mainly compatible with SVAC3.0 syntax structures. In addition, SSVC provides several different operational modes to adapt to various application scenarios and achieve an optimal trade-off between coding efficiency and complexity. The performance comparisons among simulcast stream and SSVC, single-layer stream and SSVC enhancement layer, as well as experimental data for different operational modes are provided. The coding efficiency and computational complexity are also analyzed.

LSSVC: A Learned Spatially Scalable Video Coding Scheme.

Traditional block-based spatially scalable video coding has been studied for over twenty years. While significant advancements have been made, the scope for further improvement in compression performance is limited. Inspired by the success of learned video coding, we propose an end-to-end learned spatially scalable video coding scheme, LSSVC, which provides a new solution for scalable video coding. In LSSVC, we propose to use the motion, texture, and latent information of the base layer (BL) as interlayer information for compressing the enhancement layer (EL). To reduce interlayer redundancy, we design three modules to leverage the upsampled interlayer information. Firstly, we design a contextual motion vector (MV) encoder-decoder, which utilizes the upsampled BL motion information to help compress high-resolution MV. Secondly, we design a hybrid temporal-layer context mining module to learn more accurate contexts from the EL temporal features and the upsampled BL texture information. Thirdly, we use the upsampled BL latent information as an interlayer prior for the entropy model to estimate more accurate probability distribution parameters for the high-resolution latents. Experimental results show that our scheme surpasses H.265/SHVC reference software by a large margin. Our code is available at https://github.com/EsakaK/LSSVC.

Video Performance Metric Assessment of Coding Standards H.264/AVC and MPEG-4

This paper presents a comprehensive video quality assessment that focuses on the comparison of two predominant video coders, H.264/AVC and MPEG-4, particularly at very low resolutions pertinent to web-based applications and security cameras. Objective quality metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) were employed to evaluate the performance of these coders. Through experimental analysis, it was observed that H.264/AVC offered superior performance over MPEG-4 in terms of both PSNR and SSIM values. This result underscores the efficiency of H.264/AVC in scenarios where high-quality video is essential, despite bandwidth or storage constraints.

A Discrete-Mapping-Based Cross-Component Prediction Paradigm for Screen Content Coding.

Cross-component prediction is an important intra-prediction tool in the modern video coders. Existing prediction methods to exploit cross-component correlation include cross-component linear model and its extension of multi-model linear model. These models are designed for camera captured content. For screen content coding, where videos exhibit different signal characteristics, a cross-component prediction model tailored to their characteristics is desirable. As a pioneering work, we propose a discrete-mapping based cross-component prediction model for screen content coding. Our model relies on the core observation that, screen content videos typically comprise of regions with a few distinct colors and luma value (almost always) uniquely conveys chroma value. Based on this, the proposed method learns a discrete-mapping function from available reconstructed luma-chroma pairs and uses this function to derive chroma prediction from the co-located luma samples. To achieve higher accuracy, a multi-filter approach is employed to derive co-located luma values. The proposed method achieves 2.61%, 3.51% and 3.92% Y, U and V bit-rate savings respectively over Enhanced Compression Model (ECM) 4.0, with negligible complexity, for text and graphics media under all-intra configuration.

Adaptive highly Secure and Imperceptible Video Steganography for Medical images via Deep Convolutional Neural Network

Adaptive highly Secure and Imperceptible Video Steganography for Medical images via Deep Convolutional Neural Network

High-security HEVC video steganography method using the motion vector prediction index and motion vector difference

High-security HEVC video steganography method using the motion vector prediction index and motion vector difference

A Semantic Communication and VVC Based Hybrid Video Coding System

A Semantic Communication and VVC Based Hybrid Video Coding System

HMFVC: A Human-Machine Friendly Video Compression Scheme

Recent studies on Video Coding for Machine (VCM) have achieved remarkable results. However, in practical visual-analytic applications, except for transmitting the visual feature for machine analysis, video textures are also mandatory for human monitoring and decision-making. To this end, this paper proposes a human-machine friendly video compression scheme (HMFVC) which can satisfy both human viewing and machine analysis well. First, we propose a learned semantic representation (LSR) method to extract semantic information between temporal neighboring frames. LSR could be utilized in signal reconstruction for human viewing and visual analysis for machine understanding. Second, given the proposed LSR, we design an end-to-end optimized video compression framework to jointly optimize the visual quality for human perception, analysis accuracy for machines, and compression efficiency as well. Finally, an HMFVC codec is developed, which can achieve higher action recognition accuracy and better reconstruction quality than the traditional codecs and learned video compression approaches. Specifically, HMFVC saves 77% bitrate to achieve the same analysis performance with the original videos compared to x265. To our knowledge, HMFVC is the first end-to-end optimized video compression scheme to serve both humans and machines. It is a promising framework for human-machine friendly video compression approaches.

A Statistical Study of Arien Sequences for an H.264 Prediction Module Lower Complexity

Abstract Advanced video coding, or H.264/AVC has been proved to possess a highly complex encoder with a block motion estimation (ME) scheme, entropy coding and a prediction module. Typical applications of unmanned aerial vehicles (UAVs) require real-time video compression. As a result, the current study aims to develop an approach to decrease the computing complexity of the prediction module. The software algorithm that was developed helps basic functions to be achieved in the prediction module. In fact, the user can set up, manage, deploy, and monitor the UAVs in real time. Evaluations obtained in this paper have shown that the adopted approach can significantly improve the execution time of coding and, consequently, the network performance. When comparing the original method of the Laboratory of Electronics and Information’s software with the suggested approach, experimental results show that the proposed method had achieved an increase in execution time from 3.544–15.574 %.

Optimizing H.266/VVC Intra Coding with a Genetic Algorithm: Balancing Speed and Quality

The growing need for high-definition video material requires improvements in video encoding systems that maximize encoding performance while simultaneously improving compression efficiency. This paper presents a novel genetic algorithm-based intra-coding optimization method for the H.266Versatile Video Coding (VVC) standard. One of the biggest problems in video compression is finding the ideal balance between encoding speed and video quality, which is what our approach aims to solve. Our suggested method makes use of the strong search capabilities of the evolutionary algorithm to choose the best Multi-Type Tree (MTT) partitions and coding tools from the wide range of possibilities present in H.266VVC. The wellness assessment work that guides this choice method combines criteria for perceptual appraisal of video quality and measures for coding productivity appraisal.

Facial Image Compression via Neural Image Manifold Compression

Although the recent learning-based image and video coding techniques achieve rapid development, the signal fidelity-driven target in these methods leads to the divergence to a highly effective and efficient coding framework for both human and machine. In this paper, we aim to address the issue by making use of the power of generative models to bridge the gap between full fidelity (for human vision) and high discrimination (for machine vision). Therefore, relying on existing pretrained generative adversarial networks (GAN), we build a GAN inversion framework that projects the image into a low-dimensional natural image manifold. In this manifold, the feature is highly discriminative and also encodes the appearance information of the image, named as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">latent code</i> . Taking a variational bit-rate constraint with a hyperprior model to model/suppress the entropy of image manifold code, our method is capable of fulfilling the needs of both machine and human visions at very low bit-rates. To improve the visual quality of image reconstruction, we further propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multiple latent codes</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">scalable inversion</i> . The former gets several latent codes in the inversion, while the latter additionally compresses and transmits a shallow compact feature to support visual reconstruction. Experimental results demonstrate the superiority of our method in both human vision tasks, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> . image reconstruction, and machine vision tasks, including semantic parsing and attribute prediction.

Scalable Face Image Coding via StyleGAN Prior: Toward Compression for Human-Machine Collaborative Vision.

The accelerated proliferation of visual content and the rapid development of machine vision technologies bring significant challenges in delivering visual data on a gigantic scale, which shall be effectively represented to satisfy both human and machine requirements. In this work, we investigate how hierarchical representations derived from the advanced generative prior facilitate constructing an efficient scalable coding paradigm for human-machine collaborative vision. Our key insight is that by exploiting the StyleGAN prior, we can learn three-layered representations encoding hierarchical semantics, which are elaborately designed into the basic, middle, and enhanced layers, supporting machine intelligence and human visual perception in a progressive fashion. With the aim of achieving efficient compression, we propose the layer-wise scalable entropy transformer to reduce the redundancy between layers. Based on the multi-task scalable rate-distortion objective, the proposed scheme is jointly optimized to achieve optimal machine analysis performance, human perception experience, and compression ratio. We validate the proposed paradigm's feasibility in face image compression. Extensive qualitative and quantitative experimental results demonstrate the superiority of the proposed paradigm over the latest compression standard Versatile Video Coding (VVC) in terms of both machine analysis as well as human perception at extremely low bitrates (< 0.01 bpp), offering new insights for human-machine collaborative compression.

A Hierarchical Multiscenario H.265/HEVC Video Encryption Scheme

With the pervasive application of video streaming, the security of streaming media information continually faces new challenges. Most existing encryption methods employ uniform criteria for encrypting all scenarios, leading to unnecessary mutual inhibition of algorithm security and coding efficiency. Furthermore, several encryption algorithms are inadequate for high-resolution videos. A novel, independently hierarchical video cryptosystem for H.265/high efficiency video coding (HEVC) is proposed that develops a scene-adaptive encryption strategy tailored for multiscenario videos. Additionally, we fully consider the seeds of pseudo-random number generators, syntax components in compressed codes, and scenario indicators. We analyze the visibility of encrypted videos in different scenarios and the encryption performance of various syntax parameters based on the integration of encryption for three distinct scenario categories to further enhance encryption efficiency and security. The method’s versatility is demonstrated using a diverse array of videos with significant and insignificant inter-frame motion information across varying resolutions. The experimental results from the video datasets indicate that our scheme effectively balances security and coding efficiency. Furthermore, the scene-adaptive approach can be tailored flexibly according to subscriber needs.

Latitude-Redundancy-Aware All-Zero Block Detection for Fast 360-Degree Video Coding.

The sphere-to-plane projection of 360-degree video introduces substantial stretched redundant data, which is discarded when reprojected to the 3D sphere for display. Consequently, encoding and transmitting such redundant data is unnecessary. Highly redundant blocks can be referred to as all-zero blocks (AZBs). Detecting these AZBs in advance can reduce computational and transmission resource consumption. However, this cannot be achieved by existing AZB detection techniques due to the unawareness of the stretching redundancy. In this paper, we first derive a latitude-adaptive redundancy detection (LARD) approach to adaptively detect coefficients carrying redundancy in transformed blocks by modeling the dependency between valid frequency range and the stretching degree based on spectrum analysis. Utilizing LARD, a latitude-redundancy-aware AZB detection scheme tailored for fast 360-degree video coding (LRAS) is proposed to accelerate the encoding process. LRAS consists of three sequential stages: latitude-adaptive AZB (L-AZB) detection, latitude-adaptive genuine-AZB (LG-AZB) detection and latitude-adaptive pseudo-AZB (LP-AZB) detection. Specifically, L-AZB refers to the AZB introduced by projection. LARD is used to detect L-AZB directly. LG-AZB refers to the AZB after hard-decision quantization and zeroing redundant coefficients. A novel latitude-adaptive sum of absolute difference estimation model is built to derive the threshold for LG-AZB detection. LP-AZB refers to the AZB in terms of rate-distortion optimization considering redundancy. A latitude-adaptive rate-distortion model is established for LP-AZB detection. Experimental results show that LRAS can achieve an average total encoding time reduction of 25.85% and 20.38% under low-delay and random access configurations compared to the original HEVC encoder, with only 0.16% and 0.13% BDBR increases and 0.01dB BDPSNR loss, respectively. The transform and quantization time savings are 60.13% and 59.94% on average.

Joint Rate-Distortion Optimization for Video Coding and Learning-Based In-Loop Filtering

Joint Rate-Distortion Optimization for Video Coding and Learning-Based In-Loop Filtering