Video Coding Research Articles

Learning-Based Early Transform Skip Mode Decision for VVC Screen Content Coding

One of the design goals of the recently published international video coding standard, Versatile Video Coding (VVC/H.266), is efficient coding of computer-generated video content (commonly referred to as screen content) which exhibits different signal characteristics from the usual camera-captured video (commonly referred as natural content). VVC can perform transform in multiple different ways including skipping the transform itself, which demands much computation for its best selection among many combinatory options. In this paper, we investigate designing a machine-learning-based early transform skip mode decision (ML-TSM) which makes a determination whether or not to skip the transform in an early stage by making a simple classification employing key features designed in such a way to reflect the characteristics of TSM blocks well. Compared with the VVC reference software 14.0, the proposed scheme is verified to reduce computational complexity by 11% and 4% with a Bjontegaard delta bitrate (BDBR) increase of 0.34% and 0.23% respectively under all-intra (AI) and random-access (RA) configurations.

Neural Network Based Rate Control for Versatile Video Coding

In this work, we propose a neural network based rate control algorithm for Versatile Video Coding (VVC). The proposed method relies on the modeling of the Rate-Quantization (R-Q) and Distortion-Quantization (D-Q) relationships in a data driven manner based upon the characteristics of prediction residuals. In particular, a pre-analysis framework is adopted, in an effort to obtain the prediction residuals which govern the Rate-Distortion (R-D) behaviors. By inferring from the prediction residuals with deep neural networks, the Coding Tree Unit (CTU) level R-Q and D-Q model parameters are derived, which could efficiently guide the optimal bit allocation. Subsequently, the coding parameters, including Quantization Parameter (QP) and λ, at both frame and CTU levels, are obtained according to allocated bit-rates. We implement the proposed rate control algorithm on VVC Test Model (VTM-13.0). Experimental results exhibit that the proposed rate control algorithm achieves 0.77% BD-Rate savings under Low Delay B (LDB) configurations when compared to the default rate control algorithm used in VTM-13.0. For Random Access (RA) configurations, 1.77% BD-Rate savings can be observed. Furthermore, with better bit-rate estimation, more stable buffer status can be observed, further demonstrating the advantages of the proposed rate control method.

6.3 Early Motor and Communication Indicators of ASD and ADHD and Prediction of Later Skills Based on Home Video Coding at 12 Months

6.3 Early Motor and Communication Indicators of ASD and ADHD and Prediction of Later Skills Based on Home Video Coding at 12 Months

Per-title and per-segment CRF estimation using DNNs for quality-based video coding

Per-title and per-segment CRF estimation using DNNs for quality-based video coding

Heuristic-Based Algorithms for Low-Complexity AV1 Intraprediction

Modern video encoders, such as the state-of-the-art AOMedia Video 1 (AV1) and Versatile Video Coding (VVC), offer significant compression gains at the cost of an increase in computational effort. This is partially because of the novel intra prediction techniques, including an increased number of prediction modes and block sizes supported by the codecs. To reduce this computational effort, this work proposes two heuristic-based algorithms for AV1 intra prediction: a fast mode decision that reduces the number of intra candidates with a low-cost direction detection method, and a mode-adaptive subsampling algorithm that significantly reduces the number of Sum of Squared Errors (SSE) operations after prediction. On average, a notable encoding time reduction of 29.8% was achieved, at the cost of a compression efficiency loss of 1.28%.

An efficient geometric partitioning mode by adaptive blending method for screen content coding

AbstractGeometric partitioning mode (GPM) was newly adopted in Versatile Video Coding (VVC) to increase the partitioning precision of the arbitrary boundaries of actual objects. Note that the blending process of the GPM is not suitable for screen content with various sharp and strong edges. To alleviate this drawback, three new GPM matrices are introduced, and an efficient GPM is proposed to utilize four different GPM matrices and implicitly determine one of them depending on the discontinuity and gradients of the two predictors. The proposed method achieves an average BD‐rate reduction of 1.2% and 1.6% without additional computational complexity for random access and low‐delay B configurations, respectively, over the VVC test model (VTM18.0).

High Performance VLSI Architecture for DC and Planar Modes of Intra Prediction in HEVC

Abstract: High Efficiency Video Coding (HEVC) achieves improved compression efficiency, but it introduces higher computational complexity due to intricate partitioning and increased angular modes in intra prediction. In this study, we propose a hardware architecture for handling the direct current (DC) and planar modes of intra prediction in the HEVC standard. To evaluate our proposed architecture, we performed synthesis using both TSMC 180nm and TSMC 90nm technologies. We later conducted physical implementation using Physical Cell 90nm technology. The results show significant improvements when moving from TSMC 180nm to TSMC 90nm technology, with the chip area reducing by approximately three times and power consumption decreasing by over six times. After completing the physical layout phase, we obtained a chip area of 112.19 mm2 and a power consumption of 3.88 mW. Comparing the synthesis results with the physical design phase, we observed a slight increase in chip area by around 1.5 times, while the power consumption decreased by approximately 0.4 times. The proposed architecture achieves a throughput of 27 pixels per clock cycle and supports a block size of 16x16, with the potential for further extension

Review: Dual Method Cryptography and Steganography in Video Frames in IoT

The Internet of things (IoT) is one of those emerging technologies, which are going to rule the world in the next few decades. The IoT environment not only enables Human to Machine interaction but also fosters Machine to Machine connectivity. Numerous IoT devices have poor security and insufficient computing power, making them prime targets for hackers. The IoT environment uses lightweight cryptographic techniques to address security requirements. Another security method for IoT devices is steganography. In the contemporary Internet era, the ability to secure private information is crucial, and steganography offers this capability. Due to its great ability to conceal sensitive data, video has drawn the attention of numerous academics among all forms of digital media. The main goal of this work is to examine several methods for fusing video steganography and cryptography techniques. Additionally, a thorough investigation and evaluation of a variety of video steganography methods in both compressed and raw domains are also emphasized. The comprehensive analysis of prior material makes it easier to have in-depth knowledge while creating approaches that combine cryptography with steganography. Index Terms — IoT, Lightweight, Video, Cryptography, Stenography

Semi-generative video steganography based on digital Carden Grille

Semi-generative video steganography based on digital Carden Grille

An improved hiding information by modifying selected DWT coefficients in video steganography

An improved hiding information by modifying selected DWT coefficients in video steganography

Secret data sharing through coverless video steganography based on bit plane segmentation

Recently information hiding is acquiring significance with the rising multimedia content over the web. Coverless steganography has been perceived as quite possibly the most solid way to deal with moving classified information and has turned into a research problem area. Considering video as a transporter, coverless steganography is expanding prominently because of its greater number of features than other transporters like text and images. Usually, the existing scheme hides confidential information based on single-frame features in the video. A secret data sharing through coverless video steganography technique based on bit-plane segmentation has been proposed. After extracting frames from the video, each frame is split into multiple bit-planes using BPCS(Bit-plane complexity segmentation). The hash sequences are obtained from these bit-planes by calculating the mean values of corresponding bit-plane sub-blocks. A retrieval database is constructed, including the relation between obtained hash sequences and bit-plane features. To begin with, the sender altered the secret data into a piece stream and divided it into equivalent lengths. The related retrieval information matching the fragmented secret information is transmitted to the recipient over the insecure channel. The experimental results show that the proposed technique achieves better robustness to various attacks, larger capacity, less time cost for extracting hash sequences, and a higher concealing success rate than existing coverless video steganography techniques.

Transform-Based Feature Map Compression Method for Video Coding for Machines (VCM)

The burgeoning field of machine vision has led to the development by the Moving Picture Experts Group (MPEG) of a new type of compression technology called video coding for machines (VCM), to enhance machine recognition through video information compression. This research proposes a principal component analysis (PCA)-based compression methodology for multi-level feature maps extracted from the feature pyramid network (FPN) structure. Unlike current PCA-based studies that independently carry out PCA for each feature map, our approach employs a generalized basis matrix and mean vector derived from channel correlations by a generalized PCA process to eliminate the need for a PCA process. Further compression is achieved by amalgamating high-dimensional feature maps, capitalizing on the spatial redundancy within these multi-level feature maps. As a result, the proposed VCM encoder forgoes the PCA process, and the generalized data do not incur any compression loss. It only requires compressing the coefficients for each feature map using versatile video coding (VVC). Experimental results demonstrate superior performance by our method over all feature anchors for each machine vision task, as specified by the MPEG-VCM common test conditions, outperforming previous PCA-based feature map compression methods. Notably, it achieved an 89.3% BD-rate reduction for instance segmentation tasks.

USE OF VIDEO STEGANOGRAPHY FOR REALIZATION OF CONFIDENTIAL INFORMATION TRANSMISSION CHANNELS

Today Internet and digital media are very popular, so to prevent secret information from unwanted people, there is requirement to transmit a data more securely. Video steganogra-phy is to hide the existence of the message from unauthorized party using video as cover file and hiding data in video. Data hiding is the process of embedding information in a video without changing its perceptual quality and also keep away from knowledge of existence of message. The article investigates creation and transformation of video steganographic channels via social network – Facebook, Youtube, cloud storage – Google Photos, Google Drive, WhatsApp, Telegram, Viber and Line. Keywords: video steganography, digital steganography, confidential information.

Remote ultrasound real‐time consultation and quality control system

AbstractUltrasound examination is widely used, but there are problems such as difficulties in quality control and unbalanced distribution of ultrasound resources in primary hospitals. In this article, a novel remote ultrasound real‐time consultation and quality control system was developed. The system uses advanced information science technology, medical image acquisition equipment, audio and video coding and decoding technology, combined User Datagram Protocol (UDP) and Transmission Control Protocol (TCP) transfer protocols, server cluster design technology. It realizes remote real‐time medical consultation and quality control between the doctors in the consulation center and doctors in the ultrasound examination room. The image similarity, transmission delay, and efficiency of ultrasound treatment were tested. System test data show that the quality of ultrasound images transmitted to the consultation center is the same as the quality of the original ultrasound images, the distortion is less than 1%, and the audio and video transmission delay is less than 0.2 s. The system significantly improves the efficiency and quality of ultrasound diagnosis and treatment through its high interactivity, and can also help improve the level of ultrasound diagnosis and treatment in primary hospitals.

On Content-Aware Post-Processing: Adapting Statistically Learned Models to Dynamic Content

Learning-based post-processing methods generally produce neural models that are statistically optimal on their training datasets. These models, however, neglect intrinsic variations of local video content and may fail to process unseen content. To address this issue, this article proposes a content-aware approach for the post-processing of compressed videos. We develop a backbone network, calledBackboneFormer, where a Fast Transformer using Separable Self-Attention, Spatial Attention, and Channel Attention is devised to support underlying feature embedding and aggregation. Furthermore, we introduce Meta-learning to strengthen BackboneFormer for better performance. Specifically, we propose Meta Post-Processing (Meta-PP) which leverages the Meta-learning framework to drive BackboneFormer to capture and analyze input video variations for spontaneous updating. Since the original frame is unavailable to the decoder, we devise a Compression Degradation Estimation model where a low-complexity neural model and classic operators are used collaboratively to estimate the compression distortion. The estimated distortion is then utilized to guide the BackboneFormer model for dynamic updating of weighting parameters. Experimental results demonstrate that the proposed BackboneFormer itself gains about 3.61% Bjøntegaard delta bit-rate reduction over Versatile Video Coding in the post-processing task and “BackboneFormer + Meta-PP” attains 4.32%, costing only 50K and 61K parameters, respectively. The computational complexity of MACs is 49k/pixel and 50k/pixel, which represents only about 16% of state-of-the-art methods having similar coding gains.

Efficient CU Decision Algorithm for VVC 3D Video Depth Map Using GLCM and Extra Trees

The new generation of 3D video is an international frontier research hotspot. However, the large amount of data and high complexity are core problems to be solved urgently in 3D video coding. The latest generation of video coding standard versatile video coding (VVC) adopts the quad-tree with nested multi-type tree (QTMT) partition structure, and the coding efficiency is much higher than other coding standards. However, the current research work undertaken for VVC is less for 3D video. In light of this context, we propose a fast coding unit (CU) decision algorithm based on the gray level co-occurrence matrix (GLCM) and Extra trees for the characteristics of the depth map in 3D video. In the first stage, we introduce an edge detection algorithm using GLCM to classify the CU in the depth map into smooth and complex edge blocks based on the extracted features. Subsequently, the extracted features from the CUs, classified as complex edge blocks in the first stage, are fed into the constructed Extra trees model to make a fast decision on the partition type of that CU and avoid calculating unnecessary rate-distortion cost. Experimental results show that the overall algorithm can effectively reduce the coding time by 36.27–51.98%, while the Bjøntegaard delta bit rate (BDBR) is only increased by 0.24% on average which is negligible, all reflecting the superior performance of our method. Moreover, our algorithm can effectively ensure video quality while saving much encoding time compared with other algorithms.

A Fast Algorithm for VVC Intra Coding Based on the Most Probable Partition Pattern List

Compared with High-Efficiency Video Coding (HEVC), Versatile Video Coding (VVC) has more flexible division and higher compression efficiency, but it also has higher computational complexity. In order to reduce the coding complexity, a fast algorithm based on the most probable partition pattern list (MPPPL)and pixel content similarity is proposed. Firstly, the MPPPL is constructed by using the average texture complexity difference of the sub-coding unit under different partition modes. Then, the sub-block pixel mean difference is used to decide the best partition mode or shorten the MPPPL. Finally, the selection rules of the reference lines in the intra prediction process are counted and the unnecessary reference lines are skipped by using the pixel content similarity. The experimental results show that compared with VTM-13.0, the proposed algorithm can save 52.26% of the encoding time, and the BDBR (Bjontegarrd delta bit rate) only increases by 1.23%.

Fast CU Partition Algorithm for Intra Frame Coding Based on Joint Texture Classification and CNN.

High-efficiency video coding (HEVC/H.265) is one of the most widely used video coding standards. HEVC introduces a quad-tree coding unit (CU) partition structure to improve video compression efficiency. The determination of the optimal CU partition is achieved through the brute-force search rate-distortion optimization method, which may result in high encoding complexity and hardware implementation challenges. To address this problem, this paper proposes a method that combines convolutional neural networks (CNN) with joint texture recognition to reduce encoding complexity. First, a classification decision method based on the global and local texture features of the CU is proposed, efficiently dividing the CU into smooth and complex texture regions. Second, for the CUs in smooth texture regions, the partition is determined by terminating early. For the CUs in complex texture regions, a proposed CNN is used for predictive partitioning, thus avoiding the traditional recursive approach. Finally, combined with texture classification, the proposed CNN achieves a good balance between the coding complexity and the coding performance. The experimental results demonstrate that the proposed algorithm reduces computational complexity by 61.23%, while only increasing BD-BR by 1.86% and decreasing BD-PSNR by just 0.09 dB.

Using packet trimming at the edge for in-network video quality adaption

This paper describes the effects of running in-network quality adaption by trimming the packets of layered video streams at the edge. The video stream is transmitted using the BPP transport protocol, which is like UDP, but has been designed to be both amenable to trimming and to provide low-latency and high reliability. The traffic adaption uses the Packet Wash process of Big Packet Protocol (BPP) on the transmitted Scalable Video Coding (SVC) video streams as they pass through a network function which is BPP-aware and embedded at the edge. Our previous work has either demonstrated the use of Software Defined Networking (SDN) controllers to implement Packet Wash directly, or the use of a network function in the core of the network to do the same task. This paper presents our effort to deploy and evaluate such a process at the edge, highlighting the packet trimming algorithm and showing the packet trimming effects on the streams. We compare the performance of transmitting video using BPP and the Packet Wash trimming, against alternative transmission schemes, namely UDP and HTTP adaptive streaming (HAS), presenting a number of quality parameters. The results demonstrate that providing traffic engineering using in-network quality adaption using packet trimming, provides high quality at the receiver.

Least significant bit technology for hiding text data using video steganography

Least significant bit technology for hiding text data using video steganography