Video Experts Research Articles

Complexity analysis of the VVenC versus VVC encoder

The joint video experts team (JVET) has recently finalized a next-generation open-source video codec, called versatile video coding (VVC). The new standard presents a higher gain in the run time and rate compressions. Based on the reference software (VTM, VVC Test Model) of VVC, an optimized encoder was developed these last years resulting in a fast and efficient encoder, called Fraunhofer versatile video encoder (VVenC). Based on the Bjontegaard methodology and the GPROF profiling tool, this paper presents a technical complexity analysis and comparison of both VVC and VVenC. The appropriate comparisons cover the percentage taken by each block in terms of processing time and the resulting whole encoding time. The peak signal-to-noise ratio (PSNR) and bit rate between VVC and VVenC encoders based on the common test conditions manual are also analyzed and compared. The profiling results show that the VVenC encoder presents a maximum gain of runtime and bit rate of 90% and 20% respectively in classes A and D test sequences, compared to the VVC encoder.

Virtual Classroom Competences Required by Electrical Installation and Maintenance Works Trade Teachers in Technical Colleges in Yobe State, Nigeria

This study investigated the virtual classroom proficiencies needed by Electrical Installation and Maintenance Works trade teachers in Technical Colleges in Yobe State, Nigeria. Employing a descriptive survey research design, the population of the study was 69 individuals, comprising 53 teachers and 16 ICT experts from eight Technical Colleges in Yobe State, Nigeria. Due to the manageable size of the population, the entire population was included in the study, eliminating the need for sampling. Data collection utilized the "Technical Colleges Virtual Classroom Competency Questionnaire (TCVCCQ)," structured with responses on a 5-point scale. Validation of the instrument was performed by three experts from the Department of Electrical/Electronic Technology Education, Federal College of Education (Technical), Potiskum, Yobe State, resulting in a reliability index of 0.82 determined through the Cronbach Alpha technique. Mean statistics were employed to analyze data for the research questions, while hypotheses were tested at a significance level of 0.05 using a t-test. The findings revealed a lack of proficiency among teachers in utilizing participant panels and video play features within virtual classrooms. The study recommends that principals arrange in-house workshops for teachers to acquire requisite proficiency in using participant panel features, and further suggests consultation with video experts to enhance the quality of instructional videos for improved utilization of video play features within virtual classrooms.

Fast CU size decision and intra-prediction mode decision method for H.266/VVC

AbstractH.266/Versatile Video Coding (VVC) is the most recent video coding standard developed by the Joint Video Experts Team (JVET). The quad-tree with nested multi-type tree (QTMT) architecture that improves the compression performance of H.266/VVC is introduced. Moreover, H.266/VVC contains a greater number of intra-prediction modes than H.265/High Efficiency Video Coding (HEVC), totalling 67. However, these lead to extremely the coding computational complexity. To cope with the above issues, a fast intra-coding unit (CU) size decision method and a fast intra-prediction mode decision method are proposed in this paper. Specifically, the trained Support Vector Machine (SVM) classifier models are utilized for determining CU partition mode in a fast CU size decision scheme. Furthermore, the quantity of intra-prediction modes added to the RDO mode set decreases in a fast intra-prediction mode decision scheme based on the improved search step. Simulation results illustrate that the proposed overall algorithm can decrease 55.24% encoding runtime with negligible BDBR.

Video expert assessment of high quality video for Video Assistant Referee (VAR): A comparative study

The International Football Association Board decided to introduce Video Assistant Referee (VAR) in 2018. This led to the need to develop methods for quality control of the VAR-systems. This article focuses on the important aspect to evaluate the video quality. Video Quality assessment has matured in the sense that there are standardized, commercial products and established open-source solutions to measure it with objective methods. Previous research has primarily focused on the end-user quality assessment. How to assess the video in the contribution phase of the chain is less studied. The novelties of this study are two-fold: 1) The user study is specifically targeting video experts i.e., to assess the perceived quality of video professionals working with video production. 2) Six video quality models have been independently benchmarked against the user data and evaluated to show which of the models could provide the best predictions of perceived quality. The independent evaluation is important to get unbiased results as shown by the Video Quality Experts Group. An experiment was performed involving 25 video experts in which they rated the perceived quality. The video formats tested were High-Definition TV both progressive and interlaced as well as a quarters size format that was scaled down half the size in both width and height. The videos were encoded with both H.264 and Motion JPEG for the full size but only H.264 for the quarter size. Bitrates ranged from 80 Mbit/s down to 10 Mbit/s. We could see that for H.264 that the quality was overall very good but dropped somewhat for 10 Mbit/s. For Motion JPEG the quality dropped over the whole range. For the interlaced format the degradation that was based on a simple deinterlacing method did receive overall low ratings. For the quarter size three different scaling algorithms were evaluated. Lanczos performed the best and Bilinear the worst. The performance of six different video quality models were evaluated for 1080p and 1080i. The Video Quality Metric for Variable Frame Delay had the best performance for both formats, followed by Video Multimethod Assessment Fusion method and the Video Quality Metric General model.

Tools to communicate science: looking for an effective video abstract in Ecology and Environmental Sciences

Video abstracts, filmed versions of scientific written abstracts, are an exciting trend in the world of online science videos, but, to date, the classification, conception and reception of these videos still need to be explored. This study aims to identify the most and least valued features, exploring future guidelines for producing an effective video abstract. For this purpose, 30 science video experts watched 21 video abstracts and filled out a questionnaire. Content analysis showed that video abstracts in Ecology and Environmental Sciences should be short, clear, objective, creative, dynamic and informative, mixing impactful live images with animation.

Chaotic moving video quality enhancement based on deep in-loop filtering

The Joint Video Experts Team (JVET) has announced the latest generation of the Versatile Video Coding(VVC, H.266) standard. The in-loop filter in VVC inherits the De-Blocking Filter (DBF) and Sample Adaptive Offset (SAO) of High Efficiency Video Coding (HEVC, H.265), and adds the Adaptive Loop Filter (ALF) to minimize the error between the original sample and the decoded sample. However, for chaotic moving video encoding with low bitrates, serious blocking artifacts still remain after in-loop filtering due to the severe quantization distortion of texture details. To tackle this problem, this paper proposes a Convolutional Neural Network(CNN) based VVC in-loop filter for chaotic moving video encoding with low bitrates. First, a blur-aware attention network is designed to perceive the blurring effect and to restore texture details. Then, a deep in-loop filtering method is proposed based on the blur-aware network to replace the VVC in-loop filter. Finally, experimental results show that the proposed method could averagely save 8.3% of bit consumption at similar subjective quality. Meanwhile, under close bit rate consumption, the proposed method could reconstruct more texture information, thereby significantly reducing the blocking artifacts and improving the visual quality.

Fast Mode Decision Method of Multiple Weighted Bi-Predictions Using Lightweight Multilayer Perceptron in Versatile Video Coding

Versatile Video Coding (VVC), the state-of-the-art video coding standard, was developed by the Joint Video Experts Team (JVET) of ISO/IEC Moving Picture Experts Group (MPEG) and ITU-T Video Coding Experts Group (VCEG) in 2020. Although VVC can provide powerful coding performance, it requires tremendous computational complexity to determine the optimal mode decision during the encoding process. In particular, VVC adopted the bi-prediction with CU-level weight (BCW) as one of the new tools, which enhanced the coding efficiency of conventional bi-prediction by assigning different weights to the two prediction blocks in the process of inter prediction. In this study, we investigate the statistical characteristics of input features that exhibit a correlation with the BCW and define four useful types of categories to facilitate the inter prediction of VVC. With the investigated input features, a lightweight neural network with multilayer perceptron (MLP) architecture is designed to provide high accuracy and low complexity. We propose a fast BCW mode decision method with a lightweight MLP to reduce the computational complexity of the weighted multiple bi-prediction in the VVC encoder. The experimental results show that the proposed method significantly reduced the BCW encoding complexity by up to 33% with unnoticeable coding loss, compared to the VVC test model (VTM) under the random-access (RA) configuration.

Overview of Versatile Video Coding (H.266/VVC) and Its Coding Performance Analysis

Versatile Video Coding (H.266/VVC) is the newest video coding standard jointly developed by the Joint Video Experts Team (JVET), which is organized by the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG). H.266/VVC provides about 40% bitrate reduction compared with High Efficiency Video Coding (H.265/HEVC) for the same visual quality. This paper introduces in detail the core structure of H.266/VVC by highlighting its features within block partitioning structure, intra/inter prediction, transform, quantization, and in-loop filtering, compared to its predecessor (H.265/HEVC). H.266/VVC yields significantly improved the coding performance, but it increased the computational complexity, particularly for the encoder side, which remains a problem to be tackled for successful commercialization in the future. This paper examined the statistical performance of H.266/VVC coding tools from the bitstreams encoded by the VVC test model (VTM12.0) encoder through rate-distortion optimization under the JVET common test conditions. In addition, the complexity and performance analyses are conducted on the block partitioning structure and group of picture structure. It is expected that an optimized H.266/VVC encoder can be designed and developed by minimizing the coding loss based on the analysis data.

Facilitators and hindrances to farmers’ participation in production of farmer-learning videos in central Uganda: a thematic analysis

Videos can complement extension services provision by improving dissemination of agricultural information to farmers. However, for videos to be effective, they need to have relevant video content. This study assessed enablers and hindrances to farmers’ participation in FLV production based on a coffee FLV. Identifying enablers and hindrances to participation of farmers in production of an FLV is essential for developing strategies that optimise their participation to produce relevant FLVs aligned to their contexts. A qualitative case study research design was employed. Data were collected through focus group discussions, key informant interviews and in-depth interviews, and analysed through thematic analysis guided by the Activity theory analytical framework. The study established that farmers’ attributes especially their competencies to accomplish assigned tasks in FLV production, and motivations, were salient enablers to their participation. The other enablers were related to how the FLV process was organised, which included availability and adequacy of requisite resources, existence of clear guidelines for production of FLV, utilisation of pre-existing relationships, skilfulness of video experts and field officers, and specificity of roles. The limited interactions among actors, differing expectations, and time constraints were the major hindrances to farmers’ participation in production of the FLV. Hence, video experts should provide opportunities for improving farmers’ competences through training, offering technical support, and providing for rehearsals to build confidence. Mechanisms for improving interactions between and among farmers and video experts to harmonise expectations, promote mutual understanding, and foster knowledge sharing are also needed.

Machine Learning Based Fast QTMTT Partitioning Strategy for VVenC Encoder in Intra Coding

The newest video compression standard, Versatile Video Coding (VVC), was finalized in July 2020 by the Joint Video Experts Team (JVET). Its main goal is to reduce the bitrate by 50% over its predecessor video coding standard, the High Efficiency Video Coding (HEVC). Due to the new advanced tools and features included in VVC, it actually provides high coding performances—for instance, the Quad Tree with nested Multi-Type Tree (QTMTT) involved in the partitioning block. Furthermore, VVC introduces various techniques that allow for superior performance compared to HEVC, but with an increase in the computational complexity. To tackle this complexity, a fast Coding Unit partition algorithm based on machine learning for the intra configuration in VVC is proposed in this work. The proposed algorithm is formed by five binary Light Gradient Boosting Machine (LightGBM) classifiers, which can directly predict the most probable split mode for each coding unit without passing through the exhaustive process known as Rate Distortion Optimization (RDO). These LightGBM classifiers were offline trained on a large dataset; then, they were embedded on the optimized implementation of VVC known as VVenC. The results of our experiment show that our proposed approach has good trade-offs in terms of time-saving and coding efficiency. Depending on the preset chosen, our approach achieves an average time savings of 30.21% to 82.46% compared to the VVenC encoder anchor, and a Bjøntegaard Delta Bitrate (BDBR) increase of 0.67% to 3.01%, respectively.

Video quality of video professionals for Video Assisted Referee (VAR)

Changes in the footballing world’s approach to technology and innovation contributed to the decision by the International Football Association Board (IFAB) to introduce Video Assistant Referees (VAR). The change meant that under strict protocols referees could use video replays to review decisions in the event of a “clear and obvious error” or a “serious missed incident”. This led to the need by Fédération Internationale de Football Association (FIFA) to develop methods for quality control of the VAR-systems, which was done in collaboration with RISE Research Institutes of Sweden AB. One of the important aspects is the video quality. The novelty of this study is that it has performed a user study specifically targeting video experts i.e., to measure the perceived quality of video professionals working with video production as their main occupation. An experiment was performed involving 25 video experts. In addition, six video quality models have been benchmarked against the user data and evaluated to show which of the models could provide the best predictions of perceived quality for this application. Video Quality Metric for variable frame delay (VQM_VFD) had the best performance for both formats, followed by Video Multimethod Assessment Fusion (VMAF) and VQM General model.

Early CTU Termination and Three-steps Mode Decision Method for Fast Versatile Video Coding

Versatile Video Coding (VVC) has been recently becoming popular in coding videos due to its compression efficiency. To reach this performance, Joint Video Experts Team (JVET) has introduced a number of improvement techniques to VVC coding model. Among them, VVC Intra coding introduces a new concept of quad-tree nested multi-type tree (QTMT) and extends the predicted modes with up to 67 options. As a result, the complexity of the VVC Intra encoding also greatly increases. To make VVC Intra coding more feasible in real-time applications, we propose in this paper a novel deep learning based fast QTMT and an early mode prediction method. At the first stage, we use a learned convolutional neural network (CNN) model to predict the coding unit map and then fed into the VVC encoder to early terminate the block partitioning process. After that, we design a statistical model to predict a list of most probable modes (MPM) for each selected Coding using (CU) size. Finally, we employ a so-called three-steps mode decision algorithm to estimate the optimal directional mode without sacrificing the compression performance. The proposed early CU splitting and fast intra prediction are integrated into the latest VTM reference software. Experimental results show that the proposed method can save 50.2% of encoding time with a negligible BD-Rate increase.

Low complexity inter coding scheme for Versatile Video Coding (VVC)

The latest coding standard Versatile Video Coding (VVC) developed by the Joint Video Experts Team (JVET) and Video Coding Experts Group (VCEG) was finalized in 2020. By introducing several new coding techniques, VVC improves the compression efficiency by 50% compared with H.265/HEVC. However, the coding complexity increases dramatically, which obstructs it from real-time application. To tackle this issue, a fast inter coding algorithm utilizing coding information is proposed to speed up the coding process. First, by analyzing the coding areas of the neighboring CUs, we predict the coding area of the current CU to terminate unnecessary splitting modes. Then, the temporally optimal coding mode generated during the prediction process is further utilized to shrink the candidate modes to speed up the coding process. Finally, the distribution of neighboring prediction modes are exploited to measure the motion complexity of the current CU, based on which the unnecessary prediction modes can be skipped earlier. Experimental results demonstrate that the proposed method can reduce the coding complexity by 40.08% on average with 0.07 dB BDPSNR decrease and 1.56% BDBR increase, which outperforms the state-of-the-art approach.

Versatile Video Coding Standard: A Review From Coding Tools to Consumers Deployment

The amount of video content and the number of applications based on multimedia information increase each day. The development of new video coding standards is a challenge to increase the compression rate and other important features with a reasonable increase in the computational load. The Joint Video Experts Team of the ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group (MPEG) have worked together to develop the versatile video coding standard, finalized in July 2020 as the International Standard 23090-3 (MPEG-I Part 3). This article overviews some interesting consumer electronic use cases, the compression tools described in the standard, the current available real-time implementations, and the first industrial trials done with this standard.

Selective Encryption of the Versatile Video Coding Standard

versatile video coding (VVC) is the next generation video coding standard developed by the joint video experts team (JVET) and released in July 2020. VVC introduces several new coding tools providing a significant coding gain over the high efficiency video coding (HEVC) standard. It is well known that increasing the coding efficiency adds more dependencies in the video bitstream making format-compliant encryption with the standard more challenging. In this paper we tackle the problem of selective encryption of the VVC standard in format-compliant and constant bitrate. These two constraints ensure that the encrypted bitstream can be decoded by any VVC decoder while the bitrate remains unchanged by the encryption. The selective encryption of all possible VVC syntax elements is investigated. A new algorithm is proposed to encrypt in format-compliant and constant bitrate the transform coefficients (TCs) together with other syntax elements at the level of the entropy encoder. The proposed solution was integrated and assessed under the VVC reference software model version 6.0. Experimental results showed that encryption drastically decreases the video quality while the encryption is robust against several types of attacks. The encryption space is estimated in the range of 15% to 26% of the bitstream size resulting in a lightweight encryption process. The web page of this work is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://gugautie.github.io/sevvc/</uri> .

High Efficiency Intra CU Partition and Mode Decision Method for VVC

Versatile video coding (VVC/H.266) is the newest video compression standard, which is developed by the Joint Video Experts Team. Compared with previous encoding schemes, VVC achieves higher compression efficiency by introducing a new partition structure and additional intra prediction modes but results in high computational complexity. To efficiently solve this problem of redundant processing in quad-tree with nested multi-type tree structures and intra mode prediction, we propose a texture analysis&#x2013;based ternary tree (TT) and binary tree (BT) partition strategy, and a gradient-based intra mode decision method to accelerate TT and BT partition and intra mode prediction, separately. The texture complexity and prediction direction of coding unit (CU) is calculated by texture detection method. A texture analysis&#x2013;based TT and BT partition strategy is established by using the regression method based on analyzing the texture complexity of the CU. Then, a texture analysis&#x2013;based TT and BT partition strategy is applied to reduce the redundant partition for each CU. By using the prediction direction of CU, a gradient-based intra mode decision method is established for skipping the impossible modes for each CU. Experimental results revealed that the proposed method could save 49.49% in encoding time and increase the Bjontegaard delta bit rate (BDBR) by only 0.56%. It confirms that the proposed method achieved high efficiency and a good balance between the BDBR and time saving.

Motion Vector Coding and Block Merging in the Versatile Video Coding Standard

This paper overviews the motion vector coding and block merging techniques in the Versatile Video Coding (VVC) standard developed by the Joint Video Experts Team (JVET). In general, inter-prediction techniques in VVC can be classified into two major groups: “whole block-based inter prediction” and “subblock-based inter prediction”. In this paper, we focus on techniques for whole block-based inter prediction. As in its predecessor, High Efficiency Video Coding (HEVC), whole block-based inter prediction in VVC is represented by adaptive motion vector prediction (AMVP) mode or merge mode. Newly introduced features purely for AMVP mode include symmetric motion vector difference and adaptive motion vector resolution. The features purely for merge mode include pairwise average merge, merge with motion vector difference, combined inter-intra prediction and geometric partitioning mode. Coding tools such as history-based motion vector prediction and bidirectional prediction with coding unit weights can be applied on both AMVP mode and merge mode. This paper discusses the design principles and the implementation of the new inter-prediction methods. Using objective metrics, simulation results show that the methods overviewed in the paper can jointly achieve 6.2% and 4.7% BD-rate savings on average with the random access and low-delay configurations, respectively. Significant subjective picture quality improvements of some tools are also reported when comparing the resulting pictures at same bitrates.

The High-Level Syntax of the Versatile Video Coding (VVC) Standard

Versatile Video Coding (VVC), a.k.a. ITU-T H.266 | ISO/IEC 23090-3, is the new generation video coding standard that has just been finalized by the Joint Video Experts Team (JVET) of ITU-T VCEG and ISO/IEC MPEG at its <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$19^{\mathrm {th}}$ </tex-math></inline-formula> meeting ending on July 1, 2020. This paper gives an overview of the VVC high-level syntax (HLS), which forms its system and transport interface. Comparisons to the HLS designs in High Efficiency Video Coding (HEVC) and Advanced Video Coding (AVC), the previous major video coding standards, are included. When discussing new HLS features introduced into VVC or differences relative to HEVC and AVC, the reasoning behind the design differences and the benefits they bring are described. The HLS of VVC enables newer and more versatile use cases such as video region extraction, composition and merging of content from multiple coded video bitstreams, and viewport-adaptive 360° immersive media.

Subblock-Based Motion Derivation and Inter Prediction Refinement in the Versatile Video Coding Standard

Efficient representation and coding of fine-granular motion information is one of the key research areas for exploiting inter-frame correlation in video coding. Representative techniques towards this direction are affine motion compensation (AMC), decoder-side motion vector refinement (DMVR), and subblock-based temporal motion vector prediction (SbTMVP). Fine-granular motion information is derived at subblock level for all the three coding tools. In addition, the obtained inter prediction can be further refined by two optical flow-based coding tools, the bi-directional optical flow (BDOF) for bi-directional inter prediction and the prediction refinement with optical flow (PROF) exclusively used in combination with AMC. The aforementioned five coding tools have been extensively studied and finally adopted in the Versatile Video Coding (VVC) standard. This paper presents technical details of each tool and highlights the design elements with the consideration of typical hardware implementations. Following the common test conditions defined by Joint Video Experts Team (JVET) for the development of VVC, 5.7% bitrate reduction on average is achieved by the five tools. For test sequences characterized by large and complex motion, up to 13.4% bitrate reduction is observed. Additionally, visual quality improvement is demonstrated and analyzed.

Overview of the Versatile Video Coding (VVC) Standard and its Applications

Versatile Video Coding (VVC) was finalized in July 2020 as the most recent international video coding standard. It was developed by the Joint Video Experts Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG) to serve an ever-growing need for improved video compression as well as to support a wider variety of today’s media content and emerging applications. This paper provides an overview of the novel technical features for new applications and the core compression technologies for achieving significant bit rate reductions in the neighborhood of 50% over its predecessor for equal video quality, the High Efficiency Video Coding (HEVC) standard, and 75% over the currently most-used format, the Advanced Video Coding (AVC) standard. It is explained how these new features in VVC provide greater versatility for applications. Highlighted applications include video with resolutions beyond standard- and high-definition, video with high dynamic range and wide color gamut, adaptive streaming with resolution changes, computer-generated and screen-captured video, ultralow-delay streaming, 360° immersive video, and multilayer coding e.g., for scalability. Furthermore, early implementations are presented to show that the new VVC standard is implementable and ready for real-world deployment.