High Definition Video Sequences Research Articles

Efficient VVC Intra Prediction Based on Deep Feature Fusion and Probability Estimation

The ever-growing multimedia traffic has underscored the importance of effective multimedia codecs. Among them, the up-to-date lossy video coding standard, Versatile Video Coding (VVC), has been attracting attentions of video coding community. However, the gain of VVC is achieved at the cost of significant encoding complexity, which brings the need to realize fast encoder with comparable Rate Distortion (RD) performance. In this paper, we propose to optimize the VVC complexity at intra-frame prediction, with a two-stage framework of deep feature fusion and probability estimation. At the first stage, we employ the deep convolutional network to extract the spatial-temporal neighboring coding features. Then we fuse all reference features obtained by different convolutional kernels to determine an optimal intra coding depth. At the second stage, we employ a probability-based model and the spatial-temporal coherence to select the candidate partition modes within the optimal coding depth. Finally, these selected depths and partitions are executed whilst unnecessary computations are excluded. Experimental results on standard database demonstrate the superiority of proposed method, especially for High Definition (HD) and Ultra-HD (UHD) video sequences.

A high throughput hardware architecture for deblocking filter in HEVC

This paper presents a high throughput hardware architecture for deblocking filter in high efficiency video coding (H.265/HEVC) standard. The architecture uses an efficient hybrid pipelining and parallel processing techniques for intra encoder. A single edge filter is designed to process both horizontal and vertical filtering of pixels one after the other respectively. In our proposed architecture, the video frame is divided into 32×32 blocks and each block is processed by splitting them into blocks of 8×32 pixels in a pipelined manner. Parallel processing is employed for filtering the edges which helped in improving the throughput by decreasing the processing clock cycles. It has been observed that the largest coding tree unit block that the hardware architecture can process is 64 × 64 pixels and can be achieved in 64 clock cycles. Synthesis results of the Verilog design for the proposed architecture using application specific integrated circuit 180 nm standard cell library shows that it consumes 102k 2-input NAND gates and can work at a maximum clock frequency of 250 MHz. The proposed design is capable of supporting 8k ultra high definition video sequences at 322 Frames Per Second (fps) which is the best among the existing present-day architectures.

Low-complexity QTMT partition based on deep neural network for Versatile Video Coding

Versatile Video Coding (VVC), the newest standard for future video coding, is currently under development. This proposal aimed to improve the encoder performance over the latest standard namely High Efficiency Video Coding, carried with a high increase in coding complexity. The VVC partition structure is mainly based on the quadtree with nested multi-type tree (QTMT) block scheme. Such an improvement leads to a more flexible block partition and promotes a high encoding efficiency, but generates a huge coding complexity. In order to deal with this issue, a fast QTMT intra partition algorithm, based on a deep neural network named Early Terminated Hierarchical Convolution Neural Network, is applied to predict the $$64\times $$ 64 block QT partition structure. The proposed algorithm determines the QTMT partition structure based on the decision of whether to split or skip the corresponding CU, in order to get $$128\times $$ 128 Coding Tree Unit partition architecture. In this paper, the proposed intra partition work achieves a significant speedup in encoding gain that reaches 32.96% in best cases for Ultra High Definition video sequences compared to the reference VVC software VTM-3.0. For all video sequences, 24.49% time saving is reached on average. This improvement comes with an increase of 4.18% and a decrease of 0.18 dB in terms of BDBR and BDPSNR, respectively.

Analysis of Deep Rain Streaks Removal Convolutional Neural Network-Based Post-Processing Techniques in HEVC Encoder

This paper presents Deep Rain Streaks Removal Convolutional Neural Network (Derain SRCNN) based post-processing optimization algorithm for High-Efficiency Video Coder (HEVC). Earlier, the CNN-based denoising optimization algorithm faced overfitting issues and large convergence time when training the CNN for rain streaks affected High Definition (HD) video sequences. To address these problems, Deep rain streaks removal CNN-based post-processing block is introduced in HEVC encoder. Derain SRCNN architecture consists of a parallel two residual block layer and Dual Channel Rectification Linear Unit (DCReLU) activation function with various sizes of the convolutional layer. By reducing the validate error and training the error of CNN, the overfitting issue is solved. Also, convergence time is reduced using proper learning rate and kernel weight of optimization algorithm. The proposed network provides a higher bit rate reduction and higher convergence speed for corrupted high-definition video sequences. The experiment result shows that proposed DerainSRCNN-based post-processing filtering method achieves 6.8% and 4.1% -bit rate reduction for random access (RA) and low delay [Formula: see text] frame (LDP) configuration, respectively.

Investigation of filtering of rain streaks affected video sequences under various quantisation parameter in HEVC encoder using an enhanced V‐BM4D algorithm

This study proposes an enhanced video block matching four-dimensional (V-BM4D) denoising algorithm for highly correlated rain streaks pattern video sequence in high-efficiency video coding (HEVC) encoder. Removal of structural rain streaks from high definition (HD) video sequences are challenging issues in real-time video application. Temporal information of original video sequences is distorted among the successive video frames due to the presence of natural degradation parameters such as fog, smog and rain streaks. Rain streaks are highly correlated structural noise pattern, which affects the nature of video frame compared to other natural degradations. Existing HEVC encoder is employed with in-loop filtering to protect the temporal information of original video sequences from rain streaks pattern. However, in-loop filtering is unable to remove the highly directional oriented rain pattern from video frames. To retain temporal information among the successive video frames, enhanced block matching denoising block is adopted in HEVC coder. The proposed enhanced VBM4D- based in-loop filtering eliminates the various level of rain streaks from the HD video sequences by temporal information prediction in 4D platform. Experimental results demonstrate that the proposed denoising algorithm yields better peak signal to noise ratio value and provides better bit rate saving for various HEVC configuration.

Improved Low-Bitrate HEVC Video Coding Using Deep Learning Based Super-Resolution and Adaptive Block Patching

Good-quality video coding for low-bitrate applications is essential for narrow bandwidth transmission and limited capacity storage. In this paper, we propose an adaptive downsampling-based coding model to improve the low-bitrate compression efficiency of high-efficiency video coding (HEVC). At the encoder, the video sequence is adaptively divided into key frames (KFs) and nonkey frames (NKFs), which are encoded at the original resolution and at a reduced resolution, respectively. At the decoder, a super-resolution method based on deep learning and gradient transformation is used to upscale the NKFs. To improve the quality of NKFs without additional information during decoding, we use motion estimation to find the most similar blocks between the upscaled NKFs and the associated high-resolution KFs. Then, an adaptive patching-based method is used to warp the low-quality NKF blocks with the high-quality KF blocks. Experimental results indicate that for standard high-definition test video sequences, the maximum improvement in the peak signal-to-noise ratio can reach 3.54 dB, and the critical bitrate can reach 9.89 Mb/s at a low bitrate when compared to HEVC. These results demonstrate significant improvements compared to existing methods.

Quality improvement and performance analysis of High Efficiency Video Coding under high quantization parameters and rain streaks

High Efficiency Video Coding standard is an advanced and widely used video coding standard in recent high-definition (HD) video application. This video codec faces many challenges during the video transmission over lower bandwidth channel. One of the main challenges is the noise removal from HD video sequences. For noise removal, this video codec has in-loop filters specifically. Although these loop filters are efficient, they are not effective in removing rain pattern from the video sequences. To overcome this problem, a denoising filter has to be added to the loop filters. For denoising, an optimized multi-delay block frequency domain (OMBFD) adaptive filter is presented in this paper. Based on the intensity of raindrop, it is identified from the successive video frames. Then, these predicted raindrops are removed using the existing loop filters. Nevertheless, still, noises present in the frame are removed using the proposed OMBFD filter. To control the process of filtering, the exponential forgetting factor which is used in the proposed denoising filter is optimized using firefly optimization algorithm. Simulation results show the proposed denoising algorithm affords promising performance improvement compared to other denoising filters.

Context-Based Fractional Sample Refinement for HEVC Compliant Encoding

The H.265/MPEG-H High Efficiency Video Coding compliant encoding process faces the challenge of high computational complexity. Particularly, in the case of inter-picture prediction, most of the computational resources are allocated for the motion estimation (ME) process. In turn, ME and motion compensation enable improving coding efficiency by addressing the blocks of video frames as corresponding displacements from one or more reference blocks. These displacements do not necessarily have to be limited to integer sample positions, but may have an accuracy of half sample or quarter sample positions, which are identified during fractional sample refinement. In this paper, a context-based scheme for fractional sample refinement is proposed. The scheme takes the advantage of already obtained information in prior ME steps and provides significant flexibility in terms of parameterization. In this way, it adaptively achieves a desired tradeoff between computational complexity and coding efficiency. According to the experimental results obtained for an example algorithm utilizing the proposed framework, a significant decrease in the number of search points can be achieved. For instance, considering only 6 instead of 16 fractional sample positions results in a tradeoff of only 0.4% Bjontegaard Delta-rate loss for high-definition video sequences compared with the conventional interpolation-and-search method.

Scalable Approximate DCT Architectures for Efficient HEVC-Compliant Video Coding

An approximate kernel for the discrete cosine transform (DCT) of length 4 is derived from the 4-point DCT defined by the High Efficiency Video Coding (HEVC) standard and used for the computation of DCT and inverse DCT (IDCT) of power-of-two lengths. There are two reasons for considering the DCT of length 4 as the basic module. First, it allows computation of DCTs of lengths 4, 8, 16, and 32 prescribed by the HEVC. Second, the DCTs generated by the 4-point DCT not only involve lower complexity, but also offer better compression performance. Fully parallel and area-constrained architectures for the proposed approximate DCT are proposed to have flexible tradeoff between the area and time complexities. In addition, a reconfigurable architecture is proposed where an 8-point DCT can be used in place of a pair of 4-point DCTs. Using the same reconfiguration scheme, a 32-point DCT could be configured for parallel computation of two 16-point DCTs or four 8-point DCTs or eight 4-point DCTs. The proposed reconfigurable design can support real-time coding for high-definition video sequences in the 8k ultrahigh-definition television format ( $7680\times 4320$ at 30 frames/s). A unified forward and inverse transform architecture is also proposed where the hardware complexity is reduced by sharing hardware between the DCT and IDCT computations. The proposed approximation has nearly the same arithmetic complexity and hardware requirement as those of recently proposed related methods, but involves significantly less error energy and offers better peak signal-to-noise ratio than the others when DCTs of length more than 8 are used. A detailed comparison of the complexity, energy efficiency, and compression performance of different DCT approximation schemes for video coding is also presented. It is shown that the proposed approximation provides a better compressed-image quality than other approximate DCTs. The proposed method can perform HEVC-compliant video coding with marginal degradation of video quality and a slight increase the in bit rate, with a fraction of computational complexity of the latter.

Real-time selective video encryption based on the chaos system in scalable HEVC extension

In this paper we propose a real-time selective video encryption solution in the scalable extension of High Efficiency Video Coding (HEVC) standard, referred to as SHVC. The proposed scheme encrypts a set of sensitive SHVC parameters with a minimum delay and complexity overheads. The encryption process is performed at the CABAC binstring level and fulfills both constant bitrate and format compliant video encryption requirements. In addition, it preserves all SHVC functionalities, including bitstream extraction for mid-network adaptation and error resilience.We compare the performance of three selective SHVC encryption schemes: the first scheme encrypts only the lowest SHVC layer, the second encrypts all layers and the last scheme encrypts only the highest layer. The performance of the proposed schemes is assessed over different video encryption criteria, at different scalability configurations and various High Definition (HD) video sequences. Experimental results showed that encrypt only the lowest layer or all layers enables a high security level, while encrypting only the highest layer leads to a perceptual encryption solution, by slightly decreasing the highest layer quality. Moreover, the processing complexity of the proposed solution is assessed in the context of a real-time SHVC decoder. The complexity overhead remains low and does not exceed 6% of the real-time decoding of SHVC video sequences.

Weak Classifier for Density Estimation in Eye Localization and Tracking.

In this paper, we propose a fast weak classifier that can detect and track eyes in video sequences. The approach relies on a least-squares detector based on the inner product detector (IPD) that can stimate a probability density distribution for a feature's location-which fits naturally with a Bayesian estimation cycle, such as a Kalman or particle filter. As a least-squares sliding window detector, it possesses tolerance to small variations in the desired pattern while maintaining good generalization capabilities and computational efficiency. We propose two approaches to integrating the IPD with a particle filter tracker. We use the BioID, FERET, LFPW, and COFW public datasets as well as five manually annotated high-definition video sequences to quantitatively evaluate the algorithms' performance. The video data set contains four subjects, different types of backgrounds, blurring due to fast motion, and occlusions. All code and data are available.

GHEVC: An Efficient HEVC Decoder for Graphics Processing Units

The high compression efficiency that is provided by the high efficiency video coding (HEVC) standard comes at the cost of a significant increase of the computational load at the decoder. Such an increased burden is a limiting factor to accomplish real-time decoding, specially for high definition video sequences (e.g., Ultra HD 4K). In this scenario, a highly parallel HEVC decoder for the state-of-the-art graphics processor units (GPUs) is presented, i.e., GHEVC. Contrasting to our previous contributions, the data-parallel GHEVC decoder integrates the whole decompression pipeline (except for the entropy decoding), both for intra- and interframes. Furthermore, its processing efficiency was highly optimized by keeping the decompressed frames in the GPU memory for subsequent inter frame prediction. The proposed GHEVC decoder is fully compliant with the HEVC standard, where explicit synchronization points ensure the correct HEVC module execution order. Moreover, the GPU-based HEVC decoder is experimentally evaluated for different GPU devices, an extensive range of recommended HEVC configurations and video sequences, where an average frame rate of 145, 318, and 605 frames per second for Ultra HD 4K, WQXGA, and Full HD, respectively, was obtained in the Random Access configuration with the NVIDIA GeForce GTX TITAN X GPU.

A Bit-Plane Decomposition Matrix-Based VLSI Integer Transform Architecture for HEVC

In this brief, a new very-large-scale integrated (VLSI) integer transform architecture is proposed for the High Efficiency Video Coding (HEVC) encoder. The architecture is designed based on the signed bit-plane transform (SBT) matrices, which are derived from the bit-plane decompositions of the integer transform matrices in HEVC. Mathematically, an integer transform matrix can be equally expressed by the binary weighted sum of several SBT matrices that are only composed of binary 0 or ±1. The SBT matrices are very simple and have lower bit width than the original integer transform in the form. The SBT matrices are also sparse and there are many zero elements. The sparse characteristic of SBT matrices is very helpful for saving the addition operators of SBT. In the proposed architecture, instead of the original integer transform in high bit width, the video data can be respectively transformed with the SBT matrices in lower bit width. As a result, the delay of the transform unit circuit can be significantly reduced with the proposed SBT. Moreover, exploiting the redundant element characteristic of SBT matrices, in which the elements are 0 or ±1, the adder reuse strategy is proposed for our transform architecture, which can save the circuit area efficiently. The simulation results show that by employing the proposed strategies the VLSI transform architecture can be synthesized in a proper area with a high working frequency and low latency. The architecture can support all HEVC encoders coding ultra high-definition video sequences in real time.

New adaptive filters as perceptual preprocessing for rate-quality performance optimization of video coding

In this paper, we introduce two perceptual filters as pre-processing techniques to reduce the bitrate of compressed high-definition (HD) video sequences at constant visual quality. The goal of these perceptual filters is to remove spurious noise and insignificant details from the original video prior to encoding. The proposed perceptual filters rely on two novel adaptive filters (called BilAWA and TBil) which combine the good properties of the bilateral and Adaptive Weighting Average (AWA) filters. The bilateral and AWA filters being initially dedicated to denoising, the behaviour of the proposed BilAWA and TBil adaptive filters is first analyzed in the context of noise removal on HD test images. The first set of experimental results demonstrates their effectiveness in terms of noise removal while preserving image sharpness. A just noticeable distortion (JND) model is then introduced in the novel BilAWA and TBil filters to adaptively control the strength of the filtering process, taking into account the human visual sensitivity to signal distortion. Visual details which cannot be perceived are smoothed, hence saving bitrate without compromising perceived quality. A thorough experimental analysis of the perceptual JND-guided filters is conducted when using these filters as a pre-processing step prior to MPEG-4/AVC encoding. Psychovisual evaluation tests show that the proposed BilAWA pre-processing filter leads to an average bitrate saving of about 19.3% (up to 28.7%) for the same perceived visual quality. The proposed new pre-filtering approach has been also tested with the new state-of-the-art HEVC standard and has given similar efficiency in terms of bitrate savings for constant visual quality.

Quality comparison of the HEVC and VP9 encoders performance

This paper reports a comparison between two recent video codecs, namely the HEVC and the VP9, using High Definition Video Sequences encoded with different bit rates. A subjective test for the evaluation of the provided Quality of Experience is reported. The video sequences were shown to a panel of subjects on a High Definition LED display and the subjective tests were performed using a Single Stimulus Methodology. The results shown that the HEVC encoder provides a better visual quality on low bit rates than the VP9. Similar performance was obtained for visually lossless conditions, although the HEVC requires lower bit rates to reach that level. Moreover, the correlation of the subjective evaluation and three tested objective metrics (PSNR, SSIM, and FSIM) revealed a good representation of the subjective results, particularly the SSIM and the FSIM metrics.

Adaptive Scheduling Framework for Real-Time Video Encoding on Heterogeneous Systems

To challenge real-time encoding of high-definition video sequences on heterogeneous desktop systems, a collaborative central processing units (CPU) + graphics processing unit (GPU) framework for interloop video encoding is proposed herein. The proposed framework considers the overall complexity of the collaborative interloop encoding as a unified optimization problem. Several functional blocks are integrated for simultaneous execution control, automatic data access management, performance characterization, and adaptive scheduling and load balancing. These blocks aim at fully exploiting the performance of heterogeneous devices, asymmetric bandwidth of communication links, and several levels of concurrency between computation and communication. To support a wide range of CPU and GPU architectures, a specific encoding library is developed with highly optimized algorithms for all interloop modules. The experimental results show that the proposed framework allows achieving a real-time encoding of full high-definition sequences in several CPU + GPU systems. It also delivers performance improvements of up to 61.2% over the state-of-the-art solution, while outperforming individual GPU and quad-core CPU executions by more than 2 and 5 times, respectively.

A novel Cooperative Motion Estimation Algorithm based on Particle Swarm Optimization and its multicore implementation

A cooperative motion estimation (ME) scheme using a modified Particle Swarm Optimization (PSO) algorithm is presented. The proposed algorithm is based on a multi-swarm PSO model where a swarm of PSO particles is defined for each macroblock (MB) in the frame. Motion estimation is then performed in a cooperative manner concurrently for all the MBs in the frame. Cooperation between neighboring MBs during the motion estimation process is allowed through a communication step to exchange information about the motion vectors found so far in the estimation process. This synergic relationship between the swarms of adjacent MBs allows refining the motion search and leads to both a faster convergence of the PSO process and an improvement in the resulting motion vectors. Several techniques are also proposed to improve the search capacity and computational complexity of the PSO iterations. A novel PSO initialization scheme that exploits the existing temporal correlation is proposed to remove dependency between adjacent MBs. A fitness function history preservation mechanism is also presented to prevent redundant repeated calculations of the fitness function of a given search point by the PSO particles which dramatically decreases the computational complexity. Moreover, the maximum allowed velocity of the particles is adaptively varied during the PSO iterative process which provides a balance between search exploration and exploitation. The proposed scheme exhibits a high level of data parallelism since it is capable of performing motion estimation for all the MBs of the frame in parallel rather than serially. As a result, the presented algorithm is amenable to parallel processing techniques. In this paper, a multicore implementation of our proposed algorithm is performed using the MATLAB® Parallel Computing Toolbox™ (PCT). Extensive simulations are performed to analyze the performance of the presented algorithm. It is found that the presented scheme provides improvements in terms of accuracy and computational complexity as compared to conventional fast motion estimation techniques and two state-of-the-art PSO-based ME schemes. An analysis of the parallel performance shows that the presented scheme is highly scalable and that the parallel efficiency increases with the increase in video resolution. The multicore implementation of the proposed algorithm using MATLAB could achieve a speedup of 6.21 on eight CPU cores for high-definition (HD) video sequences. The multicore performance of the proposed scheme is also compared with existing parallel algorithms in the literature and is shown to give superior results.

MCL-V: A streaming video quality assessment database

A high-definition video quality assessment (VQA) database that captures two typical video distortion types in video services (namely, “compression” and “compression followed by scaling”) is presented in this work. The VQA database, called MCL-V, contains 12 source video clips and 96 distorted video clips with subjective assessment scores. The source video clips are selected from a large pool of public-domain high-definition (HD) video sequences with representative and diversified contents. Both distortion types are perceptually adjusted to yield distinguishable distortion levels. An improved pairwise comparison method is adopted for subjective evaluation to save evaluation time. Several existing image and video quality assessment (IQA and VQA) algorithms are evaluated against the MCL-V database. The MCL-V database is publicly accessible in the link – http://mcl.usc.edu/mcl-v-database/ to facilitate future video quality assessment research of the community.

No-reference hybrid video quality assessment based on partial least squares regression

Constructing objective assessment model on visual quality is challenging, since it associates closely with many factors in human visual perception, as well as both source coding and transmission. In this paper, a no reference hybrid model for video quality assessment is proposed by employing Partial Least Squares Regression (PLSR). The hybrid model combines both bitstream-based features and network-based features, taking into video quality dependence on visual content and network conditions account. We have conducted subjective tests to validate its quality assessment accuracy. Simulation results in Network Simulator version 2 (NS-2) verify the performance advantage of the hybrid model for SD (standard definition) and HD (high definition) video sequences. Proposed model could be used as a quality monitoring tool for assessing the real-time video quality during wireless transmission. Simulation results show that the devised model can improve the quality prediction against a full-reference metric, and achieves a high correlation (0.955) between quality prediction and the actual perceived quality judged by subjective assessment.

Scalable row-based parallel H.264 decoder on embedded multicore processors

Multimedia applications are present in most mobile hand-held devices, which are still equipped with limited battery resources. The H.264 standard is currently dominating the video compression world. H.264 has high computational requirements in terms of memory, energy, and time. Many techniques emerged that optimize parallel task granularity on multicore systems ranging from groups of pictures until the smallest block of pixels. A scalable parallel technique for the motion compensation phase is proposed in this research that is based on processing of groups of macroblock rows. Moreover, a light dependency detection algorithm is added to the prediction phase that enables parallel execution and minimizes synchronization stall time. Furthermore, a parallel implementation of the deblocking filter is also implemented. The overall result is an efficient and highly scalable parallel H.264 decoder that is evaluated on a real-board platform composed of an ARM Cortex-A9 MPCore with four processors. Various low- and high-definition video sequences are used in experiments. Results show that execution time reaches a speedup of 3.3 $$\times $$ for motion compensation stage and an overall speedup of 2.3 $$\times $$ on 4 cores including communication and synchronization overhead. Energy consumption decreases up to 63 % for the whole application execution.