Group Of Pictures Research Articles

Video coding optimization in AVS2

Chinese second generation of the Audio Video Coding Standard, known as the AVS2, competing with HEVC/H.265 and AV1, has become a well-known video compression standard. Many unique tools have been developed and incorporated in AVS2. This paper proposes several encoder optimization schemes with respect to rate-distortion optimization (RDO). Specifically, by exploiting the close relationship between the total number of referenced pictures and the hierarchy of reference structure, a reference structure determined parameter (RSDP) algorithm reallocates Lagrange multiplier. Moreover, an adaptive quantization parameter offset (AQPO) algorithm is provided at both picture-level and Groups of Pictures (GoP)-level to adjust quantization parameters adaptively on top of the preassigned ones. Both RSDP algorithm and AQPO algorithm have already been adopted by AVS2 and incorporated in its reference software. Experimental results show that in the low-delay, RSDP and AQPO achieved savings of 1.35% and 1.26% on BDBR, respectively.

Detection of HEVC double compression with non-aligned GOP structures via inter-frame quality degradation analysis

Detection of double compression is of great significance to reveal video compression history. However, efficient detection algorithm for High Efficiency Video Coding (HEVC) double compression with non-aligned Group of Pictures (GOP) structures is rarely reported. To address this issue, a novel algorithm based on inter-frame quality degradation process analysis is proposed in this paper. Firstly, a brief introduction to HEVC standard and double compression with non-aligned GOP structures is provided. Next, theoretical analysis of inter-frame quality degradation process is given. Then, from the perspective of filtering decision, the inter-frame In-Loop Filtering Decision Mode (ILFDM) feature is adopted for double compression detection. Finally, the detection framework is given, which includes feature extraction, feature combination and periodic analysis. Experiments are conducted under different coding scenarios, by comparing with existing algorithms, results demonstrate that the proposed algorithm has better performance not only in terms of double compression detection, but also in original GOP size estimation. Furthermore, the proposed scheme is proved to be satisfactory in relocated I-frame detection.

Impact of Scene Content on High Resolution Video Quality.

This paper deals with the impact of content on the perceived video quality evaluated using the subjective Absolute Category Rating (ACR) method. The assessment was conducted on eight types of video sequences with diverse content obtained from the SJTU dataset. The sequences were encoded at 5 different constant bitrates in two widely video compression standards H.264/AVC and H.265/HEVC at Full HD and Ultra HD resolutions, which means 160 annotated video sequences were created. The length of Group of Pictures (GOP) was set to half the framerate value, as is typical for video intended for transmission over a noisy communication channel. The evaluation was performed in two laboratories: one situated at the University of Zilina, and the second at the VSB—Technical University in Ostrava. The results acquired in both laboratories reached/showed a high correlation. Notwithstanding the fact that the sequences with low Spatial Information (SI) and Temporal Information (TI) values reached better Mean Opinion Score (MOS) score than the sequences with higher SI and TI values, these two parameters are not sufficient for scene description, and this domain should be the subject of further research. The evaluation results led us to the conclusion that it is unnecessary to use the H.265/HEVC codec for compression of Full HD sequences and the compression efficiency of the H.265 codec by the Ultra HD resolution reaches the compression efficiency of both codecs by the Full HD resolution. This paper also includes the recommendations for minimum bitrate thresholds at which the video sequences at both resolutions retain good and fair subjectively perceived quality.

Video quality evaluation and testing verification of H.264, HEVC, VVC and EVC video compression standards

This paper is a comparative analysis with respect to video compression, coding efficiency with respect to performance and quality of the compressed video of Advanced Video Coding (H.264), High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC) and MPEG-5 Essential Video Coding (EVC). The approach used for coding and comparative analysis is by keeping the video quality (QP) and group of pictures (GOP) constant for all the four codecs. Performance with respect to quality and coding efficiency is tabulated with respect to objective quality metrics analysis like (PSNR, M-SSIM, Bitrate, compression ratio) and subjective quality analysis. From both these techniques, it is observed that the VVC is superior to EVC, HEVC and H.264 in terms of compression. Therefore, more bitrate savings is observed for VVC which is 15% to 45% more when compared to EVC. Due to the complexity of the encoder block, VVC takes almost twice the time taken to encode EVC and HEVC taking the least time to encode the video bit streams. The conclusion of the quality of video keeping system parameters constant, showed VVC performing better than EVC, followed by HEVC and then H.264 for both objective and subjective analysis. Time taken to decode the encoded sequences is almost same with very less delta difference between the video codecs. The codec configurations with algorithmic enhancements is also explained in this paper.

Adaptive Streaming of 360 Videos With Perfect, Imperfect, and Unknown FoV Viewing Probabilities in Wireless Networks.

This paper investigates adaptive streaming of one or multiple tiled 360 videos from a multi-antenna base station (BS) to one or multiple single-antenna users, respectively, in a multi-carrier wireless system. We aim to maximize the video quality while keeping rebuffering time small via encoding rate adaptation at each group of pictures (GOP) and transmission adaptation at each (transmission) slot. To capture the impact of field-of-view (FoV) prediction, we consider three cases of FoV viewing probability distributions, i.e., perfect, imperfect, and unknown FoV viewing probability distributions, and use the average total utility, worst average total utility, and worst total utility as the respective performance metrics. In the single-user scenario, we optimize the encoding rates of the tiles, encoding rates of the FoVs, and transmission beamforming vectors for all subcarriers to maximize the total utility in each case. In the multi-user scenario, we adopt rate splitting with successive decoding and optimize the encoding rates of the tiles, encoding rates of the FoVs, rates of the common and private messages, and transmission beamforming vectors for all subcarriers to maximize the total utility in each case. Then, we separate the challenging optimization problem into multiple tractable problems in each scenario. In the single-user scenario, we obtain a globally optimal solution of each problem using transformation techniques and the Karush-Kuhn-Tucker (KKT) conditions. In the multi-user scenario, we obtain a KKT point of each problem using the concave-convex procedure (CCCP). Finally, numerical results demonstrate that the proposed solutions achieve notable gains in quality, quality variation, and rebuffering time over existing schemes in all three cases. To the best of our knowledge, this is the first work revealing the impact of FoV prediction on the performance of adaptive streaming of tiled 360 videos.

A CNN-Based Prediction-Aware Quality Enhancement Framework for VVC

This paper presents a framework for Convolutional Neural Network (CNN)-based quality enhancement task, by taking advantage of coding information in the compressed video signal. The motivation is that normative decisions made by the encoder can significantly impact the type and strength of artifacts in the decoded images. In this paper, the main focus has been put on decisions defining the prediction signal in intra and inter frames. This information has been used in the training phase as well as input to help the process of learning artifacts that are specific to each coding type. Furthermore, to retain a low memory requirement for the proposed method, one model is used for all Quantization Parameters (QPs) with a Quantization Parameter (QP)-map, which is also shared between luma and chroma components. In addition to the Post Processing (PP) approach, the In-Loop Filtering (ILF) codec integration has also been considered, where the characteristics of the Group of Pictures (GoP) are taken into account to boost the performance. The proposed CNN-based Quality Enhancement (QE) framework has been implemented on top of the Versatile Video Coding (VVC) Test Model (VTM-10). Experiments show that the prediction-aware aspect of the proposed method improves the coding efficiency gain of the default CNN-based QE method by 1.52%, in terms of BD-BR, at the same network complexity compared to the default CNN-based QE filter.

Tile-Based Joint Caching and Delivery of 360° Videos in Heterogeneous Networks

The recent surge of applications involving the use of 360° video challenges mobile networks infrastructure, as 360° video files are of significant size, and current delivery and edge caching architectures are unable to guarantee their timely delivery. In this paper, we investigate the problem of joint collaborative content-aware caching and delivery of 360° videos in a video on demand setting. The proposed scheme takes advantage of 360° video encoding in multiple tiles and layers to make fine-grained decisions regarding which tiles to cache in each Small Base Station (SBS), and where to deliver them from to the end users, as users may reside in the coverage area of multiple SBSs. This permits to cache the most popular tiles in the SBSs, while the remaining tiles may be obtained through the backhaul. In addition, we explicitly consider the time delivery constraints to ensure continuous video playback. To reduce the computational complexity of the optimization problem, we simplify it by introducing a fairness constraint. This allows us to split the original problem into subproblems corresponding to Groups of Pictures (GOP). Each of the subproblems is then solved with the method of Lagrange partial relaxation. Finally, we evaluate the performance of the proposed method for various system parameters and compare it with schemes that do not consider 360° video encoding into multiple tiles and quality layers, as well as with two variants of the proposed method: one that considers layered encoding and SBSs collaboration and another that uses tiles encoding but with no SBSs collaboration. The results showcase the benefits coming from caching and delivery decisions on per tile basis and the importance of exploiting SBSs collaboration.

A Fast FoV-Switching DASH System Based on Tiling Mechanism for Practical Omnidirectional Video Services

With the development of multimedia technologies and virtual reality display devices, omnidirectional videos have gained popularity nowadays. To reduce the bandwidth requirement for omnidirectional video transmission, tile-based viewport adaptive streaming methods have been proposed in the literatures. Challenges related to decoding the tiles simultaneously with limited number of decoders, and ensuring user's viewing experience during the viewport switch are still to be solved. In this paper, a two-layer fast viewport switching dynamic adaptive streaming over HTTP (DASH) system based on tiling mechanism is proposed, which incorporates the viewing trajectory of end users. To deal with the simultaneously decoding problem, an open group of picture (GOP) technique is proposed to enable merging different types of tiles into a composite stream at the client side. To reduce the quality recovery duration after the viewport change, a fast-switching strategy is also proposed. Moreover, considering the priorities of different types of chunks, a download strategy is further proposed to adapt the bandwidth fluctuations and viewport changes. Experimental results showed that the proposed system can significantly reduce the recovery duration of high quality video by approximately 90%, which can provide a better viewing experience to end users.

Frame level rate control algorithm based on GOP level quality dependency for low-delay hierarchical video coding

Rate control plays an important role in video coding, which can minimize the distortion of compressed video with bit rate constraints. Rate control algorithm based on R−λ model is capable of controlling the bit rate precisely for high efficiency video coding (HEVC). However, quality dependency is not considered very well in λ-domain rate control algorithm when allocating bits and updating model parameters. This paper proposes a rate control algorithm based on group of pictures (GOP) level quality dependency for low-delay hierarchical coding structure. Firstly, the quality dependency between the current GOP and encoded GOPs mainly comes from the crucial frame (C-Frame). C-Frame guided GOP level quality dependency (CF-GOP-QD) model can be expressed by a CF-GOP-QD factor. Secondly, a CF-GOP-QD model based GOP level rate–distortion optimization (RDO) under the total bit constraints is proposed to guide the bit allocation processing, which is better than traditional bit allocation method. Experimental results show that the proposed rate control algorithm can achieve better rate–distortion performance, better visual quality and better rate control accuracy. Specifically, the proposed method shows 5.801% and 5.068% BD-rate saving on average compared against HM16.7 frame level rate control method under low-delay B and low-delay P configurations.

Video Compressed Sensing Using a Convolutional Neural Network

Recently, a few image compressed sensing (CS) methods based on deep learning have been developed, which achieve remarkable reconstruction quality with low computational complexity. However, these existing deep learning-based image CS methods focus on exploring intraframe correlation while ignoring interframe cues, resulting in inefficiency when directly applied to video CS. In this paper, we propose a novel video CS framework based on a convolutional neural network (dubbed VCSNet) to explore both intraframe and interframe correlations. Specifically, VCSNet divides the video sequence into multiple groups of pictures (GOPs), of which the first frame is a keyframe that is sampled at a higher sampling ratio than the other nonkeyframes. In a GOP, the block-based framewise sampling by a convolution layer is proposed, which leads to the sampling matrix being automatically optimized. In the reconstruction process, the framewise initial reconstruction by using a linear convolutional neural network is first presented, which effectively utilizes the intraframe correlation. Then, the deep reconstruction with multilevel feature compensation is proposed, which compensates the nonkeyframes with the keyframe in a multilevel feature compensation manner. Such multilevel feature compensation allows the network to better explore both intraframe and interframe correlations. Extensive experiments on six benchmark videos show that VCSNet provides better performance over state-of-the-art video CS methods and deep learning-based image CS methods in both objective and subjective reconstruction quality.

Frame-level Bit Allocation Optimization Based on<?brk?> Video Content Characteristics for HEVC

Rate control plays an important role in high efficiency video coding (HEVC), and bit allocation is the foundation of rate control. The video content characteristics are significant for bit allocation, and modeling an accurate relationship between video content characteristics and bit allocation is essential for bit allocation optimization. Therefore, in this article, a video content characteristics–based frame-level optimal bit allocation algorithm is proposed for improving the rate distortion (RD) performance of HEVC. First, the number of search points of motion estimation is used to evaluate the motion activity of video content, and the relationship between the search points and bit allocation is modeled as the search-points model. Second, the grey level co-occurrence matrix and temporal perceptual information are used to evaluate the spatial and temporal texture complexity, and the relationship between the video content texture complexity and bit allocation is modeled as the texture-complexity model. Then, the search-points model and texture-complexity model are jointly employed to allocate the coding bits for the second and third layers of the HEVC hierarchical coding structure. Finally, the remaining coding bits of a group-of-pictures (GOP) are allocated to the first layer of HEVC coding structure. To evaluate the performance of the proposed algorithm, the RD performance and bitrate accuracy are used as evaluation criteria, and the experimental results show that when compared with the popularly used R-λ model–based bit allocation algorithm, the proposed algorithm achieves an average of -3.43% BDBR reduction and 0.13 dB BDPSNR gains with only 0.02% loss of bitrate accuracy.

FJND-based fuzzy rate control of scalable video for streaming applications

A fuzzy rate controller with buffer constraint in combination with a perceptual quality controller is proposed for streaming applications of the AVC/H.264 scalable (SVC) video. The bit rate of each video layer is controlled separately by the fuzzy controller that adjusts the quantization parameter (QP) on a group of pictures (GOP) basis. The QPs of pictures are computed from the GOP QP by the well-known QP cascading technique. While the fuzzy controller provides the buffering constraint for each video layer, the quality controller tries to improve the perceptual quality of the compressed video based on the foveated just-noticeable distortion (FJND) model. The quality controller regulates the QP of each macroblock around the picture QP based on the visibility threshold of the FJND model. In these applications, the initial buffering allows slight variations of the bit rate leading to produce a variable bit rate (VBR) video bit stream with consistent quality. Experimental results show that the proposed algorithm effectively adapts to the buffer size, while strictly prevents buffer overflow and underflow. In addition, incorporating the perceptual quality controller into the fuzzy rate controller achieves higher perceptual quality at the same bit rate.

On the effectiveness of including meaningful pictures in the formation of equivalence classes.

In three experiments, 165 adult participants were trained on 12 baseline conditional discriminations and tested for the formation of three 5-member equivalence classes (A➔B➔C➔D➔E). All experiments included two reference groups; the abstract (ABS) group was trained with all abstract stimuli and the picture (PIC) group with C-stimuli as meaningful pictorial stimuli but A, B, D, and E stimuli as abstract shapes. In Experiment 1, the color of the meaningful stimuli was manipulated. In the ABS, PIC, and black-and-white groups, 33.3%, 80%, and 93.3% formed equivalence classes, respectively. In Experiment 2, participants were exposed to a test block with and without trials that included C stimuli. For the groups with and without C trials in the test, 93.3% and 86.7% formed equivalence classes, respectively, compared to 20% in the ABS group. In Experiment 3, the number of meaningful pictures and their location in stimulus classes were manipulated. One group was trained with 3 pictures (C1/B2/D3, the 3-PIC) while the other groups had 2 pictures (C1/B2 and C1/D3, the 2-PIC). In the second test block for the ABS and PIC groups, 6.7% and 86% of the participants formed equivalence classes, respectively. For the 3-PIC and the 2-PIC groups, 66.7% and 50% of the participants formed equivalence classes, respectively. Results suggest that the effects of meaningful stimuli in equivalence classes (a) cannot be attributed to the use of colorful stimuli in previous studies, (b) occur during training and are not dependent on the presence of meaningful stimuli at test, and (c) are sensitive to stimulus location.

Fast Side Information Generation for High-Resolution Videos in Distributed Video Coding Applications

Distributed video coding (DVC) is an attractive and promising scheme that suits the constrained video applications, such as wireless sensor networks or wireless surveillance systems. In DVC, estimation of fast and consistent side information (Տį) is a critical issue for instant and real-time decoding. This issue becomes even more serious for high-resolution videos. Therefore, to minimise the side information estimation computational complexity, in this work, a computationally low complex DVC codec is proposed, which uses a simple phase interpolation (Phase-I) algorithm. It performs faster for all resolutions videos, and significant results are achieved for high-resolution videos with a large group of pictures (GOP). For the proposed technique, the computation time rapidly decreases with an increase in resolution. It performs 221% to 280% faster from conventional frame interpolation method for high-resolution videos and large GOP at the cost of little degradation in the visual quality of estimated side information.

EEG Compression Using Motion Compensated Temporal Filtering and Wavelet Based Subband Coding

Electroencephalography (EEG) signals are commonly used in medical applications for prevention, diagnosis, and detection of neurological diseases. These EEG signals have also been used in designing brain computer interfaces for assistive technologies. For densely placed electrodes and long EEG recordings, a large amount of data needs to be stored, preferably in compressed form. This EEG signal compression is particularly required in an out-of-the-lab environment so that these signals are efficiently transmitted over wired/wireless communication channels. To this end, we propose a novel compression scheme for EEG signals, which exploits the intra-channel redundancy using motion compensated temporal filtering (MCTF) and discrete wavelet transform (DWT) based sub-band coding. In the pre-processing stage, multi-frame data is constructed such that each group of picture (GOP) contains information from a single channel of EEG data. This helps in removing the intra-channel redundancy. We apply MCTF on each GOP to exploit temporal redundancy, following which the DWT is applied on temporally decomposed frames to exploit spatial redundancy. Each spatio-temporal decomposed frame is assigned a bit budget for minimum distortion. For this purpose, we assign more bit budget to temporally decomposed low pass frames as compared to high pass frames. Spatio-temporal frames are then encoded at the assigned bit rate by using set partitioning in hierarchical tree (SPIHT) algorithm to create the bit stream. Our experimental results showed 4.5% and 2.4% reduction in distortion at the same data rate for BCI-3 and BCI-4 datasets, respectively. These results improve upon the reduction in data size achieved using state-of-the-art compression methods such as SPIHT and SPIHT with independent component analysis.

Rate Control for Video-based Point Cloud Compression.

Rate control is a necessary tool for video-based point cloud compression (V-PCC). However, there is no solution specified on this topic yet. In this paper, we propose the first rate control algorithm for V-PCC. Generally, a rate control algorithm is divided into two processes: bit allocation and bitrate control. In V-PCC, the total bits are composed of three parts: the header information including the auxiliary information and occupancy map, the geometry video, and the attribute video. The bit allocation aims to assign the total bits to these three parts. Since the auxiliary information and occupancy map are encoded losslessly, the bit cost of the header information is fixed. Therefore, we only need to assign bits between the geometry and attribute videos. Our first key contribution is the proposed video-level bit allocation algorithm between the geometry and attribute videos to optimize the overall reconstructed point cloud quality. Then we assign geometry and attribute video bits to each group of pictures (GOP), each frame, and each basic unit (BU). Our second key contribution is that we assign zero bits to the BUs with only unoccupied pixels. The unoccupied pixels are useless for the reconstructed quality of the point cloud and therefore should be assigned zero bits. In the bitrate control process, the encoding parameters are determined, and the model parameters are updated for each frame and BU to achieve the target bits. Our third key contribution is that we propose a BU-level model updating scheme to handle the case where various patches may be placed in different positions in neighboring frames. We use the auxiliary information to find the corresponding BU in the previous frame and apply its model parameters to the current BU. The proposed algorithms are implemented in the V-PCC and High Efficiency Video Coding (HEVC) reference software. The experimental results show that the proposed rate control algorithm can achieve significant bitrate savings compared with the state-of-the-art method.

Double compression detection for H.264 videos with adaptive GOP structure

As a blind forensic method, double compression detection is valid to multiple manipulations. However, the existing methods only consider to detect the videos with fixed Group of Pictures (GOP). In this paper, we put forward a novel double compression detection method for videos with both fixed and adaptive GOP structure in H.264 videos. Considering that video may contain adaptive GOPs caused by fast moving contents or scene changes, in our double compression detection scheme, temporal segmentation is first used to divide video into static and rapid periods which contain normal fixed and adaptive GOPs respectively. Then, new artifacts based on the sequence of frame’s byte count (FBC) are analyzed. A feature sequence composed of recognizable distances is generated by combining the artifacts in the static and rapid periods of video. Finally, to reveal the intrinsic property of the feature sequence, a scoring strategy is designed to determine whether or not double compression. The experiments demonstrate that the proposed scheme is effective to detect double compression of H.264 videos, and it outperforms other existing state-of-the-art methods.

Detection of Transcoding from H.264/AVC to HEVC Based on CU and PU Partition Types

High Efficiency Video Coding (HEVC) is a worldwide popular video coding standard due to its high coding efficiency. To make profits, forgers prefer to transcode videos from previous standards such as H.264/AVC to HEVC. To deal with this issue, an efficient method is proposed to expose such transcoded HEVC videos based on coding unit (CU) and prediction unit (PU) partition types. CU and PU partitioning are two unique syntactic units of HEVC that can reflect a video’s compression history. In this paper, CU and PU partition types of I pictures and P pictures are firstly extracted. Then, their mean frequencies are calculated and concatenated as distinguishing features, which are further sent to a support vector machine (SVM) for classification. Experimental results show that the proposed method can identify transcoded HEVC videos with high accuracy and has strong robustness against frame-deletion and shifted Group of Pictures (GOP) structure attacks.

Efficient Mobile Video Streaming via Context-Aware RaptorQ-Based Unequal Error Protection

Mobile video streaming systems typically apply the forward error correction (FEC) at the application layer to cope with packet-level transmission errors, which complements the bit-level correction mechanisms at the physical layer. However, most existing works fail to exploit the block-level dependencies in both intra and interframe coding modes of a single-layer compressed video, and thus are less efficient for the prevailing H.264/AVC and/or H.265/HEVC compatible single-layer video application. To this end, we propose a low-complexity FEC, i.e., context-aware RaptorQ (CA-RQ) with unequal error protection (UEP), to improve the error recovery performance of the single-layer mobile video streaming, through incorporating the block-level dependencies in the compressed video data. We use a packet-level video transmission distortion model that considers the dependencies in both spatial and temporal domains, to quantify the importance of video packets within a group of pictures (GoP). The compressed video packets are categorized and grouped into several classes according to their importance to construct the CA-RQ code with the UEP property. We provide a theoretical analysis on redundancy allocation bounds to demonstrate the superior performance of proposed CA-RQ over the standard RaptorQ code. In the meantime, extensive simulations have shown that our scheme not only offers much better subjective visual quality with less than 50% additional redundant symbols as compared to the Macroblock-Based UEP (MB-UEP) scheme, but also outperforms the MB-UEP and classical equal error protection (EEP)-based schemes, by a 0.45% $\sim$ 5.71% and 0.94% $\sim$ 6.78% margin, respectively, in reconstructed quality evaluated using the structural similarity (SSIM) index, across a reasonable range of redundancy proportions.

Quadtree-Based Coding Framework for High-Density Camera Array-Based Light Field Image

The size of a high-density-camera-array (HDCA)-based light field image (LFI) is usually very large, containing hundreds of high-resolution views. Therefore, there is an urgent need to efficiently compress it. Currently, no compression algorithms, specially, for the HDCA-based LFI have been designed. In this paper, we propose an algorithm based on a quadtree-based 2D hierarchical coding framework for the HDCA-based LFI data compression. The proposed framework has the following contributions. First, we organize the views of the HDCA-based LFI into a quadtree-based coding structure. Under this structure, all of the views are divided into four quadrants at the first level. Each quadrant is further sub-divided into four quadrants at each subsequent level. The process continues until the desired depth is reached. This quadtree-based coding structure can make full use of the strong inter-view correlations to improve the coding efficiency. In addition, the proposed quadtree-based structure can be easily extended to a general 2D hierarchical structure with variable group of pictures (GOP) sizes to adapt to the reference frame buffer constraint. Second, we try to improve the performance of the 2D hierarchical coding framework using the distance-based criteria for both the reference frame selection and motion vector scaling. Third, a one-pass optimal bit allocation scheme is proposed to further optimize the performance by taking the quality dependencies among various views into consideration. The proposed framework is implemented in the newest video coding standard, high efficiency video coding (HEVC). The experimental results show that the proposed quadtree-based 2D hierarchical coding framework can achieve an average of over 25% bitrate saving compared with the 1D hierarchical coding structure.