Conventional Video Codecs Research Articles

Learned scalable video coding for humans and machines

AbstractVideo coding has traditionally been developed to support services such as video streaming, videoconferencing, digital TV, and so on. The main intent was to enable human viewing of the encoded content. However, with the advances in deep neural networks (DNNs), encoded video is increasingly being used for automatic video analytics performed by machines. In applications such as automatic traffic monitoring, analytics such as vehicle detection, tracking and counting, would run continuously, while human viewing could be required occasionally to review potential incidents. To support such applications, a new paradigm for video coding is needed that will facilitate efficient representation and compression of video for both machine and human use in a scalable manner. In this manuscript, we introduce an end-to-end learnable video codec that supports a machine vision task in its base layer, while its enhancement layer, together with the base layer, supports input reconstruction for human viewing. The proposed system is constructed based on the concept of conditional coding to achieve better compression gains. Comprehensive experimental evaluations conducted on four standard video datasets demonstrate that our framework outperforms both state-of-the-art learned and conventional video codecs in its base layer, while maintaining comparable performance on the human vision task in its enhancement layer.

NN-VVC: A Hybrid Learned-Conventional Video Codec Targeting Humans and Machines

Advancements in artificial intelligence have significantly increased the use of images and videos in machine analysis algorithms, predominantly neural networks. However, the traditional methods of compressing, storing and transmitting media have been optimized for human viewers rather than machines. Current research in coding images and videos for machine analysis has evolved in two distinct paths. The first is characterized by End-to-End (E2E) learned codes, which show promising results in image coding but have yet to match the performance of leading Conventional Video Codecs (CVC) and suffer from a lack of interoperability. The second path optimizes CVC, such as the Versatile Video Coding (VVC) standard, for machine-oriented reconstruction. Although CVC-based approaches enjoy widespread hardware and software compatibility and interoperability, they often fall short in machine task performance, especially at lower bitrates. This paper proposes a novel hybrid codec for machines named NN-VVC, which combines the advantages of an E2E-learned image codec and a CVC to achieve high performance in both image and video coding for machines. Our experiments show that the proposed system achieved up to − 43.20% and − 26.8% Bjøntegaard Delta rate reduction over VVC for image and video data, respectively, when evaluated on multiple different datasets and machine vision tasks according to the common test conditions designed by the VCM study group in MPEG standardization activities. Furthermore, to improve reconstruction quality, we introduce a human-focused branch into our codec, enhancing the visual appeal of reconstructions intended for human supervision of the machine-oriented main branch.

Neural compression for hologram images and videos.

Holographic near-eye displays can deliver high-quality three-dimensional (3D) imagery with focus cues. However, the content resolution required to simultaneously support a wide field of view and a sufficiently large eyebox is enormous. The consequent data storage and streaming overheads pose a big challenge for practical virtual and augmented reality (VR/AR) applications. We present a deep-learning-based method for efficiently compressing complex-valued hologram images and videos. We demonstrate superior performance over the conventional image and video codecs.

End-to-End Rate-Distortion Optimized Learned Hierarchical Bi-Directional Video Compression.

Conventional video compression (VC) methods are based on motion compensated transform coding, and the steps of motion estimation, mode and quantization parameter selection, and entropy coding are optimized individually due to the combinatorial nature of the end-to-end optimization problem. Learned VC allows end-to-end rate-distortion (R-D) optimized training of nonlinear transform, motion and entropy model simultaneously. Most works on learned VC consider end-to-end optimization of a sequential video codec based on R-D loss averaged over pairs of successive frames. It is well-known in conventional VC that hierarchical, bi-directional coding outperforms sequential compression because of its ability to use both past and future reference frames. This paper proposes a learned hierarchical bi-directional video codec (LHBDC) that combines the benefits of hierarchical motion-compensated prediction and end-to-end optimization. Experimental results show that we achieve the best R-D results that are reported for learned VC schemes to date in both PSNR and MS-SSIM. Compared to conventional video codecs, the R-D performance of our end-to-end optimized codec outperforms those of both x265 and SVT-HEVC encoders ("veryslow" preset) in PSNR and MS-SSIM as well as HM 16.23 reference software in MS-SSIM. We present ablation studies showing performance gains due to proposed novel tools such as learned masking, flow-field subsampling, and temporal flow vector prediction. The models and instructions to reproduce our results can be found in https://github.com/makinyilmaz/LHBDC/.

Deep Predictive Video Compression Using Mode-Selective Uni- and Bi-Directional Predictions Based on Multi-Frame Hypothesis

Recently, deep learning-based image compression has shown significant performance improvement in terms of coding efficiency and subjective quality. However, there has been relatively less effort on video compression based on deep neural networks. In this paper, we propose an end-to-end deep predictive video compression network, called DeepPVCnet, using mode-selective uni- and bi-directional predictions based on multi-frame hypothesis with a multi-scale structure and a temporal-context-adaptive entropy model. Our DeepPVCnet jointly compresses motion information and residual data that are generated from the multi-scale structure via the feature transformation layers. Recent deep learning-based video compression methods were proposed in a limited compression environment using only P-frame or B-frame. Learned from the lesson of the conventional video codecs, we firstly incorporate a mode-selective framework into our DeepPVCnet with uni- and bi-directional predictive modes in a rate-distortion minimization sense. Also, we propose a temporal-context-adaptive entropy model that utilizes the temporal context information of the reference frames for the current frame coding. The autoregressive entropy models for CNN-based image and video compression is difficult to compute with parallel processing. On the other hand, our temporal-context-adaptive entropy model utilizes temporally coherent context from the reference frames, so that the context information can be computed in parallel, which is computationally and architecturally advantageous. Extensive experiments show that our DeepPVCnet outperforms AVC/H.264, HEVC/H.265 and state-of-the-art methods in an MS-SSIM perspective.

Block Compressive Sensing Single-View Video Reconstruction Using Joint Decoding Framework for Low Power Real Time Applications

Several real-time visual monitoring applications such as surveillance, mental state monitoring, driver drowsiness and patient care, require equipping high-quality cameras with wireless sensors to form visual sensors and this creates an enormous amount of data that has to be managed and transmitted at the sensor node. Moreover, as the sensor nodes are battery-operated, power utilization is one of the key concerns that must be considered. One solution to this issue is to reduce the amount of data that has to be transmitted using specific compression techniques. The conventional compression standards are based on complex encoders (which require high processing power) and simple decoders and thus are not pertinent for battery-operated applications, i.e., VSN (primitive hardware). In contrast, compressive sensing (CS) a distributive source coding mechanism, has transformed the standard coding mechanism and is based on the idea of a simple encoder (i.e., transmitting fewer data-low processing requirements) and a complex decoder and is considered a better option for VSN applications. In this paper, a CS-based joint decoding (JD) framework using frame prediction (using keyframes) and residual reconstruction for single-view video is proposed. The idea is to exploit the redundancies present in the key and non-key frames to produce side information to refine the non-key frames’ quality. The proposed method consists of two main steps: frame prediction and residual reconstruction. The final reconstruction is performed by adding a residual frame with the predicted frame. The proposed scheme was validated on various arrangements. The association among correlated frames and compression performance is also analyzed. Various arrangements of the frames have been studied to select the one that produces better results. The comprehensive experimental analysis proves that the proposed JD method performs notably better than the independent block compressive sensing scheme at different subrates for various video sequences with low, moderate and high motion contents. Also, the proposed scheme outperforms the conventional CS video reconstruction schemes at lower subrates. Further, the proposed scheme was quantized and compared with conventional video codecs (DISCOVER, H-263, H264) at various bitrates to evaluate its efficiency (rate-distortion, encoding, decoding).

3D Motion Estimation and Compensation Method for Video-Based Point Cloud Compression

A point cloud visualizes information by placing a voxel with a color value and a position value in a three-dimensional space. Since a point cloud uses hundreds of thousands or millions of points to visualize information, a large number of bits is needed compared to existing 2D media. Therefore, it is essential to compress point data for transmission and storage. The Moving Picture Expert Group (MPEG) is developing a point cloud compression method based on 2D video that takes advantage of the benefits of coding efficiency and the wide adaption of video codecs by various industries. This compression method is called video-based point cloud compression (V-PCC). Generally, video codecs use a compression method that employs a block matching algorithm. Currently, V-PCC is conducted using 2D video codecs, which means that motion information used by V-PCC is obtained from 2D video sequences. Thus, this 2D-based motion information limits the characterization of the motion in terms of 3D-points, which is also disadvantageous to compression efficiency. In this paper, we propose a method for estimating and compensating the motion in terms of a 3D object when compressing a dynamic object point cloud using a conventional video codec. The proposed 3D motion estimation and compensation technology showed higher gain overall in terms of BD-rate and was proven to effectively compress 3D point cloud content on the basis of 3D motion.

Contextual Homogeneity-Based Patch Decomposition Method for Higher Point Cloud Compression

Point cloud content is widely used to store and represent 3D volumetric objects with a complex and detailed representation from any direction of view. However, the amount of data needed for point cloud content is much larger than that of 2D representations. To overcome this difficulty, MPEG has started to develop a Video-based Point Cloud Compression (V-PCC) that is designed by projecting point cloud content into 2D content and compressing the 2D content using conventional 2D video codecs. Compression efficiency of the V-PCC can be achieved when 3D motion flow and textual conformity on 3D surfaces are preserved through 2D projections that are favorable to the 2D video codec. As mentioned above, point cloud content has a complex geometry, therefore when decomposing a point from 3D coordinates to construct a 2D patch, several situations must be considered in addition to the location of adjacent points. This paper addresses the issues in such complex geometry by proposing a method that preserves 3D homogeneity in 2D patches. Comprehensive experiments are conducted to demonstrate bitrate savings of 0.5%, 0.6%, 7.8%, 7.0% and 5.5% in random access mode and 0.1%, 0.0%, 7.0%, 4.2% and 3.3% in all intra mode for D1, D2, Y, Cb, and Cr, respectively, compared to the reference software.

Utilising Low Complexity CNNs to Lift Non-Local Redundancies in Video Coding.

Digital media is ubiquitous and produced in ever-growing quantities. This necessitates a constant evolution of compression techniques, especially for video, in order to maintain efficient storage and transmission. In this work, we aim at exploiting non-local redundancies in video data that remain difficult to erase for conventional video codecs We design convolutional neural networks with a particular emphasis on low memory and computational footprint. The parameters of those networks are trained on the fly, at encoding time, to predict the residual signal from the decoded video signal. After the training process has converged, the parameters are compressed and signalled as part of the code of the underlying video codec. The method can be applied to any existing video codec to increase coding gains while its low computational footprint allows for an application under resource-constrained conditions. Building on top of High Efficiency Video Coding, we achieve coding gains similar to those of pretrained denoising CNNs while only requiring about 1% of their computational complexity Through extensive experiments, we provide insights into the effectiveness of our network design decisions. In addition, we demonstrate that our algorithm delivers stable performance under conditions met in practical video compression: our algorithm performs without significant performance loss on very long random access segments (up to 256 frames) and with moderate performance drops can even be applied to single frames in high-resolution low delay settings.

Distributed video coding with adaptive two-step side information generation for smart and interactive media

Conventional video coding standards, such as MPEGx and H.26x, use a hybrid architecture of block-based motion compensation and discrete cosine transform (DCT) within the structure of a complex encoder and a simple decoder. Contrary to conventional video codecs, Wyner-Ziv (WZ) video coding, which is a practical application of distributed video coding (DVC) based on the Slepian-Wolf and WZ theorems, exploits the source correlation at the decoder, thereby allowing for the development of simpler encoders. However, the current WZ video coding algorithms cannot match the coding performance of conventional video coding. In order to improve the coding performance of transform-domain WZ video coding, an adaptive two-step side-information generation method is thus proposed for evaluation and analysis in this study. The proposed method uses decoded WZ frames in the down-sampled spatial resolution along with already decoded coefficients to successively improve rate-distortion (RD) performance as the decoding progresses. The experimental results show that the proposed method, compared to previous WZ video coding method, shows improved coding performance, particularly under critical conditions, such as cases with high motion content.

A New Progressively Refined Wyner-Ziv Video Coding for Low-Power Human-Centered Telehealth

With the increas of the global aging population, elderly care has become an important social issue around the world. Human-centered telehealth provides more efficient and comfortable health-care services for elderly people through collecting the elderly’s information remotely. Video taken by wearable cameras is one of the most efficient carriers for human-centered telehealth. Whereas wearable cameras are mainly limited in energy supply and computation, the conventional video codecs such as H.26x requiring encoders with powerful processing ability are thus not suitable. Distributed video coding (DVC) based on the Wyner–Ziv (WZ) coding architecture, namely, WZ video coding, can exploit the source statistics only at decoders. It thus provides an efficient solution for low-power wearable cameras. Nevertheless, the compression performance gap between the DVC and the conventional video coding still exists. One of the main reasons for this weakness is the quality of the side information (SI). As the estimation of the current WZ frame, the SI provides the important inter-frame correlation for the correlation noise statistics. In this paper, a novel algorithm is proposed for the SI refinement first. The proposed refinement algorithm iteratively learns the difference between the already decoded information of the current WZ frame and the SI, and makes a targeted refinement for the SI quality. Subsequently, a progressively refined correlation noise model is proposed based on the novel SI refinement algorithm. The progressively refined WZ video coding is thus achieved. The performance evaluations show that the proposed technique advances over the existing DVC systems. The proposed technique provides an efficient way to improve the video compression performance for the low-power wearable cameras in human-centered telehealth.

Frame differencing-based segmentation for low bit rate video codec using H.264

In video sequence coding, a combination of temporal and spatial coding technique is used in order to remove the predictable or redundant image content and encode only the unpredictable information. The objective of video compression technique is to increase the coding efficiency and to increase the data rate savings. A segmentation-based compression method is proposed to achieve this goal. In this paper, static portions are identified using frame differencing method and segmented during the encoding process. This encoded information is passed to the synthesiser. In synthesis, information which is passed during segmentation process is added to reconstruct the video. This method is integrated with the conventional video codec H.264/AVC video codec. Experimental results substantially proved the data rate is reduced by as much as 25%.

Improving low bitrate video coding via computation incorporating a priori information

The growing number of mobile users, as well as the diversification in types of services have resulted in increasing demands for wireless network bandwidth in recent years. Although evolving transmission techniques are able to enlarge the network capacity to some degree, they still cannot satisfy the requirements of mobile users. Meanwhile, following Moore's Law, the data processing capabilities of mobile user terminals are continuously improving. In this paper, we explore possible methods of trading strong computational power at wireless terminals for transmission efficiency of communications. Taking the specific scenario of wireless video conversation, we propose a model-based video coding scheme by learning the structures in multimedia contents. Benefiting from both strong computing capability and pre-learned model priors, only low-dimensional parameters need to be transmitted; and the intact multimedia contents can also be reconstructed at the receivers in real-time. Experiment results indicate that, compared to conventional video codecs, the proposed scheme significantly reduces the data rate with the aid of computational capability at wireless terminals.

ADAPTIVE GOP STRUCTURE TO H.264/AVC BASED ON SCENE CHANGE

This paper proposes an adaptive GOP structure with the new logic of frame comparison in H.264/AVC to achieve better quality and reduce bit rate. Initially Group of Pictures (GOP) is set to a fixed size. Frames are compared within that GOP using correlation. According to the correlation, GOP is changed within that fixed size. So, there will be no GOP size greater than that fixed size. This method does not calculate any threshold. Hence the time needed to calculate global or local threshold is eliminated. It is integrated with conventional video codec H.264/AVC. This method is compared with H.264/AVC of fixed GOP structure of sizes 4, 8, 12, 16, 32 and GOP structure with the length of entire video. The proposed method achieved gain in bit rate from 0.49% to 69.75% and PSNR gain from 2.5% to 0.3%.

HEVC 고성능 압축 도구들의 성능 분석을 통한 스크린 콘텐츠 응용 최적 부호화 모델

카메라에 의해 촬영되는 자연 영상과 달리 컴퓨터나 스마트폰 등과 같은 다양한 전자기기에 의해 생성/랜더링(rendering)된 영상이나 동영상을 스크린 콘텐츠라 한다. 스크린 콘텐츠는 자연영상과는 구별되는 통계적 특성을 가지고 있기 때문에 자연 영상 혹은 동영상을 대상으로 하는 지금까지의 압축표준을 이용하는 경우 압축효율이 저하될 수밖에 없다. 스크린 콘텐츠의 효율적인 압축을 위한 연구가 최근 들어 활발히 이루어지고 있으며, 특히 현재 진행 중인 고효율 동영상 부호화(HEVC : High Efficiency Video Coding)의 표준을 위한 JCT-VC(Joint Collaborative Team on Video Coding) 회의에서 스크린 콘텐츠가 가지는 특성에 적합한 압축 방식들이 지속적으로 논의 되고 있다. 본 논문에서는 HEVC 참조 소프트웨어인 HM(HEVC Test Model)의 고성능 압축 도구들에 대해 스크린 콘텐츠 압축 효율성을 분석하고, 분석을 통해 얻어진 결과를 기반으로 스크린 콘텐츠 응용에 최적화된 부호화 모델을 제시한다. 또한, 스크린 콘텐츠가 자연동영상과 구별되는 특성들과 스크린 콘텐츠 응용에서의 HEVC 적용을 위한 앞으로의 연구쟁점을 제시한다. Screen content refers to images or videos generated by various electronic devices such as computers or mobile phones, whereas natural content refers to images captured by cameras. Screen contents show different statistical characteristics from natural images, so the conventional video codecs which were developed mainly for the coding of natural videos cannot guarantee good coding performances for screen contents. Recently, researches on efficient SCC(Screen Content Coding) are being actively studied, and especially at ongoing JCT-VC(Joint Collaborative Team on Video Coding) meeting for HEVC(High Efficiency Video Coding) standard, SCC issues are being discussed steadily. In this paper, we analyze the performances of high efficient coding tools in HM(HEVC Test Model) on SCC, and present an optimized SCC model based on the analysis results. We also present the characteristics of screen contents and the future research issues as well.

적응적인 움직임 벡터 해상도를 이용한 움직임 벡터 부호화 방법

MPEG-2, MPEG-4와 같은 기존의 비디오 코덱에서는 인터 예측을 수행할 때 고정된 해상도의 움직임 벡터를 사용한다. 그러나 KTA 참조 소프트웨어에서는 움직임 벡터의 해상도를 슬라이스 단위로 선택하여 사용할 수 있는 기능을 지원한다. 그러나 선택된 하나의 움직임 벡터 해상도를 슬라이스 전체에 일괄적으로 적용하기 때문에 영상의 국지적인 특성을 반영하는데 어려움이 있다. 본 논문에서는 탐색 구간에 따라 적응적으로 움직임 벡터의 해상도를 결정하는 방법을 제안한다. 움직임 벡터의 탐색 영역을 움직임 벡터가 예측 움직임 벡터로부터 떨어진 거리에 따라 다수개의 구간으로 분할하고, 각 구간에 대하여 하나의 움직임 벡터 해상도를 할당하여 움직임 예측에 적용한다. 따라서 제안하는 방법의 부호화 효율은 각 구간을 분할하는 Threshold와 움직임 벡터를 부호화하는 엔트로피 코딩 방법에 영향을 받는다. HEVC의 참조 소프트웨어인 HM3.0을 이용하여 실험한 결과, Random Access 부호화 구조에서는 평균적으로 약 0.9%의 성능 향상을 얻을 수 있었으며, Low Delay 부호화 구조에 B picture를 적용한 경우는 약 0.6%, P picture를 적용한 경우에서는 약 2.7%의 평균 발생 비트량 감소를 확인할 수 있었다. In most conventional video codecs, such as MPEG-2 and MPEG-4, inter coding is performed with the fixed motion vector resolution. When KTA software was developed, resolution for MVs can be selected in each slice. Although KTA codec uses a variety of resolutions for ME, the selected resolution is applied over the entire pixels in the slice and the statistical property of the local area is not considered. In this paper, we propose an adaptive decision scheme for motion vector resolution which depends on region, where MV search area is divided to multiple regions according to the distance from PMV. In each region, the assigned resolution is used to estimate MV. Each region supports different resolution for ME from other regions. The efficiency of the proposed scheme is affected from threshold values to divide the search area and the entropy coding method to encode the estimated MV. Simulation results with HM3.0 which is the reference software of HEVC show that the proposed scheme provides bit rate gains of 0.9%, 0.6%, and 2.9% in Random Access, Low Delay with B picture, and Low Delay with P picture structures, respectively.

Subsampled Block-Matching for Zoom Motion Compensated Prediction

Motion compensated prediction plays a vital role in achieving enormous video compression efficiency in advanced video coding standards. Most practical motion compensated prediction techniques implicitly assume pure translational motions in the video contents for effective operation. Some attempts aiming at more general motion models are usually too complex requiring parameter estimation in practical implementation. In this paper, zoom motion compensation is investigated to extend the assumed model to support both zoom and translation motions. To accomplish practical complexity, a novel and efficient subsampled block-matching zoom motion estimation technique is proposed which makes use of the interpolated reference frames for subpixel motion estimation in a conventional hybrid video coding structure. Specially designed subsampling patterns in block matching are used to realize the translation and zoom motion estimation and compensation. No zoom parameters estimation and additional frame buffers are required in the encoder implementation. The complexity of the decoder is similar to the conventional hybrid video codec that supports subpixel motion compensation. The overall increase in memory requirement and computational complexity is moderate. Experimental results show that the new technique can achieve up to 9.89% bitrate reduction using KTA2.2r1 reference software implementation.

Novel prediction schemes for error resilient video coding

A conventional video codec uses encoder reconstruction of previous frames for motion compensated prediction. This is designed to minimize the encoder prediction error and assumes error free transmission. In this paper we use a modified prediction mechanism both at the encoder and decoder and propose techniques to improve the error resilience of H.264/AVC when transmitted over error prone networks. In our schemes we provide greater emphasis on Intra pixels during the formation of the reference frame used for prediction, thereby achieving better resilience. We also incorporate leaky prediction to further improve the robustness. We apply leaky prediction selectively at a macroblock level based on a simple mean square error metric in order to reduce the bit-rate penalty. Substantial performance gains have been observed in simulations. The effectiveness of using leaky prediction can be observed in medium and fast moving video sequences.

3D motion estimation for depth image coding in 3D video coding

In this paper, a new solution is introduced for the efficient compression of 3D video based on color and depth maps. While standard video codecs are designed for coding monoscopic videos, their application to depth maps is found to be suboptimal. With regard to the special properties of the depth maps, we propose an extension to conventional video coding in order to take advantage of object motion in the depth direction. Instead of performing a 2D motion search, as is common in conventional video codecs, we propose the use of a 3D motion search that is able to better exploit the temporal correlations of 3D content. In this new framework, the motion of blocks in depth maps are described using 3D motion vectors (x, y, z), representing the horizontal, vertical, and depth directions respectively. This leads to more accurate motion prediction and a smaller residual. The experimental results show that the proposed technique delivers an improvement in motion compensation, which leads to gains in compression efficiency.

Distributed Video Coding Using LDPC Codes for Wireless Video

Popular video coding standards like H.264 and MPEG working on the principle of motion-compensated pre-dictive coding demand much of the computational resources at the encoder increasing its complexity. Such bulky encoders are not suitable for applications like wireless low power surveillance, multimedia sensor networks, wireless PC cameras, mobile camera phones etc. New video coding scheme based on the principle of distributed source coding is looked upon in this paper. This scheme supports a low complexity encoder, at the same time trying to achieve the rate distortion performance of conventional video codecs. Current im-plementation uses LDPC codes for syndrome coding.