Interframe Coding Research Articles

CC-SMC: Chain coding-based segmentation map lossless compression

A segmentation map, either static or dynamic, refers to a two-dimensional picture that may vary with time and indicates the segmentation label per pixel. Both the semantic map and the occupancy map in video-based point cloud compression (V-PCC) belong to the segmentation map we referred to. The semantic map can work for many machine vision tasks like tracking and has been used as a layer of image representation in some image compression methods. The occupancy map constitutes a part of the point cloud coding bitstream. Since segmentation maps are widely used, how to efficiently compress them is of interest. We propose a segmentation map lossless compression scheme namely CC-SMC, exploiting the nature of segmentation maps that usually contain limited colors and sharp edges. Specifically, we design a chain coding-based scheme combined with quadtree-based block partitioning. For intraframe coding, one block is partitioned recursively with a quadtree structure, until the block contains only one color, is smaller than a threshold, or satisfies the defined chain coding condition. We revise the three-orthogonal chain coding method to incorporate contextual information and design effective intraframe prediction methods. For interframe coding, one block may find a reference block; the chain difference between the current and the reference blocks is coded. We implement the proposed scheme and test it on several different kinds of segmentation maps. Compared with advanced lossless image compression techniques, our proposed scheme obtains more than 10% bits reduction as well as more than 20% decoding time-saving. The code is available at https://github.com/Yang-Runyu/CC-SMC.

Conditional Residual Coding: A Remedy for Bottleneck Problems in Conditional Inter Frame Coding

Conditional Residual Coding: A Remedy for Bottleneck Problems in Conditional Inter Frame Coding

Fumos: Neural Compression and Progressive Refinement for Continuous Point Cloud Video Streaming.

Point cloud video (PCV) offers watching experiences in photorealistic 3D scenes with six-degree-of-freedom (6-DoF), enabling a variety of VR and AR applications. The user's Field of View (FoV) is more fickle with 6-DoF movement than 3-DoF movement in 360-degree video. PCV streaming is extremely bandwidth-intensive. However, current streaming systems require hundreds of Mbps bandwidth, exceeding the bandwidth capabilities of commodity devices. To save bandwidth, FoV-adaptive streaming predicts a user's FoV and only downloads point cloud data falling in the predicted FoV. But it is difficult to accurately predict the user's FoV even 2-3 seconds before playback due to 6-DoF. Misprediction of FoV or network bandwidth dips results in frequent stalls. To avoid rebuffering, existing systems would cause incomplete FoV and degraded experience, deteriorating the user's quality of experience (QoE). In this paper, we describe Fumos, a novel system that preserves interactive experience by avoiding playback stalls while maintaining high perceptual quality and high compression rate. We find a research gap in inter-frame redundant utilization and progressive mechaism. Fumos has three crucial designs, including (1) Neural compression framework with inter-frame coding, namely N-PCC, which achieves both bandwidth efficiency and high fidelity. (2) Progressive refinement streaming framework that enables continuous playback by incrementally upgrading a fetched portion to a higher quality (3) System-level adaptation that employs Lyapunov optimization to jointly optimize the long-term user QoE. Experimental results demonstrate that Fumos significantly outperforms Draco, achieving an average decoding rate acceleration of over 260×. Moreover, the proposed compression framework N-PCC attains remarkable BD-Rate gains, averaging 91.7% and 51.7% against the state-of-the-art point cloud compression methods G-PCC and V-PCC, respectively.

Preliminary study of cine-MRI compression in MR-guided radiotherapy.

Cine-magnetic resonance imaging (MRI) is currently used in real-time tumor tracking during magnetic resonance (MR)-guided radiotherapy. As a type of MRI specified for motion tracking, a few minutes' acquisition results in thousands of 2-dimensional (2D) images. For MR-guided radiotherapy consisting of multiple treatment fractions, the large number of cine-MRI images would be disproportionate to the tight clinical data storage available. To alleviate this issue, the feasibility of compression of cine-MRI via video encoders was investigated in this study. The cine-MRI images were first sorted into 3 sequences according to their plane orientations. Then, each sequence was reordered according to their acquisition times [time-based (TB)] or content similarities [similarity-based (SB)]. As a result, 3 sequences were obtained for 3 plan orientations. Next, the obtained sequences were processed by a video encoder and the corresponding 3 video files were achieved. We employed 3 popular video encoders: Motion JPEG (M-JPEG), Advanced Video Coding (AVC), and High Efficiency Video Coding (HEVC). The performances of the sequence reordering methods and video encoders were evaluated based on a total of 150 image sets. The mean correlation quantities for SB sequences were higher than those for TB sequences by 3% (sagittal), 2% (coronal), and 1% (transverse), respectively. The average compression ratio (CR) yielded by the SB sequences was higher than that achieved by the TB sequences. Comparing with M-JPEG, the CRs obtained by AVC and HEVC were increased by 58% and 62% (sagittal), 16% and 23% (coronal), and 48% and 56% (transverse), respectively. Among the 3 video encoders, the highest CRs and restoration accuracy were achieved by HEVC. HEVC with inter-frame coding is more effective in reducing the redundant information in consecutive images. It is feasible to implement the video encoder for high-performance cine-MRI compression.

On-board MRI image compression using video encoder for MR-guided radiotherapy.

Magnetic resonance imaging (MRI) is currently used for online target monitoring and plan adaptation in modern image-guided radiotherapy. However, storing a large amount of data accumulated during patient treatment becomes an issue. In this study, the feasibility to compress MRI images accumulated in MR-guided radiotherapy using video encoders was investigated. Two sorting algorithms were employed to reorder the slices in multiple MRI sets for the input sequence of video encoder. Three cropping algorithms were used to auto-segment regions of interest for separate data storage. Four video encoders, motion-JPEG (M-JPEG), MPEG-4 (MP4), Advanced Video Coding (AVC or H.264) and High Efficiency Video Coding (HEVC or H.265) were investigated. The compression performance of video encoders was evaluated by compression ratio and time, while the restoration accuracy of video encoders was evaluated by mean square error (MSE), peak signal-to-noise ratio (PSNR), and video quality matrix (VQM). The performances of all combinations of video encoders, sorting methods, and cropping algorithms were investigated and their effects were statistically analyzed. The compression ratios of MP4, H.264 and H.265 with both sorting methods were improved by 26% and 5%, 42% and 27%, 72% and 43%, respectively, comparing to those of M-JPEG. The slice-prioritized sorting method showed a higher compression ratio than that of the location-prioritized sorting method for MP4 (P=0.00000), H.264 (P=0.00012) and H.265 (P=0.00000), respectively. The compression ratios of H.265 were improved significantly with the applications of morphology algorithm (P=0.01890 and P=0.00530), flood-fill algorithm (P=0.00510 and P=0.00020) and level-set algorithm (P=0.02800 and P=0.00830) for both sorting methods. Among the four video encoders, H.265 showed the best compression ratio and restoration accuracy. The compression ratio and restoration accuracy of video encoders using inter-frame coding (MP4, H.264 and H.265) were higher than that of video encoders using intra-frame coding (M-JPEG). It is feasible to implement video encoders using inter-frame coding for high-performance MRI data storage in MR-guided radiotherapy.

Wavelet-Based Fast Decoding of 360 ° Videos.

In this paper, we propose a wavelet-based video codec specifically designed for VR displays that enables real-time playback of high-resolution 360° videos. Our codec exploits the fact that only a fraction of the full 360° video frame is visible on the display at any time. To load and decode the video viewport-dependently in real time, we make use of the wavelet transform for intra- as well as inter-frame coding. Thereby, the relevant content is directly streamed from the drive, without the need to hold the entire frames in memory. With an average of 193 frames per second at 8192 × 8192 -pixel full-frame resolution, the conducted evaluation demonstrates that our codec's decoding performance is up to 272% higher than that of the state-of-the-art video codecs H.265 and AV1 for typical VR displays. By means of a perceptual study, we further illustrate the necessity of high frame rates for a better VR experience. Finally, we demonstrate how our wavelet-based codec can also directly be used in conjunction with foveation for further performance increase.

Consistent Quality Oriented Rate Control in HEVC Via Balancing Intra and Inter Frame Coding

Consistent quality oriented rate control in video coding has attracted much more attention. However, the existing efforts only focus on decreasing variations between every two adjacent frames, but neglect coding trade-off problem between intra and inter frames. In this paper, we deal with it from a new perspective, where intra frame quantization parameter (IQP) and rate control are optimized for balanced coding. First, due to the importance of intra frames, a new framework is proposed for consistent quality oriented IQP prediction, and then we remove unqualified IQP candidates using the proposed penalty term. Second, we extensively evaluate possible features, and select target bits per pixel for all remaining frames, average and standard variance of frame QPs, where equivalent acquisition methods for QP features are given. Third, predicted IQPs are clipped effectively according to bandwidth and previous information for better bit rate accuracy. Compared with High Efficiency Video Coding (HEVC) reference baseline, experiments demonstrate that our method reduces quality fluctuation greatly by 37.2% on frame-level standard variance of peak-signal-noise-ratio (PSNR) and 45.1% on that of structural similarity (SSIM). Moreover, it also can have satisfactory results on Rate-Distortion (R-D) performance, bit accuracy and buffer control.

Neural Reference Synthesis for Inter Frame Coding

This work proposes the neural reference synthesis (NRS) to generate high-fidelity reference block for motion estimation and motion compensation (MEMC) in inter frame coding. The NRS is comprised of two submodules: one for reconstruction enhancement and the other for reference generation. Although numerous methods have been developed in the past for these two submodules using either handcrafted rules or deep convolutional neural network (CNN) models, they basically deal with them separately, resulting in limited coding gains. By contrast, the NRS proposes to optimize them collaboratively. It first develops two CNN-based models, namely EnhNet and GenNet. The EnhNet only uses spatial correlations within the current frame for reconstruction enhancement and the GenNet is then augmented by further aggregating temporal correlations across multiple frames for reference synthesis. However, a direct concatenation of EnhNet and GenNet without considering the complex temporal reference dependency across inter frames would implicitly induce iterative CNN processing and cause the data overfitting problem, leading to visually-disturbing artifacts and oversmoothed pixels. To tackle this problem, the NRS applies a new training strategy to coordinate the EnhNet and GenNet for more robust and generalizable models, and also devises a lightweight multi-level R-D (rate-distortion) selection policy for the encoder to adaptively choose reference blocks generated from the proposed NRS model or conventional coding process. Our NRS not only offers state-of-the-art coding gains, e.g., >10% BD-Rate (Bjøntegaard Delta Rate) reduction against the High Efficiency Video Coding (HEVC) anchor for a variety of common test video sequences encoded at a wide bit range in both low-delay and random access settings, but also greatly reduces the complexity relative to existing learning-based methods by utilizing more lightweight DNNs. All models are made publicly accessible at https://github.com/IVC-Projects/NRS for reproducible research.

Region-level bit allocation for rate control of 360-degree videos using cubemap projection

Featuring with more uniform sampling density in the sphere domain and less non-uniform geometric deformations in the planar domain, variants of cubemap projection (CMP) format enable the higher compression ratio in on-going 360-degree video coding standardization. Different from single-view videos, 360-degree CMP videos feature with content discontinuity combined with the abrupt change of motion vectors between some adjacent faces. However, there is few bit allocation scheme designed for rate control of video coding of CMP format. Thus, this paper proposes a region-level bit allocation scheme for rate control of interframe coding of CMP format. The proposed scheme consists of two parts. The first part is machine learning based high HEVC coding cost region detection for individual faces, where the feature descriptor of a CTU consists of the face based texture complexity, motion magnitude, motion density, and temporal coherence of motion vector. The second part is fitting function based region-level bit allocation. Different from previous work, bits are assigned to the high coding cost region and non-high coding cost region in individual faces of CMP format. Experimental results indicate that the proposed scheme achieves higher bitrate accuracy and larger BD-WS-PSNR compared with the original rate control scheme of the reference software of HEVC, HM16.16 with the 360Lib.

DDCA: A Distortion Drift-Based Cost Assignment Method for Adaptive Video Steganography in the Transform Domain

Cost assignment plays a key role in coding performance and security of video steganography. Existing cost assignment methods (for adaptive video steganography) are designed for specific transform coefficients rather than all transform coefficients. In addition, existing video steganographic frameworks do not allow Syndrome-Trellis Codes (STCs) to modify all transform coefficients in both intra-coded and inter-coded frames at the same time. To address these limitations, in this article, we first propose a novel video steganographic framework. Then, we give a theoretical analysis of distortion drift in both intra- and inter-coding procedures. Based on the analysis, we design a Distortion Drift-Based Cost Assignment method, hereafter referred to as DDCA. DDCA considers the inner-block, inter-block and inter-frame distortion costs in order to improve the coding performance and the security of stego videos when the embedding payload is fixed. We conducted extensive experiments using two video datasets to evaluate the proposed video steganographic framework and DDCA, in terms of the coding performance and the security. Our experiments show that the proposed framework outperforms three recent state-of-the-art methods, for example the coding performance and the security of stego videos can benefit from DDCA by making full use of all nonzero transform coefficients.

Detection of transcoded HEVC videos based on in-loop filtering and PU partitioning analyses

With the increasing maturity of video editing technology, forgers are more inclined to transcode videos to High Efficiency Video Coding (HEVC) videos, as HEVC not only enables people to enjoy high-definition videos but also allows broadcasters to stream it more efficiently across networks. Therefore, to verify the originality and authenticity, it is of great significance to propose an algorithm for detecting transcoded HEVC videos. In this paper, a theoretical model of video transcoding is first constructed, and a novel transcoding detection algorithm based on In-loop Filtering and Prediction Units (PU) Partition (IFPP) is proposed. By analyzing the statistical characteristics of strong and normal filtering modes in deblocking filtering and calculating offset values in Sample Adaptive Offset (SAO) filtering, the transcoding traces in inter-coded frames can be captured. In addition, PU partition statistics are also extracted to make full use of traces in intra-coded frames. By fusing these subfeatures, the proposed IFPP feature with 17 dimensions can be obtained, which is further fed to the Support Vector Machine (SVM) classifier. Finally, experiments are conducted on datasets with various coding parameters. Results show that the proposed algorithm outperforms existing algorithms and has better robustness.

An Effective CU Depth Decision Method for HEVC Using Machine Learning

This paper presents an effective machine learning-based depth selection algorithm for CTU (Coding Tree Unit) in HEVC (High Efficiency Video Coding). Existing machine learning methods are limited in their ability in handling the initial depth decision of CU (Coding Unit) and selecting the proper set of input features for the depth selection model. In this paper, we first propose a new classification approach for the initial division depth prediction. In particular, we study the correlation of the texture complexity, QPs (quantization parameters) and the depth decision of the CUs to forecast the original partition depth of the current CUs. Secondly, we further aim to determine the input features of the classifier by analysing the correlation between depth decision of the CUs, picture distortion and the bit-rate. Using the found relationships, we also study a decision method for the end partition depth of the current CUs using bit-rate and picture distortion as input. Finally, we formulate the depth division of the CUs as a binary classification problem and use the nearest neighbor classifier to conduct classification. Our proposed method can significantly improve the efficiency of inter-frame coding by circumventing the traversing cost of the division depth. It shows that the mentioned method can reduce the time spent by 34.56% compared to HM-16.9 while keeping the partition depth of the CUs correct.

Context-Adaptive Inverse Quantization for Inter-Frame Coding

In the hybrid video coding framework, quantization is the key technique to achieve lossy compression. The information loss caused by the quantization may be reduced to improve compression efficiency, by using either encoder-side rate-distortion optimized quantization or decoder-side filtering. Nonetheless, the existing studies did not extensively use the already encoded information, i.e., context, to reduce the quantization loss. We address this issue and propose a context-adaptive inverse quantization method, namely, CAIQ. Specifically, for inter-frame coding, we analyze the correlation between the prediction signal (generated by motion compensated prediction) and the residual signal, as well as the correlation within the residual signal itself. We then present linear as well as nonlinear yet lightweight models to exploit the observed correlations in the frequency domain. Our models provide an optional inverse quantization mode by referring to the prediction signal, which is available at the decoder side. Next, block-level mode selection regarding the CAIQ method is used at the encoder side. We integrate the proposed CAIQ method into the reference software of Versatile Video Coding. We perform an extensive study of the models and analyze their resulting compression efficiency gain and encoding/decoding complexity. Experimental results show that our CAIQ method improves compression performance especially for high-resolution videos and at high bit rates.

Temporal Masking with Luma Adjusted Interframe Coding for Underwater Exploration Using Acoustic Channel

Transmission of video information from a submerged sensory source may lead to numerous applications such as surveillance, deep sea exploration, monitoring and repairing of underwater oil pipes etc. Such application requires a real time video transmission. It is obviously a challenging task to achieve efficient underwater video transmission because of the limited channel bandwidth and water is not a uniform medium due to which the transmitted signal is more error prone. Due to the limited spectrum, transmitted information must be compressed in a way that its quality does not deteriorate and at the same time, it must achieve low bit rate. In this regard, we propose an algorithm of temporal masking with Luma adjustment to reduce the video information for efficient transmission and reduce the need of over compression. Proposed algorithm is applied on the moving picture expert group (MPEG) video standard. A 2D-least mean squared adaptive filter (LMSAF) is embedded at the receiver end in order to reduce the effect of aquatic noises and further compensate the overall quality. Motion in the video sequence is exploited to extract motion vectors for temporal masking and also to achieve desired bit rate for acoustic applications. Structural Similarity Index Matrix (SSIM) acquired by implementing the proposed algorithm is 0.908 while the standard H.264/AVC was 0.925, but the compression ratio was reduced to 5.03 in comparison to 7.33, which clearly shows the significance of the proposed algorithm in reducing the information size.

Motion vector modification distortion analysis-based payload allocation for video steganography

Video steganography forms a covert communication channel by data embedding in cover elements. To consider inter-frame mutual embedding impacts, this paper proposes a payload allocation strategy in video steganography based on motion vector modification distortion analysis. Firstly, the motion vector modification distortion caused by data embedding is analyzed. Then, a rate–distortion model reflecting the residue deviation propagation in successive inter-coded frames is derived. According to this model, the residue deviation propagation weight of each inter-coded frame can be computed. Finally, an inter-frame payload allocation strategy is designed in order to restrain the residue deviation propagation. Experimental results demonstrate that the proposed payload allocation strategy can enhance existing motion vector-based video steganographic methods in terms of undetectability and video coding performance. Besides, the lower computational complexity can be achieved.

1D representation of Laplacian eigenmaps and dual k ‐nearest neighbours for unified video coding

This study proposes a framework of video coding based on Laplacian eigenmaps (LEM) and its related embedding and reconstruction algorithm (ERA). Firstly, a one-dimensional (1D) representation of LEM is adopted to achieve an extremely low bit per pixel (BPP). Secondly, dual k-nearest neighbours, which keeps neighbour relationships both in high-dimensional data space and low-dimensional representation space and overcomes the disadvantage of classical non-linear dimensionality reduction methods which cannot preserve the neighbour properties in both of the spaces, based ERA of LEM is employed to gain extraordinarily high peak-signal-to-noise ratio (PSNR). Thirdly, a unified framework of video coding is fit for intra-frame, inter-frame and multi-view video coding. Finally, it is evaluated by simulation experiments that, in the situation of low bitrate transmission, the proposed method can attain better performance of BPP and PSNR than that of the state-of-the-art methods, such as highly efficient video coding.

Recent Advances on HEVC Inter-Frame Coding: From Optimization to Implementation and Beyond

High Efficiency Video Coding (HEVC) has doubled the video compression ratio with equivalent subjective quality as compared to its predecessor H.264/AVC. The significant coding efficiency improvement is attributed to many new techniques. Inter-frame coding is one of the most powerful yet complicated techniques therein and has posed high computational burden thus main obstacle in HEVC-based real-time applications. Recently, plenty of research has been done to optimize the inter-frame coding, either to reduce the complexity for real-time applications, or to further enhance the encoding efficiency. In this paper, we provide a comprehensive review of the state-of-the-art techniques for HEVC inter-frame coding from three aspects, namely fast inter coding solutions, implementation on different hardware platforms as well as advanced inter coding techniques. More specifically, different algorithms in each aspect are further subdivided into sub-categories and compared in terms of pros, cons, coding efficiency and coding complexity. To the best of our knowledge, this is the first such comprehensive review of the recent advances of the inter-frame coding for HEVC and hopefully it would help the improvement, implementation and applications of HEVC as well as the ongoing development of the next generation video coding standard.

Variable-Length Coding With Shared Incremental Redundancy: Design Methods and Examples

Variable-length (VL) coding with feedback is a commonly used technique that can approach point-to-point Shannon channel capacity with a significantly shorter average codeword length than fixed-length coding without feedback. This paper uses the inter-frame coding of Zeineddine and Mansour, originally introduced to address varying channel-state conditions in broadcast wireless communication, to approach capacity on point-to-point channels using VL codes without feedback. The per-symbol complexity is comparable to decoding the VL code with feedback (plus the additional complexity of a small peeling decoder amortized over many VL codes) and presents the opportunity for encoders and decoders that utilize massive parallel processing, where each VL decoder can process simultaneously. This paper provides an analytical framework and a design process for the degree distribution of the inter-frame code that allows the feedback-free system to achieve 96% or more of the throughput of the original VL code with feedback. As examples of VL codes, we consider non-binary (NB) low-density parity-check (LDPC), binary LDPC, and convolutional VL codes. The NB-LDPC VL code with an 8-bit CRC and an average codeword length of 336 bits achieves 85% of capacity with four rounds of ACK/NACK feedback. The proposed scheme using shared incremental redundancy without feedback achieves 97% of that performance or 83% of the channel capacity.

Efficient Mobile Video Streaming via Context-Aware RaptorQ-Based Unequal Error Protection

Mobile video streaming systems typically apply the forward error correction (FEC) at the application layer to cope with packet-level transmission errors, which complements the bit-level correction mechanisms at the physical layer. However, most existing works fail to exploit the block-level dependencies in both intra and interframe coding modes of a single-layer compressed video, and thus are less efficient for the prevailing H.264/AVC and/or H.265/HEVC compatible single-layer video application. To this end, we propose a low-complexity FEC, i.e., context-aware RaptorQ (CA-RQ) with unequal error protection (UEP), to improve the error recovery performance of the single-layer mobile video streaming, through incorporating the block-level dependencies in the compressed video data. We use a packet-level video transmission distortion model that considers the dependencies in both spatial and temporal domains, to quantify the importance of video packets within a group of pictures (GoP). The compressed video packets are categorized and grouped into several classes according to their importance to construct the CA-RQ code with the UEP property. We provide a theoretical analysis on redundancy allocation bounds to demonstrate the superior performance of proposed CA-RQ over the standard RaptorQ code. In the meantime, extensive simulations have shown that our scheme not only offers much better subjective visual quality with less than 50% additional redundant symbols as compared to the Macroblock-Based UEP (MB-UEP) scheme, but also outperforms the MB-UEP and classical equal error protection (EEP)-based schemes, by a 0.45% $\sim$ 5.71% and 0.94% $\sim$ 6.78% margin, respectively, in reconstructed quality evaluated using the structural similarity (SSIM) index, across a reasonable range of redundancy proportions.

SenseVid : A traffic trace based tool for QoE Video transmission assessment dedicated to Wireless Video Sensor Networks

Video applications are being a key component to enhance traditional wireless sensor networks (WSN) applications. As a result, sensor network researchers need adequate and easy to use tools to assess the performances of their proposals. The existing transmission evaluation tools either consider video sequences along with codecs that are unsuitable for WSNs or make use of low cost compression methods for still images without inter-frame coding required for efficient video transmission. In this paper, we present SenseVid, an open source video transmission and evaluation tool that considers WSN specific characteristics. Besides low energy intra-frame compression based on fast pruned discrete cosine transform (DCT), a low complexity inter-frame encoding is provided to allow efficient support of video flows. A configurable fine-grain energy model is provided where both video capture and encoding cost are accounted for on a per frame basis. Video traffic differentiation based on priority levels is also provided. SenseVid adopts the video traffic traces approach, allowing its use in any simulation or real testbed environment. Using SenseVid, the user is able to reconstruct the received video considering lost packets during its transmission as well as estimating the achieved quality of service (QoS) and quality of experience (QoE).