Intra-frame Coding Research Articles

CC-SMC: Chain coding-based segmentation map lossless compression

A segmentation map, either static or dynamic, refers to a two-dimensional picture that may vary with time and indicates the segmentation label per pixel. Both the semantic map and the occupancy map in video-based point cloud compression (V-PCC) belong to the segmentation map we referred to. The semantic map can work for many machine vision tasks like tracking and has been used as a layer of image representation in some image compression methods. The occupancy map constitutes a part of the point cloud coding bitstream. Since segmentation maps are widely used, how to efficiently compress them is of interest. We propose a segmentation map lossless compression scheme namely CC-SMC, exploiting the nature of segmentation maps that usually contain limited colors and sharp edges. Specifically, we design a chain coding-based scheme combined with quadtree-based block partitioning. For intraframe coding, one block is partitioned recursively with a quadtree structure, until the block contains only one color, is smaller than a threshold, or satisfies the defined chain coding condition. We revise the three-orthogonal chain coding method to incorporate contextual information and design effective intraframe prediction methods. For interframe coding, one block may find a reference block; the chain difference between the current and the reference blocks is coded. We implement the proposed scheme and test it on several different kinds of segmentation maps. Compared with advanced lossless image compression techniques, our proposed scheme obtains more than 10% bits reduction as well as more than 20% decoding time-saving. The code is available at https://github.com/Yang-Runyu/CC-SMC.

Perceptual Quality-Oriented Rate Allocation via Distillation from End-to-End Image Compression

Mainstream image/video coding standards, exemplified by the state-of-the-art H.266/VVC, AVS3, and AV1, follow the block-based hybrid coding framework. Due to the block-based framework, encoders designed for these standards are easily optimized for peak signal-to-noise ratio (PSNR) but have difficulties optimizing for the metrics more aligned to perceptual quality, e.g., multi-scale structural similarity (MS-SSIM), since these metrics cannot be accurately evaluated at the small block level. We address this problem by leveraging inspiration from the end-to-end image compression built on deep networks, which is easily optimized through network training for any metric as long as the metric is differentiable. We compared the trained models using the same network structure but different metrics and observed that the models allocate rates in different ratios. We then propose a distillation method to obtain the rate allocation rule from end-to-end image compression models with different metrics and to utilize such a rule in the block-based encoders. We implement the proposed method on the VVC reference software—VTM and the AVS3 reference software—HPM, focusing on intraframe coding. Experimental results show that the proposed method on top of VTM achieves more than 10% BD-rate reduction than the anchor when evaluated with MS-SSIM or LPIPS, which leads to concrete perceptual quality improvement.

A convolutional neural network-based rate control algorithm for VVC intra coding

The Versatile Video Coding (VVC) has shown significant improvements in Rate-Distortion (R-D) performance compared to its predecessor, High Efficiency Video Coding (HEVC). However, it still encounters several challenges. One of these challenges is the efficient allocation of bits among all Coding Tree Units (CTUs). Additionally, there is a lack of prior information for intra-frame coding, particularly for the first frame. After CTU-level bit allocation, only fixed parameters can be used to determine the λ for CTUs, which does not result in optimal rate–distortion performance. To tackle above challenges, we propose a rate control solution based on Convolutional Neural Network (CNN). This approach utilizes CNN to predict the key parameters α and β in the R-D model, addressing the problem of lacking prior information in intra-frame coding. Subsequently, the predicted α and β values are used to adaptively allocate bits for each CTU. Our proposed algorithm is implemented in VTM-16.0 under Common Test Conditions (CTC). Experimental results show that, compared to the default rate control algorithm in VTM-16.0, our proposed algorithm enhances R-D performance by 0.96% while maintaining rate control accuracy.

Audience-oriented Aesthetic and Creative Research on Animated Films Based on Visual Communication Design

Abstract Animation movie conveys the connotation of animation to the audience through visual symbols, and the audience-oriented animation movie improves the aesthetic ability and diversifies the creative design. This paper starts with the analysis of visual communication design technology in animated films, analyzes the principle of visual communication design of animated images, and realizes the production of animated films by using multi-visual 3D animation modeling and texture rendering technology. The Sobel edge operator is introduced to optimize the image edge data of animated movies, so as to improve the image quality of animated movies. The development trend of visual communication design is discussed, and the aesthetic characteristics of visual communication design for animated films and the mutual influence between them are analyzed. In order to explore the aesthetic expression and creative design methods of animated films, quantitative analysis of data was conducted using current animated film works. The results show that the bit code rate in the full intra-frame coding mode of optimized production of animated images using visual communication technology is −1.87%, the highest quality factor is around 0.95, and the variation of smoothness is kept between 80% and 90%. Audience-oriented visual communication design of animated films should be based on the utility of information conveyance and rhythmic rhyme design and prompt the creative design of animated films through association, whimsy, metaphor, style and other ways.

Fast CU Partition Algorithm for Intra Frame Coding Based on Joint Texture Classification and CNN.

High-efficiency video coding (HEVC/H.265) is one of the most widely used video coding standards. HEVC introduces a quad-tree coding unit (CU) partition structure to improve video compression efficiency. The determination of the optimal CU partition is achieved through the brute-force search rate-distortion optimization method, which may result in high encoding complexity and hardware implementation challenges. To address this problem, this paper proposes a method that combines convolutional neural networks (CNN) with joint texture recognition to reduce encoding complexity. First, a classification decision method based on the global and local texture features of the CU is proposed, efficiently dividing the CU into smooth and complex texture regions. Second, for the CUs in smooth texture regions, the partition is determined by terminating early. For the CUs in complex texture regions, a proposed CNN is used for predictive partitioning, thus avoiding the traditional recursive approach. Finally, combined with texture classification, the proposed CNN achieves a good balance between the coding complexity and the coding performance. The experimental results demonstrate that the proposed algorithm reduces computational complexity by 61.23%, while only increasing BD-BR by 1.86% and decreasing BD-PSNR by just 0.09 dB.

On-board MRI image compression using video encoder for MR-guided radiotherapy.

Magnetic resonance imaging (MRI) is currently used for online target monitoring and plan adaptation in modern image-guided radiotherapy. However, storing a large amount of data accumulated during patient treatment becomes an issue. In this study, the feasibility to compress MRI images accumulated in MR-guided radiotherapy using video encoders was investigated. Two sorting algorithms were employed to reorder the slices in multiple MRI sets for the input sequence of video encoder. Three cropping algorithms were used to auto-segment regions of interest for separate data storage. Four video encoders, motion-JPEG (M-JPEG), MPEG-4 (MP4), Advanced Video Coding (AVC or H.264) and High Efficiency Video Coding (HEVC or H.265) were investigated. The compression performance of video encoders was evaluated by compression ratio and time, while the restoration accuracy of video encoders was evaluated by mean square error (MSE), peak signal-to-noise ratio (PSNR), and video quality matrix (VQM). The performances of all combinations of video encoders, sorting methods, and cropping algorithms were investigated and their effects were statistically analyzed. The compression ratios of MP4, H.264 and H.265 with both sorting methods were improved by 26% and 5%, 42% and 27%, 72% and 43%, respectively, comparing to those of M-JPEG. The slice-prioritized sorting method showed a higher compression ratio than that of the location-prioritized sorting method for MP4 (P=0.00000), H.264 (P=0.00012) and H.265 (P=0.00000), respectively. The compression ratios of H.265 were improved significantly with the applications of morphology algorithm (P=0.01890 and P=0.00530), flood-fill algorithm (P=0.00510 and P=0.00020) and level-set algorithm (P=0.02800 and P=0.00830) for both sorting methods. Among the four video encoders, H.265 showed the best compression ratio and restoration accuracy. The compression ratio and restoration accuracy of video encoders using inter-frame coding (MP4, H.264 and H.265) were higher than that of video encoders using intra-frame coding (M-JPEG). It is feasible to implement video encoders using inter-frame coding for high-performance MRI data storage in MR-guided radiotherapy.

Time Delay Optimization of Compressing Shipborne Vision Sensor Video Based on Deep Learning

As the technology for offshore wireless transmission and collaborative innovation in unmanned ships continues to mature, research has been gradually carried out in various countries on methods of compressing and transmitting perceptual video while driving ships remotely. High Efficiency Video Coding (H.265/HEVC) has played an extremely important role in the field of Unmanned Aerial Vehicle (UAV) and autopilot, and as one of the most advanced coding schemes, its performance in compressing visual sensor video is excellent. According to the characteristics of shipborne vision sensor video (SVSV), optimizing the coding aspects with high computational complexity is one of the important methods to improve the video compression performance. Therefore, an efficient video coding technique is proposed to improve the efficiency of SVSV compression. In order to optimize the compression performance of SVSV, an intra-frame coding delay optimization algorithm that works in the intra-frame predictive coding (PC) session by predicting the Coding Unit (CU) division structure in advance is proposed in combination with deep learning methods. The experimental results show that the total compression time of the algorithm is reduced by about 45.49% on average compared with the official testbed HM16.17 for efficient video coding, while the Bjøntegaard Delta Bit Rate (BD-BR) increased by an average of 1.92%, and the Peak Signal-to-Noise Ratio (BD-PSNR) decreased by an average of 0.14 dB.

Partition Map Prediction for Fast Block Partitioning in VVC Intra-Frame Coding

The Versatile Video Coding (VVC) standard introduces a block partitioning structure known as quadtree plus nested multi-type tree (QTMTT), which allows more flexible block partitioning compared to its predecessors, like High Efficiency Video Coding (HEVC). Meanwhile, the partition search (PS) process, which is to find out the best partitioning structure for optimizing the rate-distortion cost, becomes far more complicated for VVC than for HEVC. Also, the PS process in VVC reference software (VTM) is not friendly to hardware implementation. We propose a partition map prediction method for fast block partitioning in VVC intra-frame encoding. The proposed method may replace PS totally or be combined with PS partially, thereby achieving adjustable acceleration of the VTM intra-frame encoding. Different from the previous methods for fast block partitioning, we propose to represent a QTMTT-based block partitioning structure by a partition map, which consists of a quadtree (QT) depth map, several multi-type tree (MTT) depth maps, and several MTT direction maps. We then propose to predict the optimal partition map from the pixels through a convolutional neural network (CNN). We propose a CNN structure, known as Down-Up-CNN, for the partition map prediction, where the CNN structure emulates the recursive nature of the PS process. Moreover, we design a post-processing algorithm to adjust the network output partition map, so as to obtain a standard-compliant block partitioning structure. The post-processing algorithm may produce a partial partition tree as well; then based on the partial partition tree, the PS process is performed to obtain the full tree. Experimental results show that the proposed method achieves 1.61× to 8.64× encoding acceleration for the VTM-10.0 intra-frame encoder, with the ratio depending on how much PS is performed. Especially, when achieving 3.89× encoding acceleration, the compression efficiency loss is 2.77% in BD-rate, which is a better tradeoff than the previous methods.

A Fast CU Partition Algorithm Based on Gradient Structural Similarity and Texture Features

The H.266/Versatile Video Coding (VVC) standard poses a great challenge for encoder design due to its high computational complexity and long encoding time. In this paper, the fast partitioning decision of coding blocks is investigated to reduce the computational complexity and save the coding time of VVC intra-frame predictive coding. A fast partitioning algorithm of VVC intra-frame coding blocks based on gradient structure similarity and directional features is proposed. First, the average gradient structure similarity of four sub-coding blocks under the current coding block is calculated, and two thresholds are set to determine whether the current coding block terminates the partitioning early or performs quadtree partitioning. Then, for the coding blocks that do not satisfy the above thresholds, the standard deviation of the vertical and horizontal directions of the current coding block is calculated to determine the texture direction and skip unnecessary partitioning to reduce computational complexity. Based on the VTM10.0 platform, this paper evaluates the performance of the designed fast algorithm for partitioning within the VVC coding unit. Compared with VTM10.0, the encoding rate is improved by 1.38% on average, and the encoder execution time is reduced by 49.32%. The overall algorithm achieves a better optimization of the existing VVC intra-frame coding technique.

Relationship between families and kindergartens in Hungary in the 1950s

Partnership working and co-operation between parents and early education and care settings is a widely researched topic worldwide. However, little is known about how the relationship between families and kindergartens developed in the historically significant period of the 1950s in Hungary, which marks the beginnings of socialism and a period of rapid expansion of early childhood education and care. This study aims to explore how the expectations of raising ‘socialist citizens’ was incorporated into educational and policy documents and other written resources relating to kindergarten and family education. Purposive sampling selection identified 80 documents as data sources, which were subjected to qualitative content analysis. Intra-frame coding was done by hand using a combined inductive and deductive approach. Employing a constructivist theoretical lens, the analysis focused on both the manifest and latent content of the selected documents and resulted in seven main themes. The findings confirmed that the ideologically driven policy decisions not only influenced the relationship between families and kindergartens but also legitimised the efforts to build a socialist system of early education through organised collaboration and related propaganda work. This study is significant as it offers a more nuanced understanding of the relationship between kindergartens and families in the ‘50s and with that provides foundations for further analytical work of the socialist pedagogical past.

A Fast Decision Algorithm for VVC Intra-Coding Based on Texture Feature and Machine Learning.

Due to the development and application of information technology, a series of modern information technologies represented by 5G, big data, and artificial intelligence are changing rapidly, and people's requirements for video coding standards have become higher. In the High-Efficiency Video Coding (HEVC) standard, the coding block division is not flexible enough, and the prediction mode is not detailed enough. A new generation of Versatile Video Coding (VVC) standards was born. VVC inherits the hybrid coding framework adopted by HEVC, improves the original technology of each module, introduces a series of new coding technologies, and builds on this greatly improving the coding efficiency. Compared with HEVC, the block division structure of VVC has undergone great changes, retaining the quad-tree (QT) division method and increasing the multi-type tree (MTT) division method, which brings high coding complexity. To reduce the computational complexity of VVC coding block division, a fast decision algorithm for VVC intra-frame coding based on texture characteristics and machine learning is proposed. First, we analyze the characteristics of the CU partition structure decision and then use the texture complexity of the CU partition structure decision to terminate the CU partition process early; for CUs that do not meet the early termination of the partition, use the global sample information, local sample information, and context information. The three-category feature-trained tandem classifier framework predicts the division type of CU. The experimental results show that in the full intra mode, compared with the existing VTM10.0, the encoding output bit rate is increased by 1.36%, and the encoding time is saved by 52.63%.

Error Concealment in the Density Field of a Spatiotemporal Image Sequence.

One of the most difficult challenges of multimedia transmission during the last two decades has been the retrieval of degraded or missing regions of images and videos while maintaining satisfactory perceptual accuracy. The objective is to retrieve lost data by using the similarity between frames. Usually, error concealment (EC) schemes depend on replacing incorrect data with data that are identical to the initial. This is possible because video contains a high degree of self-similarity. This research focuses on applying an EC approach in transform-domain video sequences. To conduct EC on films, they must first be translated to frames and then transformed using one of the available transformations into frequency-domain images. Using successive frames, it is possible to recover lost or incorrect data from images. Intra-coded frames (I-frames) may be used to recreate lost knowledge in predictive (P-frames) and bidirectional predictive frames (B-frames). I-frame knowledge that has been lost may be restored using previous intra-coded frames. The use of wavelet error concealment generated more precise results than the other techniques. In this study, it was discovered that covering faults in the density sector with wavelets produces more reliable results than the other techniques.

Viewport-Dependent Delivery for Conversational Immersive Video

Real-time immersive video has been an area of interest for the research and standardization community over the past few years. It has gained particular importance in the era of 5G services. 3GPP (Third Generation Partnership Project) Release 17 includes Immersive teleconferencing for Multimedia Telephony services over IP Multimedia System. Viewport-dependent delivery (VDD) is a mechanism for bandwidth savings by assigning more bits to the part of the 360-degree video currently in the viewport and fewer or no bits to the areas of the 360-degree video that a user does not currently view. The technique has been extensively studied in the context of adaptive HTTP streaming of immersive video, where the 360-degree video is pre-encoded in different versions, and the appropriate version is selected and requested by the viewer’s client application, depending on their current viewport orientation. However, conversational video has more stringent latency requirements than HTTP streaming; the time from capture to rendering is in the order of a few milliseconds for a good user experience. Conversational immersive video also allows adapting the encoding during the media delivery session according to the user viewport and network bandwidth characteristics. This paper explores tile-based and viewport region encoding for VDD and how they apply to conversational video. In particular, we explore the challenges related to the absence of periodic intra-coded frames in conversational video. Based on simulation results, we show that VDD methods may not always be a performant alternative to viewport-independent delivery in conversational use cases. We also discuss possible encoder optimization that would improve the VDD performance.

Performance analysis of VVC intra coding

This article presents a performance analysis of Versatile Video Coding (VVC) intra-frame prediction. VVC is the next generation of video coding standards, which has been developed to supply the demand of upcoming video applications. VVC brings several innovations and enhancements for the intra-frame prediction to improve the encoding efficiency. These improvements comprise larger block sizes, more flexible block partitioning, more angular intra-frame prediction modes, multiple transform selection, non-separable secondary transform, among others. This article provides a detailed description of these tools, discussing how they work together in the intra-frame coding flow to raise the compression performance. Moreover, this article presents encoding complexity, encoding usage distribution, and rate-distortion-complexity analyses of the intra-frame prediction tools over different quantization scenarios. Based on these analyses, this article provides support for future works focusing on VVC intra-frame coding, including complexity reduction, complexity control, and real-time hardware design.

DDCA: A Distortion Drift-Based Cost Assignment Method for Adaptive Video Steganography in the Transform Domain

Cost assignment plays a key role in coding performance and security of video steganography. Existing cost assignment methods (for adaptive video steganography) are designed for specific transform coefficients rather than all transform coefficients. In addition, existing video steganographic frameworks do not allow Syndrome-Trellis Codes (STCs) to modify all transform coefficients in both intra-coded and inter-coded frames at the same time. To address these limitations, in this article, we first propose a novel video steganographic framework. Then, we give a theoretical analysis of distortion drift in both intra- and inter-coding procedures. Based on the analysis, we design a Distortion Drift-Based Cost Assignment method, hereafter referred to as DDCA. DDCA considers the inner-block, inter-block and inter-frame distortion costs in order to improve the coding performance and the security of stego videos when the embedding payload is fixed. We conducted extensive experiments using two video datasets to evaluate the proposed video steganographic framework and DDCA, in terms of the coding performance and the security. Our experiments show that the proposed framework outperforms three recent state-of-the-art methods, for example the coding performance and the security of stego videos can benefit from DDCA by making full use of all nonzero transform coefficients.

Detection of transcoded HEVC videos based on in-loop filtering and PU partitioning analyses

With the increasing maturity of video editing technology, forgers are more inclined to transcode videos to High Efficiency Video Coding (HEVC) videos, as HEVC not only enables people to enjoy high-definition videos but also allows broadcasters to stream it more efficiently across networks. Therefore, to verify the originality and authenticity, it is of great significance to propose an algorithm for detecting transcoded HEVC videos. In this paper, a theoretical model of video transcoding is first constructed, and a novel transcoding detection algorithm based on In-loop Filtering and Prediction Units (PU) Partition (IFPP) is proposed. By analyzing the statistical characteristics of strong and normal filtering modes in deblocking filtering and calculating offset values in Sample Adaptive Offset (SAO) filtering, the transcoding traces in inter-coded frames can be captured. In addition, PU partition statistics are also extracted to make full use of traces in intra-coded frames. By fusing these subfeatures, the proposed IFPP feature with 17 dimensions can be obtained, which is further fed to the Support Vector Machine (SVM) classifier. Finally, experiments are conducted on datasets with various coding parameters. Results show that the proposed algorithm outperforms existing algorithms and has better robustness.

Transform Network Architectures for Deep Learning Based End-to-End Image/Video Coding in Subsampled Color Spaces

Most of the existing deep learning based end-to-end image/video coding (DLEC) architectures are designed for non-subsampled RGB color format. However, in order to achieve a superior coding performance, many state-of-the-art block-based compression standards such as High Efficiency Video Coding (HEVC/H.265) and Versatile Video Coding (VVC/H.266) are designed primarily for YUV 4:2:0 format, where U and V components are subsampled by considering the human visual system. This paper investigates various DLEC designs to support YUV 4:2:0 format by comparing their performance against the main profiles of HEVC and VVC standards under a common evaluation framework. Moreover, a new transform network architecture is proposed to improve the efficiency of coding YUV 4:2:0 data. The experimental results on YUV 4:2:0 datasets show that the proposed architecture significantly outperforms naive extensions of existing architectures designed for RGB format and achieves about 10% average BD-rate improvement over the intra-frame coding in HEVC.

Efficient Depth Intra Frame Coding in 3D-HEVC by Corner Points.

To improve the coding performance of depth maps, 3D-HEVC includes several new depth intra coding tools at the expense of increased complexity due to a flexible quadtree Coding Unit/Prediction Unit (CU/PU) partitioning structure and a huge number of intra mode candidates. Compared to natural images, depth maps contain large plain regions surrounded by sharp edges at the object boundaries. Our observation finds that the features proposed in the literature either speed up the CU/PU size decision or intra mode decision and they are also difficult to make proper predictions for CUs/PUs with the multi-directional edges in depth maps. In this work, we reveal that the CUs with multi-directional edges are highly correlated with the distribution of corner points (CPs) in the depth map. CP is proposed as a good feature that can guide to split the CUs with multi-directional edges into smaller units until only single directional edge remains. This smaller unit can then be well predicted by the conventional intra mode. Besides, a fast intra mode decision is also proposed for non-CP PUs, which prunes the conventional HEVC intra modes, skips the depth modeling mode decision, and early determines segment-wise depth coding. Furthermore, a two-step adaptive corner point selection technique is designed to make the proposed algorithm adaptive to frame content and quantization parameters, with the capability of providing the flexible tradeoff between the synthesized view quality and complexity. Simulation results show that the proposed algorithm can provide about 66% time reduction of the 3D-HEVC intra encoder without incurring noticeable performance degradation for synthesized views and it also outperforms the previous state-of-the-art algorithms in term of time reduction and ∆ BDBR.

Retraction Note: Edge-Oriented Spatial Interpolation for Error Concealment of Consecutive Blocks

This paper proposes a low-complexity spatial-domain error concealment (EC) algorithm for recovering consecutive blocks error in still images or intra-coded (I) frames of video sequences. The proposed algorithm works with the following steps. Firstly the Sobel operator is performed on the top and bottom adjacent pixels to detect the most probable edge direction of current block area. After that one-dimensional (1-D) matching is used on the available block boundaries. Displacement between edge direction candidate and most probable edge direction is taken into consideration as an important factor to improve stability of 1-D boundary matching. Then the corrupted pixels are recovered by linear weighting interpolation along the estimated edge direction. Finally the interpolated values are merged to get last recovered picture. Simulation results demonstrate that the proposed algorithms obtain good subjective quality and higher PSNR than the methods in literatures for most images.

Frame-Wise CNN-Based Filtering for Intra-Frame Quality Enhancement of HEVC Videos

The paper proposes a novel frame-wise filtering method based on Convolutional Neural Networks (CNNs) for enhancing the quality of HEVC decoded videos. A novel deep neural network architecture is proposed for post-filtering the entire intra-coded videos. A novel scheme utilizing frame-size patches is employed for training the network. The proposed method filters the luminance channel separately from the pair of chrominance channels. A novel patch generation paradigm is proposed where, for each color channel, the corresponding mode map is generated based on the HEVC intra-prediction mode index and block segmentation. The proposed CNN-based filtering method is an alternative to the traditional HEVC built-in in-loop filtering module for intra-coded frames. Experimental results on standard test sequences show that the proposed method outperforms the HEVC standard with average BD-rate savings of 11.1% and an average BD-PSNR improvement of 0.602 dB. The average relative improvement in ΔPSNR is around 105% at QP = 42 and around 85% at QP = 32 compared with state-of-the-art machine-learning-based methods.