Rate-distortion Performance Research Articles

Nonlinear Transform Source-Channel Coding for Semantic Communications

In this paper, we propose a class of high-efficiency deep joint source-channel coding methods that can closely adapt to the source distribution under the nonlinear transform, it can be collected under the name nonlinear transform source-channel coding (NTSCC). In the considered model, the transmitter first learns a nonlinear analysis transform to map the source data into latent space, then transmits the latent representation to the receiver via deep joint source-channel coding. Our model incorporates the nonlinear transform as a strong prior to effectively extract the source semantic features and provide side information for source-channel coding. Unlike existing conventional deep joint source-channel coding methods, the proposed NTSCC essentially learns both the source latent representation and an entropy model as the prior on the latent representation. Accordingly, novel adaptive rate transmission and hyperprior-aided codec refinement mechanisms are developed to upgrade deep joint source-channel coding. The whole system design is formulated as an optimization problem whose goal is to minimize the end-to-end transmission rate-distortion performance under established perceptual quality metrics. Across test image sources with various resolutions, we find that the proposed NTSCC transmission method generally outperforms both the analog transmission using the standard deep joint source-channel coding and the classical separation-based digital transmission. Notably, the proposed NTSCC method can potentially support future semantic communications due to its content-aware ability and perceptual optimization goal.

Novel Projection Schemes for Graph-Based Light Field Coding.

In light field compression, graph-based coding is powerful to exploit signal redundancy along irregular shapes and obtains good energy compaction. However, apart from high time complexity to process high dimensional graphs, their graph construction method is highly sensitive to the accuracy of disparity information between viewpoints. In real-world light field or synthetic light field generated by computer software, the use of disparity information for super-rays projection might suffer from inaccuracy due to vignetting effect and large disparity between views in the two types of light fields, respectively. This paper introduces two novel projection schemes resulting in less error in disparity information, in which one projection scheme can also significantly reduce computation time for both encoder and decoder. Experimental results show projection quality of super-pixels across views can be considerably enhanced using the proposals, along with rate-distortion performance when compared against original projection scheme and HEVC-based or JPEG Pleno-based coding approaches.

Two-Stage Octave Residual Network for End-to-End Image Compression

Octave Convolution (OctConv) is a generic convolutional unit that has already achieved good performances in many computer vision tasks. Recent studies also have shown the potential of applying the OctConv in end-to-end image compression. However, considering the characteristic of image compression task, current works of OctConv may limit the performance of the image compression network due to the loss of spatial information caused by the sampling operations of inter-frequency communication. Besides, the correlation between multi-frequency latents produced by OctConv is not utilized in current architectures. In this paper, to address these problems, we propose a novel Two-stage Octave Residual (ToRes) block which strips the sampling operation from OctConv to strengthen the capability of preserving useful information. Moreover, to capture the redundancy between the multi-frequency latents, a context transfer module is designed. The results show that both ToRes block and the incorporation of context transfer module help to improve the Rate-Distortion performance, and the combination of these two strategies makes our model achieve the state-of-the-art performance and outperform the latest compression standard Versatile Video Coding (VVC) in terms of both PSNR and MS-SSIM.

A Novel Online Correlation Noise Model Based on Band Coefficients Mean to Achieve Low Computational and Coding-Efficient Distributed Video Codec

Distributed video coding (DVC) is a novel coding paradigm that offers low computational encoding relative to conventional video-coding framework at the expense of high-decoding computational complexity. The challenging part of this video-coding framework is achieving better rate-distortion (RD) compared with conventional codec performance. A suitable and accurate correlation noise model (CNM) is crucial in improving the RD performance by achieving high coding efficiency and making decoding less computationally demanding. Since the correlation is nonstationary and time-variant and can vary from frame to frame, offline CNM estimation is not feasible for practical applications and real-time decoding. An online CNM may be the solution to this problem. In DVC, neither Wyner–Ziv frame (WZF) nor estimated side information (SI) of the corresponding WZF is available at the encoder. Therefore, online estimation of the CNM and its parameters can be quite challenging. The contribution of this research work is a novel online CNM which is computed by taking the mean of each transformed coefficient band and deployed for two different codecs. Our proposed codec, DIVCOM, which stands for “Distributed Video Coding with Online Band Mean Correlation Noise Model”, outperforms the existing baseline codec, DISCOVER (DIS), in both coding efficiency and peak signal-to-noise ratio (PSNR). The DIVCOM codec achieves coding efficiency of up to 8.05 kbps, and PSNR ranges from 0.0245 dB to 0.18 dB. An extended version of DIVCOM incorporating phase-based side information called PDIVCOM achieves coding efficiency up to 10.9 kbps, and PSNR ranges from 0.019 to 0.17 dB compared to DIS.

Low-Complexity Adaptive Sampling of Block Compressed Sensing Based on Distortion Minimization.

Block compressed sensing (BCS) is suitable for image sampling and compression in resource-constrained applications. Adaptive sampling methods can effectively improve the rate-distortion performance of BCS. However, adaptive sampling methods bring high computational complexity to the encoder, which loses the superiority of BCS. In this paper, we focus on improving the adaptive sampling performance at the cost of low computational complexity. Firstly, we analyze the additional computational complexity of the existing adaptive sampling methods for BCS. Secondly, the adaptive sampling problem of BCS is modeled as a distortion minimization problem. We present three distortion models to reveal the relationship between block sampling rate and block distortion and use a simple neural network to predict the model parameters from several measurements. Finally, a fast estimation method is proposed to allocate block sampling rates based on distortion minimization. The results demonstrate that the proposed estimation method of block sampling rates is effective. Two of the three proposed distortion models can make the proposed estimation method have better performance than the existing adaptive sampling methods of BCS. Compared with the calculation of BCS at the sampling rate of 0.1, the additional calculation of the proposed adaptive sampling method is less than 1.9%.

Proportional–integral–derivative-based intra rate controller for low-delay applications of high efficiency video coding screen content coding

Most screen content (SC) video applications are real time, and a low-delay transmission is needed. An effective low-delay transmission requires a precise video rate controller to fit the video bit rate to the capacity of the transmission channel. The rate controller has to support the unique standard coding tools of SC video. We propose a unique intra rate controller based on a unique content-based rate-distortion model for low delay applications of high efficiency video coding SC coding. The proposed model supports the unique SC coding tools and different modes and sizes of coding tree units. The proportional–integral–derivative control concept is used for frame-level bit allocation and rate control. The rate controller utilizes a small size virtual buffer to provide low-delay buffering constraints for the bit stream. Experimental results show that the proposed rate controller can control the bit rate of test sequences with a low buffering delay (about 20 ms) while the rate-distortion performance is maintained.

Deep motion‐compensation enhancement in video compression

This work introduces the multiframe motion-compensation enhancement network (MMCE-Net), a deep-learning tool aimed at improving the performance of current video coding standards based on motion-compensation, such as H.265/HEVC. The proposed method improves the inter-prediction coding efficiency by enhancing the accuracy of the motion-compensated frame and thereby improving the rate-distortion performance. MMCE-Net is a neural network that jointly exploits the predicted coding unit and two co-located coding units from previous reference frames to improve the estimation of the temporal evolution of the scene. This letter describes the architecture of MMCE-Net, how it is integrated into H.265/HEVC and the corresponding performance.

Reversible Data Hiding in Encrypted Image via Secret Sharing Based on GF(p) and GF(2⁸)

Secret sharing is a useful method which divides a secret message into several shares for security. During the recovery procedure, only when sufficient shares are obtained, the secret message can be recovered. This paper proposes two novel reversible data hiding schemes in encrypted image via secret sharing over Galois fields <inline-formula> <tex-math notation="LaTeX">${{GF}}{({p})}$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">${{GF}}{(}{{2}^{{8}}}{)}$ </tex-math></inline-formula>. The content owner first applies a specific encryption method through block and pixel permutation and Shamir&#x2019;s secret sharing. Then, the theoretical demonstration is introduced to explain that the generated shares are suitable for data embedding over <inline-formula> <tex-math notation="LaTeX">${{GF}}{({p})}$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">${{GF}}{(}{{2}^{{8}}}{)}$ </tex-math></inline-formula>. Finally, two embedding algorithms over <inline-formula> <tex-math notation="LaTeX">${{GF}}{({p})}$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">${{GF}}{(}{{2}^{{8}}}{)}$ </tex-math></inline-formula> are presented, and on the receiver side, with different keys, additional data can be extracted correctly and original image can be recovered losslessly. Experimental results show that our schemes can achieve better rate-distortion performance than some state-of-the-art schemes.

Causal Contextual Prediction for Learned Image Compression

Over the past several years, we have witnessed impressive progress in the field of learned image compression. Recent learned image codecs are commonly based on autoencoders, that first encode an image into low-dimensional latent representations and then decode them for reconstruction purposes. To capture spatial dependencies in the latent space, prior works exploit hyperprior and spatial context model to build an entropy model, which estimates the bit-rate for end-to-end rate-distortion optimization. However, such an entropy model is suboptimal from two aspects: (1) It fails to capture global-scope spatial correlations among the latents. (2) Cross-channel relationships of the latents remain unexplored. In this paper, we propose the concept of separate entropy coding to leverage a serial decoding process for causal contextual entropy prediction in the latent space. A <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">causal context model</i> is proposed that separates the latents across channels and makes use of channel-wise relationships to generate highly informative adjacent contexts. Furthermore, we propose a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">causal global prediction model</i> to find global reference points for accurate predictions of undecoded points. Both these two models facilitate entropy estimation without the transmission of overhead. In addition, we further adopt a new group-separated attention module to build more powerful transform networks. Experimental results demonstrate that our full image compression model outperforms standard VVC/H.266 codec on Kodak dataset in terms of both PSNR and MS-SSIM, yielding the state-of-the-art rate-distortion performance.

Adaptive reversible data hiding for JPEG images with multiple two-dimensional histograms

Joint photographic experts group (JPEG) can provide good quality with small file size but also eliminate extensively the redundancies of images. Therefore, hiding data into JPEG images in terms of maintaining high visual quality at small file sizes has been a great challenge for researchers. In this paper, an adaptive reversible data hiding method for JPEG images containing multiple two-dimensional (2D) histograms is proposed. Adaptability is mainly reflected in three aspects. The first one is to preferentially select sharper histograms for data embedding after K histograms are established by constructing the kth (k∈{1,2,…,K}) histogram using the kth non-zero alternating current (AC) coefficient of all the quantized discrete cosine transform blocks. On the other hand, to fully exploit the strong correlation between coefficients of one histogram, the smoothness of each coefficient is estimated by a block smoothness estimator so that a sharply-distributed 2D-histogram is constructed by combining two coefficients with similar smoothness into a pair. The pair corresponding to low complexity is selected priorly for data embedding, leading to high embedding performance while maintaining low file size. Besides, we design multiple embedding strategies to adaptively select the embedding strategy for each 2D histogram. Experimental results demonstrate that the proposed method can achieve higher rate–distortion performance which maintaining lower file storage space, compared with previous studies.

Rate–distortion optimal evolutionary algorithm for JPEG quantization with multiple rates

The JPEG standard is used extensively in image-related applications. The optimization on a JPEG-standard compressor is of high importance for image storage and transmission. In this paper, we propose a versatile and efficient quantization optimization method for image compression. It is based on multi-objective optimization and is JPEG standard-compatible. Unlike the existing optimization methods, our method fully considers the rate–distortion optimal principle and provides several optimal solutions to address the multiple-rate requirement in applications. In the multi-objective evolutionary optimization framework, the fitness of each quantization table for JPEG compression is evaluated efficiently by the searching in a look-up table, which is constructed based on the statistics of each DCT band in a pre-defined manner. Then, the population update strategy based on the rate–distortion optimal principle is recommended to guide the evolution toward the best rate–distortion performance. Furthermore, to maintain the population’s diversity and uniformity, convex-hull based environmental selection is recommended to identify the solutions at the first Pareto front, and the identified solutions are enriched further with scaling operation to fill the blank rate range. The experimental results for several classic datasets demonstrate the superiority of our method in terms of solution distribution, coding efficiency, and computational complexity.

Accelerating Transform Algorithm Implementation for Efficient Intra Coding of 8K UHD Videos

Real-time ultra-high-definition (UHD) video applications have attracted much attention, where the encoder side urgently demands the high-throughput two-dimensional (2D) transform hardware implementation for the latest video coding standards. This article proposes an effective acceleration method for transform algorithm in UHD intra coding based on the third generation of audio video coding standard (AVS3). First, by conducting detailed statistical analysis, we devise an efficient hardware-friendly transform algorithm that can reduce running cycles and resource consumption remarkably. Second, to implement multiplierless computation for saving resources and power, a series of shift-and-add unit (SAU) hardwares are investigated to have much less adoptions of shifters and adders than the existing methods. Third, different types of hardware acceleration methods, including calculation pipelining, logical-loop unrolling, and module-level parallelism, are designed to efficaciously support the data-intensive high frame-rate 8K UHD video coding. Finally, due to the scarcity of 8K video sources, we also provide a new dataset for the performance verification. Experimental results demonstrate that our proposed method can effectively fulfill the real-time 8K intra encoding at beyond 60 fps, with very negligible loss on rate-distortion (R-D) performance, which is averagely 0.98% Bjontegaard-Delta Bit-Rate (BD-BR).

Low complexity mode selection for H.266/VVC intra coding

The newest video compression standard, called Versatile Video Coding (H.266/VVC), outperforms its predecessor High Efficiency Video Coding (HEVC) by up to 50% in the matter of coding performance. The achievement results from the integrating of several sophisticated tools such as 32 extended angular modes for intra coding, advanced partitioning structure, and additional transform selection. However, these tools negatively deliver the computational cost of the VVC Test Model (VTM) in all-intra encoding configuration. The existing fast intra coding algorithms achieve a significant saving in computational time with a comparable rate distortion (RD) performance of VVC plus additional computational complexity. In this paper, we study the original strategy of VVC intra mode decision and analyze the correlation among the sum of absolute transformed differences (SATD) costs of the rough mode decision (RMD) and the RD costs of the rate distortion optimization (RDO) processes. Based on the useful correlations with an effective threshold formulation, we propose a fast intra prediction mode selection for the most time-consuming RDO process by selecting a small candidate set based on the SATD costs and calculating the time-consuming RD costs for only intra prediction modes of the small candidate set without a significant overhead. According to the experimental results, our proposed system can reduce up to 27.38% and on average 21.07% of the computation complexity of VTM version 5.0 with an insignificant quality degradation.

Consistent Quality Oriented Rate Control in HEVC Via Balancing Intra and Inter Frame Coding

Consistent quality oriented rate control in video coding has attracted much more attention. However, the existing efforts only focus on decreasing variations between every two adjacent frames, but neglect coding trade-off problem between intra and inter frames. In this paper, we deal with it from a new perspective, where intra frame quantization parameter (IQP) and rate control are optimized for balanced coding. First, due to the importance of intra frames, a new framework is proposed for consistent quality oriented IQP prediction, and then we remove unqualified IQP candidates using the proposed penalty term. Second, we extensively evaluate possible features, and select target bits per pixel for all remaining frames, average and standard variance of frame QPs, where equivalent acquisition methods for QP features are given. Third, predicted IQPs are clipped effectively according to bandwidth and previous information for better bit rate accuracy. Compared with High Efficiency Video Coding (HEVC) reference baseline, experiments demonstrate that our method reduces quality fluctuation greatly by 37.2% on frame-level standard variance of peak-signal-noise-ratio (PSNR) and 45.1% on that of structural similarity (SSIM). Moreover, it also can have satisfactory results on Rate-Distortion (R-D) performance, bit accuracy and buffer control.

Green Visual Sensor of Plant: An Energy-Efficient Compressive Video Sensing in the Internet of Things.

Internet of Things (IoT) realizes the real-time video monitoring of plant propagation or growth in the wild. However, the monitoring time is seriously limited by the battery capacity of the visual sensor, which poses a challenge to the long-working plant monitoring. Video coding is the most consuming component in a visual sensor, it is important to design an energy-efficient video codec in order to extend the time of monitoring plants. This article presents an energy-efficient Compressive Video Sensing (CVS) system to make the visual sensor green. We fuse a context-based allocation into CVS to improve the reconstruction quality with fewer computations. Especially, considering the practicality of CVS, we extract the contexts of video frames from compressive measurements but not from original pixels. Adapting to these contexts, more measurements are allocated to capture the complex structures but fewer to the simple structures. This adaptive allocation enables the low-complexity recovery algorithm to produce high-quality reconstructed video sequences. Experimental results show that by deploying the proposed context-based CVS system on the visual sensor, the rate-distortion performance is significantly improved when comparing it with some state-of-the-art methods, and the computational complexity is also reduced, resulting in a low energy consumption.

Learning-Based Rate Control for Video-Based Point Cloud Compression.

Due to limited transmission resources and storage capacity, efficient rate control is important in Video-based Point Cloud Compression (V-PCC). In this paper, we propose a learning-based rate control method to improve the rate-distortion (RD) performance of V-PCC. A low-latency synchronous rate control structure is designed to reduce the overhead of pre-coding. The basic unit (BU) parameters are predicted accurately based on our proposed CNN-LSTM neural network, instead of the online updating approach, which can be inaccurate due to low consistency between adjacent 2D frames in V-PCC. When determining the quantization parameters for the BU, a patch-based clipping method is proposed to avoid unnecessary clipping. This approach is able to improve the RD performance and subjective dynamic point cloud quality. Experiments show that our proposed rate control method outperforms present approaches.

Synthetic Aperture Radar Image Compression Based on a Variational Autoencoder

Given the uniqueness of synthetic aperture radar (SAR) images, traditional optical image compression algorithms cannot fully exploit their redundant information. To improve SAR image compression in terms of rate–distortion performance and visual perception, an end-to-end SAR image compression convolutional neural network (CNN) model based on a variational autoencoder is proposed. The proposed CNN model consists of a main autoencoder and a hyper autoencoder. To reduce dependencies in latent space, a joint transform of linear CNN and nonlinear generalized divisive normalization (GDN) activation is applied in the main autoencoder. Moreover, residual blocks are combined with the transforms to boost the efficiency of feature learning and make use of subpixels to improve the quality of reconstructed images. Instead of a fixed entropy model, a conditioned entropy model that works with a hyperprior network is used to learn the distribution of latents, which helps to further improve the compression quality. During training, the model is optimized by evaluating the rate–distortion performance. The experimental results show that the proposed method can achieve better distortion performance than JPEG, JPEG2000, and the available CNN-based method in terms of objective evaluation criteria and human vision perception quality.

Inter-Frame Dependency-Based Rate Control for VVC Low-Delay Coding

In this letter, we propose two solutions for the rate control of the VVC low-delay coding. Both solutions are developed by determining the bit allocation factors for video frames based on their dependency. Specifically, we design the first solution according to the distortion correlation between the key-frame and its subsequent frames. With this solution, the bit allocation factors are determined by applying multi-pass coding on the video to build up the cross-frame distortion model. This model offers us a straightforward way to achieve better rate control performance but results in a rather high complexity. To solve the complexity problem, we propose the second solution based on the difference between frames. In this solution, we construct the bit allocation model and apply it to frames so that we can adaptively determine the allocation factors with a low complexity. The experimental results demonstrate that our proposed two solutions can offer better rate-distortion performances than the state-of-the-art method.

APMC: Adjacent Pixels Based Measurement Coding System for Compressively Sensed Images

Compressed sensing is now regarded as an effective method for dimension reduction during signal acquisition. One significant, yet under-addressed issue regarding the transmission of compressively sensed images is how to compress and code measurements, the output of a compressed sensing sensor. As the spatially adjacent correlation in the measurement domain is weak, conventional encoding algorithms cannot be applied directly for measurements. In this paper, we propose the adjacent pixels based measurement coding system (APMC) to generate compressed image bit-streams. Firstly, an adjacent pixels based measurement matrix (APMM) is applied to embed the pixel-domain boundary information of each block to the measurement domain. By adopting APMM, the pixel-domain information can be efficiently used for measurement-domain intra prediction. Moreover, to avoid the interference of pixels that are far apart and achieve a high prediction accuracy, we employ boundary measurements of neighboring blocks as references for prediction. Finally, we propose a rate control algorithm to process the residuals between measurements and predictions, to generate a coded bit sequence for transmitting. Experimental results demonstrated superiority in rate-distortion performance and bandwidth costs in transmitting as compared to previous schemes. Compared to the state-of-the-art, this work achieves a 24&#x0025; decrease in bitrate and a 1.68 dB increase in Peak Signal-to-Noise Ratio (PSNR) on average.

Rate-Distortion Optimized Graph for Point Cloud Attribute Coding

Recent years have witnessed remarkable success of Graph Fourier Transform (GFT) in point cloud attribute compression. Existing researches mainly utilize geometry distance to define graph structure for coding attribute (e.g., color), which may distribute high weights to the edges connecting points across texture boundaries. In this case, these geometry-based graphs cannot model attribute differences between points adequately, thus limiting the compression efficiency of GFT. Hence, we firstly utilize attribute itself to refine the distance-based weight values by setting penalty function, which smoothens signal variations on graph and concentrates more energies in the low frequencies. Then, adjacency matrices acting as penalty function variables are transmitted to decoder with extra bit overheads. To balance the attribute smoothness on graph and the cost of coding adjacency matrices, we finally propose the graph based on Rate-Distortion (RD) optimization and find the optimal adjacency matrix. Experimental results show that our algorithm improves RD performance compared with competitive platforms. Moreover, additional experiments also analyze the gain source by evaluating the effectiveness of RD optimized graphs.